OoD Generalization

(currently maintained by Dinghuai Zhang and Irina Rish)

Surveys:

Domain Generalization: A Survey

Papers:

Invariant Risk Minimization

Invariant Risk Minimization: An Information Theoretic View (blog)

Leon Bottou's talk: Learning Representations Using Causal Inference (Workshop on Theory of Deep Learning: Where next?)

Linear unit-tests for invariance discovery several linear settings as testbed for IRM, REx, etc...

Invariant Risk Minimization Games (ICML2020) game-theoretic reformulation of IRM (Kartik Ahuja's talk on IRM Games paper )

Shortcut Learning in Deep Neural Networks (Nature machine intelligence) A relevant survey

Invariant Rationalization interesting formulation, invariance + mutual information

Domain Extrapolation via Regret Minimization similar to IRM, replace the loss in a regret form

Out-of-Distribution Generalization via Risk Extrapolation (REx) (slides) (ICML2021) generalization of IRM, propose MM-REx and V-REx (use variance among losses from domains as penalty)

Learning Robust Representations with Score Invariant Learning modification of REx, replace loss with || \nabla_{\theta} loss ||

Risk Variance Penalization: From Distributional Robustness to Causality generalization of REx, replace variance with standard deviation....

Generalization and Invariances in the Presence of Unobserved Confounding seems exactly same as (while 4 months later than) REx...

Out-of-Distribution Generalization with Maximal Invariant Predictor modification to V-REx, replace "loss in one domain" with "gradient of loss in that domain"

Learning Causal Models Online IRM setting + continual learning

Self-training Avoids Using Spurious Features Under Domain Shift theoretically prove self-training can prevent from using spurious feature in linear setting

Gradient Starvation: A Learning Proclivity in Neural Networks analyze sigmoid binary classification with Legendre dual, propose a new regularization ||y_hat||.

Improving out-of-distribution generalization via multi-task self-supervised pretraining

Learning explanations that are hard to vary (ILC) (ICLR2021) Invariant Learning Consistency (ILC) criterion, (approx) AND-masking of gradients to only keep directions consistent (across domains)

In Search of Lost Domain Generalization (ICLR2021)

The Risks of Invariant Risk Minimization (ICLR2021) When the number of environments is small, IRM will fall

Understanding the Failure Modes of Out-of-Distribution Generalization (ICLR2021)

REPRESENTATION LEARNING VIA INVARIANT CAUSAL MECHANISMS (ICLR2021) improve contrastive learning with style-augmentation

SYSTEMATIC GENERALISATION WITH GROUP INVARIANT PREDICTIONS (ICLR2021)

Empirical or Invariant Risk Minimization? A Sample Complexity Perspective (ICLR2021) Theoretically calculate sample complexity of IRM (which is similar to ERM)

ICLR2021 rejected submissions:

OUT-OF-DISTRIBUTION PREDICTION WITH INVARIANT RISK MINIMIZATION: THE LIMITATION AND AN EFFECTIVE FIX irm + transfer learning loss trick (like mmd loss)

INVARIANT CAUSAL REPRESENTATION LEARNING

LATENT CAUSAL INVARIANT MODEL

Domain-Free Adversarial Splitting for Domain Generalization IRM + environment unaware setting + link to fairness

OUT-OF-DISTRIBUTION GENERALIZATION ANALYSIS VIA INFLUENCE FUNCTION analysis via influence function

LEARNING ROBUST MODELS USING THE PRINCIPLE OF INDEPENDENT CAUSAL MECHANISMS causal + normalizing flow + HSIC for independence

Does Invariant Risk Minimization Capture Invariance? (AISTATS2021 oral) fairly good paper, analyzing when will IRMv1 fail (to get the solution of IRM), and when will IRM itself fail

Linear Regression Games: Convergence Guarantees to Approximate Out-of-Distribution Solutions (AISTATS2021)

Meta-Learned Invariant Risk Minimization MAML + REx ...

Evading the Simplicity Bias: Training a Diverse Set of Models Discovers Solutions with Superior OOD Generalization punish the dot product of gradients from different models to enforce diversity + OOD model selection

Environment Inference for Invariant Learning (ICML2021) demographic unaware setting (and some discussion related to fairness)

Predict then Interpolate: A Simple Algorithm to Learn Stable Classifiers (ICML2021)

Heterogeneous Risk Minimization (ICML2021)

Just Train Twice: Improving Group Robustness without Training Group Information (ICML2021) train once, then re-train while up-weighting the data that previously has large loss

OoD-Bench: Benchmarking and Understanding Out-of-Distribution Generalization Datasets and Algorithms

Towards a Theoretical Framework of Out-of-Distribution Generalization adopt a conditional DA like term to bound OOD error

An Information-theoretic Approach to Distribution Shifts

Causality & ML

Causality for Machine Learning

Invariance, Causality and Robustness

Causal inference using invariant prediction: identification and confidence intervals ICP (invariant causal prediction) paper

Invariant Causal Prediction for Nonlinear Models nonlinear ICP

Invariant Causal Prediction for Block MDPs ICP in MDP settings

Invariant Models for Causal Transfer Learning causal transfer paper

Anchor regression: heterogeneous data meets causality

Learning Independent Causal Mechanisms

Recurrent Independent Mechanisms

Learning Neural Causal Models from Unknown Interventions