# OoD Generalization

**Surveys:**

Domain Generalization: A Survey

**Papers:**

**Leon Bottou's talk: **Learning Representations Using Causal Inference (Workshop on Theory of Deep Learning: Where next?)

**Leon Bottou's talk:**Learning Representations Using Causal Inference (Workshop on Theory of Deep Learning: Where next?)

### Invariant Risk Minimization Games (ICML2020) game-theoretic reformulation of IRM (**Kartik Ahuja's **talk on IRM Games paper )

**Kartik Ahuja's**talk on IRM Games paper )

Invariant Rationalization interesting formulation, invariance + mutual information

Domain Extrapolation via Regret Minimization similar to IRM, replace the loss in a regret form

Out-of-Distribution Generalization via Risk Extrapolation (REx) (slides) (ICML2021) generalization of IRM, propose MM-REx and V-REx (use variance among losses from domains as penalty)

Learning Robust Representations with Score Invariant Learning modification of REx, replace loss with || \nabla_{\theta} loss ||

Risk Variance Penalization: From Distributional Robustness to Causality generalization of REx, replace variance with standard deviation....

Generalization and Invariances in the Presence of Unobserved Confounding seems exactly same as (while 4 months later than) REx...

Out-of-Distribution Generalization with Maximal Invariant Predictor modification to V-REx, replace "loss in one domain" with "gradient of loss in that domain"

Learning Causal Models Online IRM setting + continual learning

Self-training Avoids Using Spurious Features Under Domain Shift theoretically prove self-training can prevent from using spurious feature in linear setting

Gradient Starvation: A Learning Proclivity in Neural Networks analyze sigmoid binary classification with Legendre dual, propose a new regularization ||y_hat||.

Improving out-of-distribution generalization via multi-task self-supervised pretraining

Learning explanations that are hard to vary (ILC) (ICLR2021) Invariant Learning Consistency (ILC) criterion, (approx) AND-masking of gradients to only keep directions consistent (across domains)

In Search of Lost Domain Generalization (ICLR2021)

The Risks of Invariant Risk Minimization (ICLR2021) When the number of environments is small, IRM will fall

Understanding the Failure Modes of Out-of-Distribution Generalization (ICLR2021)

REPRESENTATION LEARNING VIA INVARIANT CAUSAL MECHANISMS (ICLR2021) improve contrastive learning with style-augmentation

SYSTEMATIC GENERALISATION WITH GROUP INVARIANT PREDICTIONS (ICLR2021)

Empirical or Invariant Risk Minimization? A Sample Complexity Perspective (ICLR2021) Theoretically calculate sample complexity of IRM (which is similar to ERM)

### ICLR2021 rejected submissions:

### OUT-OF-DISTRIBUTION PREDICTION WITH INVARIANT RISK MINIMIZATION: THE LIMITATION AND AN EFFECTIVE FIX irm + transfer learning loss trick (like mmd loss)

### Domain-Free Adversarial Splitting for Domain Generalization IRM + environment unaware setting + link to fairness

### LEARNING ROBUST MODELS USING THE PRINCIPLE OF INDEPENDENT CAUSAL MECHANISMS causal + normalizing flow + HSIC for independence

### Does Invariant Risk Minimization Capture Invariance? (AISTATS2021 oral) fairly good paper, analyzing when will IRMv1 fail (to get the solution of IRM), and when will IRM itself fail

### Evading the Simplicity Bias: Training a Diverse Set of Models Discovers Solutions with Superior OOD Generalization punish the dot product of gradients from different models to enforce diversity + OOD model selection

### Environment Inference for Invariant Learning (ICML2021) demographic unaware setting (and some discussion related to fairness)

### Just Train Twice: Improving Group Robustness without Training Group Information (ICML2021) train once, then re-train while up-weighting the data that previously has large loss

### Towards a Theoretical Framework of Out-of-Distribution Generalization adopt a conditional DA like term to bound OOD error

**Causality & ML**

**Causality & ML**