flooding

Demo code for the flooding regularizer proposed in "Do We Need Zero Training Loss After Achieving Zero Training Error?" (ICML 2020) by Takashi Ishida, Ikko Yamane, Tomoya Sakai, Gang Niu, and Masashi Sugiyama.

Introduction

Overparameterized deep networks have the capacity to memorize training data with zero training error. Even after memorization, the training loss continues to approach zero, making the model overconfident and the test performance degraded.

Flooding is a direct solution to this issue. It intentionally prevents further reduction of the training loss when it reaches a reasonably small value, which is called the "flooding level". Flooding makes the loss float around the flooding level by doing

• (mini-batched) gradient descent as usual but
• gradient ascent if the training loss is below the flooding level.

This can be implemented by adding one line to your code:

loss = (loss-b).abs() + b  # b is the flooding level.

and is compatible with any stochastic optimizer and other regularizers. Figures from the paper: (a) shows 3 different concepts related to overfitting. [A] shows the generalization gap increases, while training & test losses decrease. [B] also shows the increasing gap, but the test loss starts to rise. [C] shows the training loss becoming (near-)zero. We avoid [C] by flooding the bottom area, visualized in (b), which forces the training loss to stay around a constant. This leads to a decreasing test loss once again. The paper presents experiments with CIFAR-10 shown in (c)–(d) supporting these claims.

Demo

Requirements

• pytorch 1.3.1
• mlflow 1.5.0
• matplotlib 3.1.1

Execution

Run the following python script for a simple demonstration of flooding with synthetic datasets:

python run_demo_synth.py

Two png files will be saved in your directory, which are visualizations of the results with synthetic datasets. See the paper for the details of the synthetic datasets.