Val loss #1899

kohya-ss · 2025-01-26T12:28:39Z

#1864 and small fix.

We only want to be enabling grad if we are training.

… to calculate validation loss. Balances the influence of different time steps on training performance (without affecting actual training results)

…_trackers library code

… fix multiple datasets

…plit check and warning

Divergence is the difference between training and validation to allow a clear value to indicate the difference between the two in the logs.

Adding metadata recording for validation arguments Add comments about the validation split for clarity of intention

…idation dataset

Validation loss

…upport for training scripts

rockerBOO and others added 30 commits November 5, 2023 12:35

Add validation loss

5b19bda

new ratio code

33c311e

Add validation split of datasets

3de9e6c

Update args to validation_seed and validation_split

a93c524

Add process_batch for train_network

c892521

Removed/cleanup a line

e545fdf

Remove unnecessary subset line from collate

9c591bd

Set grad enabled if is_train and train_text_encoder

569ca72

We only want to be enabling grad if we are training.

val

b558a5b

improve

78cfb01

Update train_network.py

923b761

Update train_network.py

47359b8

fix timesteps

a51723c

only use train subset to val

7d84ac2

Update train_network.py

befbec5

Update train_network.py

63e58f7

Update train_network.py

a6c41c6

fix

bd7e229

Merge remote-tracking branch 'kohya-ss/dev' into val

5d7ed0d

Update train_network.py

d05965d

fix control net

b5e8045

Merge branch 'main' into val

086f600

Update config_util.py

36d4023

Update train_util.py

229c5a3

Update config_util.py

3b251b7

Update config_util.py

459b125

Update train_util.py

89ad69b

Update config_util.py

fde8026

Remove unnecessary is_train changes and use apply_debiased_estimation…

31507b9

… to calculate validation loss. Balances the influence of different time steps on training performance (without affecting actual training results)

Update train_db.py

1db4951

rockerBOO and others added 26 commits January 3, 2025 15:25

Move get_huber_threshold_if_needed

695f389

Add step break for validation epoch. Remove unused variable

1f9ba40

Add missing functions for training batch

1c0ae30

Use self.get_noise_pred_and_target and drop fixed timesteps

bbf6bbd

Revert train_db.py

f4840ef

Cleanup unused code and formatting

1c63e7c

Add validate_every_n_epochs, change name validate_every_n_steps

c64d1a2

Fix validate epoch, cleanup imports

f885029

Clean up some validation help documentation

fcb2ff0

Set validation steps in multiple lines for readability

742bee9

Remove unused train_util code, fix accelerate.log for wandb, add init…

1231f51

…_trackers library code

Fix documentation, remove unused function, fix bucket reso for sd1.5,…

556f3f1

… fix multiple datasets

Handle tuple return from generate_dataset_group_by_blueprint

9fde0d7

Revert bucket_reso_steps to correct 64

1e61392

Fix incorrect destructoring for load_abritrary_dataset

d6f158d

Apply is_training_dataset only to DreamBoothDataset. Add validation_s…

264167f

…plit check and warning

Add divergence to logs

4c61adc

Divergence is the difference between training and validation to allow a clear value to indicate the difference between the two in the logs.

Fix regularization images with validation

2bbb40c

Adding metadata recording for validation arguments Add comments about the validation split for clarity of intention

Fix validate_every_n_steps always running first step

0456858

Fix validate_every_n_steps for gradient accumulation

ee9265c

Remove Validating... print to fix output layout

25929dd

Disable repeats for validation datasets

b489082

Fix loss recorder on 0. Fix validation for cached runs. Assert on val…

c04e5df

…idation dataset

Merge pull request #1864 from rockerBOO/validation-loss-upstream

b833d47

Validation loss

Fix to work with validation dataset

58b82a5

README.md: Update recent updates section to include validation loss s…

e852961

…upport for training scripts

kohya-ss merged commit f1ac81e into sd3 Jan 26, 2025
2 checks passed

kohya-ss deleted the val-loss branch January 26, 2025 12:32

This was referenced Jan 26, 2025

Validation Loss Enhancements #1900

Open

Implement Test and Validation Set Loss #1897

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Val loss #1899

Val loss #1899

kohya-ss commented Jan 26, 2025

Val loss #1899

Val loss #1899

Conversation

kohya-ss commented Jan 26, 2025