v23.1.0

[bmaltais]

• Update sd-scripts to 0.8.7

  • The default value of huber_schedule in Scheduled Huber Loss is changed from exponential to snr, which is expected to give better results.

  • Highlights

    • The dependent libraries are updated. Please see Upgrade and update the libraries.
      • Especially imagesize is newly added, so if you cannot update the libraries immediately, please install with pip install imagesize==1.4.1 separately.
      • bitsandbytes==0.43.0, prodigyopt==1.0, lion-pytorch==0.0.6 are included in the requirements.txt.
        • bitsandbytes no longer requires complex procedures as it now officially supports Windows.
      • Also, the PyTorch version is updated to 2.1.2 (PyTorch does not need to be updated immediately). In the upgrade procedure, PyTorch is not updated, so please manually install or update torch, torchvision, xformers if necessary (see Upgrade PyTorch).
    • When logging to wandb is enabled, the entire command line is exposed. Therefore, it is recommended to write wandb API key and HuggingFace token in the configuration file (.toml). Thanks to bghira for raising the issue.
      • A warning is displayed at the start of training if such information is included in the command line.
      • Also, if there is an absolute path, the path may be exposed, so it is recommended to specify a relative path or write it in the configuration file. In such cases, an INFO log is displayed.
      • See #1123 and PR #1240 for details.
    • Colab seems to stop with log output. Try specifying --console_log_simple option in the training script to disable rich logging.
    • Other improvements include the addition of masked loss, scheduled Huber Loss, DeepSpeed support, dataset settings improvements, and image tagging improvements. See below for details.

  • Training scripts

    • train_network.py and sdxl_train_network.py are modified to record some dataset settings in the metadata of the trained model (caption_prefix, caption_suffix, keep_tokens_separator, secondary_separator, enable_wildcard).
    • Fixed a bug that U-Net and Text Encoders are included in the state in train_network.py and sdxl_train_network.py. The saving and loading of the state are faster, the file size is smaller, and the memory usage when loading is reduced.
    • DeepSpeed is supported. PR #1101 and #1139 Thanks to BootsofLagrangian! See PR #1101 for details.
    • The masked loss is supported in each training script. PR #1207 See Masked loss for details.
    • Scheduled Huber Loss has been introduced to each training scripts. PR #1228 Thanks to kabachuha for the PR and cheald, drhead, and others for the discussion! See the PR and Scheduled Huber Loss for details.
    • The options --noise_offset_random_strength and --ip_noise_gamma_random_strength are added to each training script. These options can be used to vary the noise offset and ip noise gamma in the range of 0 to the specified value. PR #1177 Thanks to KohakuBlueleaf!
    • The options --save_state_on_train_end are added to each training script. PR #1168 Thanks to gesen2egee!
    • The options --sample_every_n_epochs and --sample_every_n_steps in each training script now display a warning and ignore them when a number less than or equal to 0 is specified. Thanks to S-Del for raising the issue.

  • Dataset settings

    • The English version of the dataset settings documentation is added. PR #1175 Thanks to darkstorm2150!
    • The .toml file for the dataset config is now read in UTF-8 encoding. PR #1167 Thanks to Horizon1704!
    • Fixed a bug that the last subset settings are applied to all images when multiple subsets of regularization images are specified in the dataset settings. The settings for each subset are correctly applied to each image. PR #1205 Thanks to feffy380!
    • Some features are added to the dataset subset settings.
      • secondary_separator is added to specify the tag separator that is not the target of shuffling or dropping.
        • Specify secondary_separator=";;;". When you specify secondary_separator, the part is not shuffled or dropped.
      • enable_wildcard is added. When set to true, the wildcard notation {aaa|bbb|ccc} can be used. The multi-line caption is also enabled.
      • keep_tokens_separator is updated to be used twice in the caption. When you specify keep_tokens_separator="|||", the part divided by the second ||| is not shuffled or dropped and remains at the end.
      • The existing features caption_prefix and caption_suffix can be used together. caption_prefix and caption_suffix are processed first, and then enable_wildcard, keep_tokens_separator, shuffling and dropping, and secondary_separator are processed in order.
      • See Dataset config for details.
    • The dataset with DreamBooth method supports caching image information (size, caption). PR #1178 and #1206 Thanks to KohakuBlueleaf! See DreamBooth method specific options for details.

  • Image tagging

    • The support for v3 repositories is added to tag_image_by_wd14_tagger.py (--onnx option only). PR #1192 Thanks to sdbds!
      • Onnx may need to be updated. Onnx is not installed by default, so please install or update it with pip install onnx==1.15.0 onnxruntime-gpu==1.17.1 etc. Please also check the comments in requirements.txt.
    • The model is now saved in the subdirectory as --repo_id in tag_image_by_wd14_tagger.py . This caches multiple repo_id models. Please delete unnecessary files under --model_dir.
    • Some options are added to tag_image_by_wd14_tagger.py.
      • Some are added in PR #1216 Thanks to Disty0!
      • Output rating tags --use_rating_tags and --use_rating_tags_as_last_tag
      • Output character tags first --character_tags_first
      • Expand character tags and series --character_tag_expand
      • Specify tags to output first --always_first_tags
      • Replace tags --tag_replacement
      • See Tagging documentation for details.
    • Fixed an error when specifying --beam_search and a value of 2 or more for --num_beams in make_captions.py.

  • About Masked loss
    The masked loss is supported in each training script. To enable the masked loss, specify the --masked_loss option.

    The feature is not fully tested, so there may be bugs. If you find any issues, please open an Issue.

    ControlNet dataset is used to specify the mask. The mask images should be the RGB images. The pixel value 255 in R channel is treated as the mask (the loss is calculated only for the pixels with the mask), and 0 is treated as the non-mask. The pixel values 0-255 are converted to 0-1 (i.e., the pixel value 128 is treated as the half weight of the loss). See details for the dataset specification in the LLLite documentation.

  • About Scheduled Huber Loss
    Scheduled Huber Loss has been introduced to each training scripts. This is a method to improve robustness against outliers or anomalies (data corruption) in the training data.

    With the traditional MSE (L2) loss function, the impact of outliers could be significant, potentially leading to a degradation in the quality of generated images. On the other hand, while the Huber loss function can suppress the influence of outliers, it tends to compromise the reproduction of fine details in images.

    To address this, the proposed method employs a clever application of the Huber loss function. By scheduling the use of Huber loss in the early stages of training (when noise is high) and MSE in the later stages, it strikes a balance between outlier robustness and fine detail reproduction.

    Experimental results have confirmed that this method achieves higher accuracy on data containing outliers compared to pure Huber loss or MSE. The increase in computational cost is minimal.

    The newly added arguments loss_type, huber_schedule, and huber_c allow for the selection of the loss function type (Huber, smooth L1, MSE), scheduling method (exponential, constant, SNR), and Huber's parameter. This enables optimization based on the characteristics of the dataset.

    See PR #1228 for details.

    • loss_type: Specify the loss function type. Choose huber for Huber loss, smooth_l1 for smooth L1 loss, and l2 for MSE loss. The default is l2, which is the same as before.
    • huber_schedule: Specify the scheduling method. Choose exponential, constant, or snr. The default is snr.
    • huber_c: Specify the Huber's parameter. The default is 0.1.

    Please read Releases for recent updates.`

• Added GUI support for the new parameters listed above.

• Moved accelerate launch parameters to a new Accelerate launch accordion above the Model accordion.

• Added support for Debiased Estimation loss to Dreambooth settings.

• Added support for "Dataset Preparation" defaults via the config.toml file.

• Added a field to allow for the input of extra accelerate launch arguments.

• Added new caption tool from https://github.com/kainatquaderee