Stable Diffusion XL fine-tuning with Dreambooth & Lora: how to structure local dataset for fine-tuning with ROI

[SylwiaNowakowska]

Hi!

I am using https://github.com/huggingface/diffusers/blob/main/examples/dreambooth/train_dreambooth_lora_sdxl.py
I am having a custom dataset locally, which is structured in a following way:

data/metadata.jsonl
data/train/01.png
data/train/02.png
....

With the metadata.json file as follows:
{"file_name": "train/01.png", "text": "In the style of MAMM, a view X of a healthy patient"}
{"file_name": "train/02.png", "text": "In the style of MAMM, a view Y of a healthy patient"}
....

The fine-tuning works.

In the next step, I would like to take the fine-tuned model and train on images which show abnormality.
I have ROI coordinates for the abnormality.

I would be grateful, if you could tell me how shall I structure the dataset for this task?

Shall I add the ROI coordinates to a prompt in similar way as here: https://huggingface.co/docs/datasets/image_dataset (section: "object detection")?
If yes, should it look sth like this?
{"file_name": "train/01.png", "text": "In the style of MAMM, a view X, "objects" : {"bbox": [[302.0, 109.0, 73.0, 52.0]], "categories": [abnormality A]}}}

Or to add the ROIs as a segmentation map in form of an image? (How to structure the folder then and the metadata.jsonl?)

I would appreciate your help.

[sayakpaul]

Cc: @lhoestq from the datasets team.

[SylwiaNowakowska]

I would appreciate your input on that.

[lhoestq]

The format you mentioned sounds good to me :) but it also depends on the training script you want to use, since it may expect the data in a specific format.

Which training script are you using for object detection ?

[SylwiaNowakowska]

Thx for your feedback.
I am using: I am using https://github.com/huggingface/diffusers/blob/main/examples/dreambooth/train_dreambooth_lora_sdxl.py

Could you let me know, if the ROI should be better in the metadata.jsonl or saved as masks in .png format?

[lhoestq]

I think this script is to train a dreambooth model only using text and image - it doesn't seem to support bounding boxes or categories.

Maybe you can simply filter your images to keep the ones with abnormalities and use this filtered dataset instead ? You could even mention the abnormality name in the texts.

[SylwiaNowakowska]

Thank you for the answer.
Do you plan to extend the script to support training with ROI?

[sayakpaul]

No, we don't plan on doing that because our scripts are not meant to be exhaustive.

[SylwiaNowakowska]

Thx!