feat: Add feature to store custom_files and config in a subdir for each experiment

[mattcamp]

The feature enables each training session to have it's config and custom_files stored within a subdir under experiments/.

This simplifies being able to locate the config and files that were used for a particular training session.

An additional flag is added to system.env to control the experiment name, which when set will cause the rest of the config to be loaded from $DR_DIR/experiments/<experiment_name>/run.env and $DR_DIR/experiments/<experiment_name>/custom_files/.

[larsll]

Question; does things like dr-increment-training and the -1 flag to dr-upload-model work?

[mattcamp]

Question; does things like dr-increment-training and the -1 flag to dr-upload-model work?

Yes, they work fine, although the increment-training will just update the current run.env inside the same experiment folder.

I plan to add another command to increment the whole experiment folder which will also update the DR_EXPERIMENT_NAME and increment model prefix + set pretrained prefix.

[larsll]

Currently we are using bin/activate.sh as default, but it also supports bin/activate.sh run-1.env as an example; rather than having it in system.env, could we have it as bin/activate.sh -e <experiment>?

[larsll]

Would this not better be a run.env variable?