Adds the ability to add images to an existing run

[ajstewart]

This adds the feature of adding an image to an existing run.

• Rebuilds the initial association data frames from the current parquet files available.
• Performs pre-run checks to see if the job config has actually changed - will not run if any other parameters have changed.
• Adding mode is default re-run mode, to force a complete re-run there is the option --complete-rerun when running.
• Creates backups of parquets for completed runs so that the next adding run can use them.
• Also creates a config_prev.py for future jobs.
• Sources are updated using a SQL statement.
• Ideal and new source stages are left the same (this enables images to be added that are anywhere in time, i.e. not limited to only adding images in the future.)
• Forced extractions are only run for those measurements needed.
• At the upload stage the old parquets are used once again to see which ones do not need to be uploaded.
• UI changes to accommodate the new option.
• Adds various tests to test the new feature.

I have tested it on a large scale run by adding epoch 12 to epochs 1 - 11 and it works well.

Also fixes issues #444 and #451 that were found while developing this feature.

Fixes #212.
Fixes #444.
Fixes #451.

@dliptai Fixes #461

[srggrs]

@ajstewart minor things mostly about indentation

@ellawang44 I can see vast_pipeline/tests/test_regression/gen_config.py but why config files are still present? (e.g. vast_pipeline/tests/pipeline-runs/regression/parallel-deruiter/config.py)

[ellawang44]

@srggrs Just finished changing over to generating all the config files on the fly in fd9bb23. The config files under vast_pipeline/tests/pipeline-runs/regression should all be gone now.

I had to modify the template slightly because sometimes the inputs are dictionaries but the template assumes lists. This change should not affect previously generated config files.

[srggrs]

@srggrs Just finished changing over to generating all the config files on the fly in fd9bb23. The config files under vast_pipeline/tests/pipeline-runs/regression should all be gone now.

I had to modify the template slightly because sometimes the inputs are dictionaries but the template assumes lists. This change should not affect previously generated config files.

Thanks! I think the test suit should be modified as well... or maybe not? Anyhow the tests fails.... due to missing folders/files.

[ellawang44]

@srggrs Just finished changing over to generating all the config files on the fly in fd9bb23. The config files under vast_pipeline/tests/pipeline-runs/regression should all be gone now.
I had to modify the template slightly because sometimes the inputs are dictionaries but the template assumes lists. This change should not affect previously generated config files.

Thanks! I think the test suit should be modified as well... or maybe not? Anyhow the tests fails.... due to missing folders/files.

The test suite is working on local machines. Something weird is happening with github actions.

[srggrs]

I think this is ready to go, one minor comment on readability of the imports in the regression tests, but not really important.

Thanks for putting this together!

Please use the squash and merge option when merging, as this is a massive PR

[srggrs]

LGTM