Catch exception raised in hf prep properly

[j316chuck]

Description

Issue: There is a barrier deadlock in build_from_hf if an Exception is raised which causes errored runs to not fail properly and burn compute.

Concretely, the non local rank 0 processes depend on a signal file to be written but if a rank 0 process raises an exception then we never reach the dist.barrier() state resulting in a deadlock.

Solution: We add a try catch around the dataset preparation code to ensure that local rank 0 gets to writing/removing the signal file enabling all processes to reach the dist.barrier() state.

Tests:

python3   -m composer.cli.launcher -n 2 --master_port 26002   -m pytest -v --durations=20 -m 'gpu' -o tmp_path_retention_policy=none  -k "test_finetuning_dataloader_small"

=============================================================================================================== slowest 20 durations ================================================================================================================
3.69s call     tests/test_dataloader.py::test_finetuning_dataloader_small_data[True-True-2-4]
3.09s call     tests/test_dataloader.py::test_finetuning_dataloader_small_data[False-True-4-8]
3.05s call     tests/test_dataloader.py::test_finetuning_dataloader_small_data[False-False-2-8]
2.95s call     tests/test_dataloader.py::test_finetuning_dataloader_small_data[False-True-2-8]
2.74s call     tests/test_dataloader.py::test_finetuning_dataloader_small_data[True-False-2-8]
2.73s call     tests/test_dataloader.py::test_finetuning_dataloader_small_data[False-False-4-8]
2.63s call     tests/test_dataloader.py::test_finetuning_dataloader_small_data[True-True-4-8]
2.63s call     tests/test_dataloader.py::test_finetuning_dataloader_small_data[True-True-2-8]
2.56s call     tests/test_dataloader.py::test_finetuning_dataloader_small_data[False-True-2-4]
2.55s call     tests/test_dataloader.py::test_finetuning_dataloader_small_data[True-False-2-4]
2.50s call     tests/test_dataloader.py::test_finetuning_dataloader_small_data[False-True-4-4]
2.49s call     tests/test_dataloader.py::test_finetuning_dataloader_small_data[True-False-4-8]
2.43s call     tests/test_dataloader.py::test_finetuning_dataloader_small_data[True-False-4-4]
2.36s call     tests/test_dataloader.py::test_finetuning_dataloader_small_data[False-False-2-4]
2.32s call     tests/test_dataloader.py::test_finetuning_dataloader_small_data[True-True-4-4]
2.27s call     tests/test_dataloader.py::test_finetuning_dataloader_small_data[False-False-4-4]
1.97s setup    tests/test_dataloader.py::test_finetuning_dataloader_small_data[True-True-2-4]
0.24s setup    tests/test_dataloader.py::test_finetuning_dataloader_small_data[True-False-2-4]
0.24s setup    tests/test_dataloader.py::test_finetuning_dataloader_small_data[False-True-2-4]
0.24s setup    tests/test_dataloader.py::test_finetuning_dataloader_small_data[False-True-4-8]
================================================================================================= 16 passed, 1030 deselected, 24 warnings in 48.55s =================================================================================================

Also ported the test_finetuning_dataloader_small to GPU tests since it wasn't being ran

[dakinggg]

Also, lets add a test for this. world size 2 with a broken jsonl file should reproduce for the test I think?