You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
When examples_per_shard == 1 and a restart happens there is no way to tell if the previous shard was already captured and saved or not. Also the scaaml.io.Dataset or DatasetFiller are not equipped to handle saving single example multiple times. To make this reliable there needs to be a way to say "shard saved".
The text was updated successfully, but these errors were encountered:
When examples_per_shard == 1 and a restart happens there is no way to tell if the previous shard was already captured and saved or not. Also the scaaml.io.Dataset or DatasetFiller are not equipped to handle saving single example multiple times. To make this reliable there needs to be a way to say "shard saved".
The text was updated successfully, but these errors were encountered: