You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
We should skip the prepare_data step in our data modules unless downloading is actually required.
Rationale
Instantiating the dataset is slow. Currently, all of our datamodules instantiate the dataset at least twice on the off chance that someone is requesting the dataset to be downloaded. This isn't needed most of the time.
The solution is to replace the prepare_data() method with the code above in all of our data modules. This code will check the "download" keyword argument and only download the dataset if it is set to true. This will significantly reduce the time required to instantiate the dataset and speed up our data modules.
Summary
We should skip the
prepare_data
step in our data modules unless downloading is actually required.Rationale
Instantiating the dataset is slow. Currently, all of our datamodules instantiate the dataset at least twice on the off chance that someone is requesting the dataset to be downloaded. This isn't needed most of the time.
Implementation
A better solution would be to replace:
with this:
in all of our data modules.
Alternatives
No response
Additional information
No response
The text was updated successfully, but these errors were encountered: