Dynamic Dataset creation form #210
Conversation
Fixing logic where pip installs the updated packages (to avoid the entire image rebuild), so make sure deps are ignored (as this was the slow part).
Adding type GET_SERIALIZED_FORM to get the entire serialized state of a dataset.
Refactor to ensure it takes advantage of connection handling via its async context manager logic.
Add new function to get serialized dataset details to DatasetExternalClient.
Fix issue with not acquiring a session, and wrapping some things in try block to catch and log exceptions.
Fixing bug in response to LIST_FILES query, where reason text was not being assembled entirely correctly.
Adding optional offset and length params to abstract interface definition of get_data, and adding get_file_stat abstract method.
Adding support for querying about dataset items.
Updating to support indicating start and size of data for partial transfers.
Optimizing the reloading of datasets on object store manager startup.
Accounting for changes to get_data parameters and implementing get_file_stat.
Updating service to respond to GET_DATASET_ITEMS queries and requests for data with an offset start (i.e. partials).
…d label's interact
…mit when display is set to none
|
@robertbartel, do either the catchment id's or data id's have an expected format (thinking regex)? I ask so that I could add the expected patterns to their respective form fields. |
|
@aaraney, the catchment ids probably do: The data ids probably do not. I have basically been using the dataset name as the data id values, although that's for human easy of use and not necessarily going to continue once things are more heavily utilized. It occurs to me that (object store) dataset names actually can't contain underscore characters, because their correspondence to bucket names in MinIO and MinIO's bucket name rules. |
Thanks for clarifying, I am going to take that to mean we should hold off restricting "valid" catchment ids, at least for now.
👍. It seems like in the future dataset id's might be something we will generate ourselves. Is that fair?
Cool cool. I will add filename restriction to avoid that. |
|
I added filename and start / end time validation, meaning you cannot submit the form without these fields set properly. Invalid form entry shows meaningful error messages so the person can make corrections. |
aaraney
changed the title
|
Closing; consolidating these and other changes in #240. |
closes #198
Once this is done, it will need to be merged after #197.