Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add option to list data files directly in configuration file #40

Closed
aribrill opened this issue Jul 19, 2018 · 0 comments · Fixed by #117
Closed

Add option to list data files directly in configuration file #40

aribrill opened this issue Jul 19, 2018 · 0 comments · Fixed by #117
Labels
enhancement

Comments

@aribrill
Copy link
Collaborator

Since YAML provides a convenient way to include lists directly in the config file, allow Data:file_list to accept either a path to a file containing file paths (the current method) or a list of file paths written directly in the config file.
aribrill added a commit that referenced this issue May 1, 2019
@aribrill
Refactor run_model and configuration to use DL1DH
ea9916d 
Replace DataLoader and ImageMapper with DL1DataReader and use Transforms
to replace DataProcessor. The config file has been rewritten to match
the DL1DH API. In addition, the input_fn and associated config arguments
have been cleaned up to fix #41. In fact, because DL1DH now returns the
examples directly, there is no need to use map at all and prefetching
should be solely relied on for parallelizing the data input pipeline. In
addition, this commit incidentally fixes #40 to allow data files to be
listed directly in the config file.
@aribrill aribrill mentioned this issue May 3, 2019
Merged
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Integrate DL1DataHandler ctlearn-project/ctlearn
1 participant
@aribrill