Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Skip dirs in parse_directory #478

Merged
merged 1 commit into from
Oct 18, 2024
Merged

Skip dirs in parse_directory #478

merged 1 commit into from
Oct 18, 2024

Conversation

aulemahal
Copy link
Collaborator

Pull Request Checklist:

  • This PR addresses an already opened issue (for bug fixes / features)
    • This PR fixes #xyz
  • (If applicable) Documentation has been added / updated (for bug fixes / features).
  • (If applicable) Tests have been added.
  • This PR does not seem to break the templates.
  • CHANGELOG.rst has been updated (with summary of main changes).
    • Link to issue (:issue:number) and pull request (:pull:number) has been added.

What kind of change does this PR introduce?

  • parse_directory : New argument skip_dirs, a list of folders to not go into when parsing.

Does this PR introduce a breaking change?

No.

Other information:

The MRCC5 catalog (him again) has two issues this help fix:

  • Some simulations are copied in two sub-folders of the list of directories to parse. This is because I want to give some time to people to adapt to the new disk. The entries are exactly the same except for the path, so there's no way to filter them out currently with the other arguments of parse_directory.
  • There are some many files.... This skipping will help the parsing time by skipping folders I know are garbage and would be rejected anyway by other filters.

@aulemahal aulemahal requested a review from RondeauG October 18, 2024 15:40
Copy link
Collaborator

@RondeauG RondeauG left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@aulemahal aulemahal merged commit 937326f into main Oct 18, 2024
17 checks passed
@aulemahal aulemahal deleted the parse-dir-skip branch October 18, 2024 15:52
@aulemahal aulemahal mentioned this pull request Oct 21, 2024
6 tasks
aulemahal added a commit that referenced this pull request Oct 21, 2024
<!-- Please ensure the PR fulfills the following requirements! -->
<!-- If this is your first PR, make sure to add your details to the
AUTHORS.rst! -->
### Pull Request Checklist:
- [ ] This PR addresses an already opened issue (for bug fixes /
features)
    - This PR fixes #xyz
- [ ] (If applicable) Documentation has been added / updated (for bug
fixes / features).
- [ ] (If applicable) Tests have been added.
- [ ] This PR does not seem to break the templates.
- [x] CHANGELOG.rst has been updated (with summary of main changes).
- [x] Link to issue (:issue:`number`) and pull request (:pull:`number`)
has been added.

### What kind of change does this PR introduce?

Small fix upon #478. Skips even if a directory to skip is one of the
entries in `directory` or a parent of one of the entries.
I implemented it because I misunderstood an error I was having, but
still, this could be used in the specific scenario where you are running
`parse_directory` through an incremental stack of xscen configurations.
To remove en entry of the `directories` list of a previous
configuration, you would only need to add a `skip_dirs` entry, and not
rewrite the filtered `directories`.

This also adds logging calls when skipping. And a precision to the doc
about the absoluteness of paths to skip.

### Does this PR introduce a breaking change?
No.

### Other information:
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants