Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature: make allow file format lists configurable #60

Closed
sallain opened this issue Oct 2, 2024 · 4 comments · Fixed by #61
Closed

Feature: make allow file format lists configurable #60

sallain opened this issue Oct 2, 2024 · 4 comments · Fixed by #61

Comments

@sallain
Copy link
Contributor

sallain commented Oct 2, 2024

Is your feature request related to a problem? Please describe.

The allow file format list is currently hardcoded. Anyone who wants to use this feature will have their own list of allowed formats, so they would have to change the code to suit their requirements. This also puts the list at risk during upgrades/redeployments as it may be reset to the hardcoded default.

Describe the solution you'd like

Make the allow file format list configurable.

In a short discussion with @djjuhasz, he suggested we could either use the enduro.toml config file, or by adding dedicated config files - with a dedicated config file, we could use CSVs instead of TOML, which would be more user-friendly. Open to any option, however!

Describe alternatives you've considered

Keep it as is and cry every time a user forgets to reset their list.

Additional context

@djjuhasz djjuhasz self-assigned this Oct 3, 2024
@djjuhasz djjuhasz transferred this issue from artefactual-sdps/temporal-activities Oct 4, 2024
@djjuhasz
Copy link
Contributor

djjuhasz commented Oct 4, 2024

@sallain I moved this issue from the https://github.com/artefactual-sdps/temporal-activities repository because the "validate-file-format" activity is currently local to preprocessing-sfa. I do think we should move "validate-file-format" to the "temporal-activities" repo so it can be used in other Temporal projects, but it will be easier to do the configuration development here and then move the activity afterwards.

@djjuhasz djjuhasz moved this from 👍 Ready to ⏳ In Progress in Enduro Oct 4, 2024
@djjuhasz
Copy link
Contributor

djjuhasz commented Oct 4, 2024

@sallain we currently only have an "allowed file format" list — any format that is not on the list is disallowed. Should I add the ability to have a "disallowed file format" list as well? If we do add the disallow list, what should happen if a format is not on either list?

@sallain
Copy link
Contributor Author

sallain commented Oct 4, 2024

Thank you @djjuhasz!

No need for a disallow list at this time - I'll edit the issue. My bad.

@sallain sallain changed the title Feature: make allow/disallow file format lists configurable Feature: make allow file format lists configurable Oct 4, 2024
djjuhasz added a commit that referenced this issue Oct 4, 2024
djjuhasz added a commit that referenced this issue Oct 8, 2024
Fixes #60

- Add a "fileFormat" section to the preprocessing config file
- Add a "AllowlistPath" config option to the "fileFormat" section
- If no "AllowlistPath" is configured, then file format validation will
  be skipped
- Add an allowed format CSV file to the dev and enduro kube configs
@djjuhasz djjuhasz linked a pull request Oct 8, 2024 that will close this issue
djjuhasz added a commit that referenced this issue Oct 8, 2024
Fixes #60

- Add a "fileFormat" section to the preprocessing config file
- Add a "AllowlistPath" config option to the "fileFormat" section
- If no "AllowlistPath" is configured, then file format validation will
  be skipped
- Add an allowed format CSV file to the dev and enduro kube configs
djjuhasz added a commit that referenced this issue Oct 8, 2024
Fixes #60

- Add a "fileFormat" section to the preprocessing config file
- Add a "AllowlistPath" config option to the "fileFormat" section
- If no "AllowlistPath" is configured, then file format validation will
  be skipped
- Add an allowed format CSV file to the dev and enduro kube configs
djjuhasz added a commit that referenced this issue Oct 8, 2024
Fixes #60

- Add a "fileFormat" section to the preprocessing config file
- Add a "AllowlistPath" config option to the "fileFormat" section
- If no "AllowlistPath" is configured, then file format validation will
  be skipped
- Add an allowed format CSV file to the dev and enduro kube configs
djjuhasz added a commit that referenced this issue Oct 8, 2024
Fixes #60

- Add a "fileFormat" section to the preprocessing config file
- Add a "AllowlistPath" config option to the "fileFormat" section
- If no "AllowlistPath" is configured, then file format validation will
  be skipped
- Add an allowed format CSV file to the dev and enduro kube configs
djjuhasz added a commit that referenced this issue Oct 9, 2024
Fixes #60

- Add a "fileFormat" section to the preprocessing config file
- Add a "AllowlistPath" config option to the "fileFormat" section
- If no "AllowlistPath" is configured, then file format validation will
  be skipped
- Add an allowed format CSV file to the dev and enduro kube configs
djjuhasz added a commit that referenced this issue Oct 11, 2024
Fixes #60

- Add a "fileFormat" section to the preprocessing config file
- Add a "AllowlistPath" config option to the "fileFormat" section
- If no "AllowlistPath" is configured, then file format validation will
  be skipped
- Add an allowed format CSV file to the dev and enduro kube configs
djjuhasz added a commit that referenced this issue Oct 15, 2024
Fixes #60

- Add a "fileFormat" section to the preprocessing config file
- Add a "AllowlistPath" config option to the "fileFormat" section
- If no "AllowlistPath" is configured, then file format validation will
  be skipped
- Add an allowed format CSV file to the dev and enduro kube configs
@github-project-automation github-project-automation bot moved this from ⏳ In Progress to 🎉 Done in Enduro Oct 15, 2024
@sallain
Copy link
Contributor Author

sallain commented Oct 30, 2024

@scollazo tested this with the following note: "I tested by removing Jp2000 from the list of allowed formats, launching a sip (it fails with the proper error/path), updating the list again, launching the same sip, and it works. no restart needed!"

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Archived in project
Development

Successfully merging a pull request may close this issue.

2 participants