Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tracking file provenance #3711

Open
astro-friedel opened this issue Dec 2, 2024 · 0 comments · May be fixed by #3712
Open

Tracking file provenance #3711

astro-friedel opened this issue Dec 2, 2024 · 0 comments · May be fixed by #3712

Comments

@astro-friedel
Copy link

Is your feature request related to a problem? Please describe.
I would like to have the ability to track file provenance for each files used (input) or created (output) by an App. The provenance information should include:

  • File name
  • Creation date
  • File size
  • What App created it (if it doesn't already exist)
    • What arguments were given to the App
    • What environment was the App running in
  • What other Apps used the file

Describe the solution you'd like
The system would need to be able to track files (already does), and capture information about them when they are created, or used for the first time, then track their usage through the rest of the workflow. Ideally, this should require minimal changes to existing workflows. Using the existing monitoring framework is a good candidate as it already has the ability to log information to a database.

Describe alternatives you've considered
The only alternatives I have come up with are manually tracking the input and output files (not too bad for a small workflow, but very laborious, if not impossible for large workflows), and adding code in each App to monitor files created (but this would require also creating infrastructure that is alredy provided by the monitoring framework).

Additional context
It would be nice to be able to easily access the provenance information, perhaps through the parsl-visualizer.

@astro-friedel astro-friedel linked a pull request Dec 2, 2024 that will close this issue
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant