Tracking file provenance

[astro-friedel]

Description

This PR introduces the ability to track file provenance to Parsl. The provenance information includes:

• File name
• Creation date
• File size
• md5 sum
• What App created it (if it doesn't already exist)
  • What arguments were given to the App
  • What environment was the App running in
• What other Apps used the file

This feature uses the existing monitoring infrastructure to capture the requisite information and store it in a databse. The file provenance information can also be accessed via the parsl-visualizer interface. An additional keyword argument for the MonitoringHub has been added capture_file_provenance to enable/disable the provenance tracking. By default it is False, turning off the provenance tracking.

While this type of file information can be tracked manually for smaller workflows, larger workflows require an automated solution, like this one.

Changed Behaviour

In general, users will not change Parsl's behavior, as the provenance tracking code is turned off by default. Enabling it will only gather additional information in the background. The only change users should outwardly see is the workflow slowdown typically seen when unsing the monitoring framework.

Fixes

This fixes #3711

Type of change

• New feature
• Update to human readable text: Documentation/error messages/comments