Add flag to write job reports to disk

[mcastorina]

Description:

Adds a hook to the SourceManager to track unit progress and write the finished metrics to a file.

Example output of source that supports units:

{
  "data": {
    "unit": {
      "source_unit_id": "/tmp/secretsandstuff"
    },
    "parent": {
      "job_id": 1,
      "source_id": 1,
      "source_name": "trufflehog - filesystem"
    },
    "start_time": "2024-01-11T16:16:27.684967-08:00",
    "end_time": "2024-01-11T16:16:27.687357-08:00",
    "total_chunks": 3,
    "total_bytes": 365,
    "errors": []
  },
  "version": 1
}
{
  "data": {
    "unit": {
      "source_unit_id": "/tmp/not-found"
    },
    "parent": {
      "job_id": 1,
      "source_id": 1,
      "source_name": "trufflehog - filesystem"
    },
    "start_time": "2024-01-11T16:16:27.684982-08:00",
    "end_time": "2024-01-11T16:16:27.685008-08:00",
    "total_chunks": 0,
    "total_bytes": 0,
    "errors": [
      "error chunking unit \"/tmp/not-found\": unable to get file info: stat /tmp/not-found: no such file or directory"
    ]
  },
  "version": 1
}

Example output of source that does not support units:

{
  "data": {
    "parent": {
      "job_id": 1,
      "source_id": 1,
      "source_name": "trufflehog - github"
    },
    "start_time": "2024-01-11T16:17:09.050788-08:00",
    "end_time": "2024-01-11T16:17:26.966082-08:00",
    "total_chunks": 63609,
    "total_bytes": 386489349,
    "errors": []
  },
  "version": 1
}

Checklist:

• Tests passing (make test-community)?
• Lint passing (make lint this requires golangci-lint)?

[mcastorina]

I'm not fully convinced this is a good solution and am open to alternatives.

Another idea is to spawn a goroutine to periodically flush the hook and look for metrics that are finished to write out.

[rosecodym]

What don't you like about it?

[mcastorina]

It's a specific option for this specific hook. If there are other hooks that similarly need to be flushed, the solution doesn't scale. I think that's acceptable for now since there's just this one hook, but a system that knows about a Flush method and calls it might be nice.

[rosecodym]

I think that keeping things simple is fine for now. If/when we add more flushable hooks we might have a better sense of what would be a good framework for them.

[mcastorina]

This is Hidden for now so we can change it without breaking users.

[rosecodym]

What don't you like about it?

[rosecodym]

i have some questions/suggestions but nothing blocking

[rosecodym]

logr.Discard() here feels like a code smell. why don't we want the hook to log?

[rosecodym]

I think that keeping things simple is fine for now. If/when we add more flushable hooks we might have a better sense of what would be a good framework for them.

going to change failure mode approach

[rosecodym]

i love this!