Emit structured log events to ApplicationInsights

[TomAugspurger]

Description

This adds some structured logging to Application Insights for various events ("thing" Created / Finished for Workflow Run, Job, Job Partition, and Task).

Fixes # (issue)

Type of change

Please delete options that are not relevant.

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• This change requires a documentation update

How Has This Been Tested?

Deployed to staging. See #308 (comment) for some sample results.

[TomAugspurger]

Test failure was

Job test-job failed. Status: failed
 -- Job partitions thread failed with update_job_partition_run_status() missing 4 required positional arguments: 'workflow_id', 'dataset_id', 'job_id', and 'partition_id'
--------------------------- Captured stdout teardown ---------------------------

I'm surprised that mypy didn't catch that.

[TomAugspurger]

I've deployed this to staging. Here are some sample logs:

WorkflowRun

timestamp	severityLevel	type	recordLevel	datasetId	level	runId
2024-07-10T18:53:35.21652Z	1	WorkflowRunCreated	WorkflowRun	test-bug	INFO	3392a265-43a3-40bb-a493-d22afe7c427d
2024-07-10T18:59:10.174577Z	2	WorkflowRunFinished	WorkflowRun	test-bug	WARNING	3392a265-43a3-40bb-a493-d22afe7c427d

Job

timestamp	severityLevel	type	recordLevel	datasetId	level	runId	jobId	partitionId	status
2024-07-10T18:53:35.276653Z	1	JobCreated	Job	test-bug	INFO	3392a265-43a3-40bb-a493-d22afe7c427d	a	null	running
2024-07-10T18:59:10.005458Z	2	JobFinished	Job	test-bug	WARNING	3392a265-43a3-40bb-a493-d22afe7c427d	a	null	failed
2024-07-10T19:49:40.634756Z	1	JobCreated	Job	test-bug	INFO	6cb80e84-18d7-4f6e-86e4-fc7415823eaa	a	null	running

Task

timestamp	severityLevel	type	recordLevel	datasetId	level	runId	jobId	partitionId	taskId	status	errors
2024-07-10T19:49:43.284043Z	1	TaskCreated	Task	test-bug	INFO	6cb80e84-18d7-4f6e-86e4-fc7415823eaa	a	0	b	submitted	null
2024-07-10T19:54:15.014362Z	2	TaskFinished	Task	test-bug	WARNING	6cb80e84-18d7-4f6e-86e4-fc7415823eaa	a	0	b	failed	The task exited with an exit code representing a failure

Unfortunately they errors field isn't too helpful. This was for a workflow that deliberately failed with a ZeroDivisionError. I think "The task exited with an exit code representing a failure" is all that's available from the Batch API.

---

I'm not sure why JobPartition isn't showing up, but IMO that's much less valuable than WorkflowRun and Tasks.