Ad-hoc tasks remain as "running" when they timeout

[mikecote]

I noticed if you make an alerting rule's action run past the 5m default timeout, it remains as a task within the .kibana_task_manager index with a running status indefinitely.

Sample task document taken after running for approx 8m

{
        "_index" : ".kibana_task_manager_8.1.0_001",
        "_id" : "task:6f5f69e0-4d4c-11ec-9951-179550661734",
        "_score" : 5.7254505,
        "_source" : {
          "migrationVersion" : {
            "task" : "8.0.0"
          },
          "task" : {
            "taskType" : "actions:.server-log",
            "retryAt" : null,
            "runAt" : "2021-11-24T17:32:02.302Z",
            "scope" : [
              "actions"
            ],
            "traceparent" : "00-1c0305d3e0590dfa0e8f7253f1010df2-a671ece938c0abd1-01",
            "startedAt" : "2021-11-24T17:32:04.856Z",
            "state" : "{}",
            "params" : "{\"spaceId\":\"default\",\"actionTaskParamsId\":\"6f287b60-4d4c-11ec-9951-179550661734\"}",
            "ownerId" : "kibana:5b2de169-2785-441b-ae8c-186a1936b17d",
            "scheduledAt" : "2021-11-24T17:32:02.302Z",
            "attempts" : 1,
            "status" : "running"
          },
          "references" : [ ],
          "updated_at" : "2021-11-24T17:32:04.908Z",
          "coreMigrationVersion" : "8.1.0",
          "type" : "task"
        }
      }

I do however see an attempt at canceling the task, but the task doesn't implement a cancel function.

[2021-11-24T12:37:04.899-05:00][WARN ][plugins.taskManager] Cancelling task actions:.server-log "6f5f69e0-4d4c-11ec-9951-179550661734" as it expired at 2021-11-24T17:37:04.856Z after running for 05m 00s (with timeout set at 5m).

[elasticmachine]

Pinging @elastic/kibana-alerting-services (Team:Alerting Services)