[Fleet]: After cancelling the Request for Schedule Upgrade, Upgrade scheduled label is not removed from the Agents tab.

[harshitgupta-qasource]

Kibana Build details:

VERSION: 8.12.1
BUILD: 70228
COMMIT: 3457f326b763887d154c9da00bd4e489221a2ff3

Host OS and Browser version: All, All

Preconditions:

1. 8.12.1 Kibana Cloud environment should be available.
2. Policy should be created.
3. 8.12.1 agent should be deployed
4. Endpoint security should be added to policy.

Steps to reproduce:

1. Navigate to Fleet Tab and select the Agent
2. Schedule upgrade using API for 8.12.1 agent to 8.12.2.
3. Now cancel the schedule upgrade from agent acitvity.
4. Observe that "Upgrade scheduled" label is still visible.

Expected:
After cancelling the Request for Schedule Upgrade, Upgrade scheduled label should be removed from the Agents tab.

Screencast:

Agents.-.Fleet.-.Elastic.Mozilla.Firefox.2024-02-20.17-15-05.mp4

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[harshitgupta-qasource]

@amolnater-qasource Kindly review

[amolnater-qasource]

Secondary Review for this ticket is Done.

[jlind23]

@amolnater-qasource @harshitgupta-qasource is this a new bug you found in 8.12 or something that was already existing?

[harshitgupta-qasource]

Hi @jlind23

While testing the elastic/kibana#168502 feature on 8.12.1 and then attempting to upgrade the agent via scheduled upgrade to 8.12.2, we have discovered this issue.

[juliaElastic]

Should this issue be moved to elastic-agent repo? I didn't find any logic in fleet-server regarding cancel action.
Though I found that kibana itself updates agent docs to clear the Updating state here, so probably we can clear upgrade_details here too.
Can we confirm that cancelled action is not executed at the scheduled time? (to confirm if the bug doesn't impact agent).

Tested this, and I can confirm that the agent cancelled the action, so the bug is that the agent is stuck in upgrade scheduled state, but doesn't upgrade.

14:58:03.876
elastic_agent
[elastic_agent][info] Cancel action id: 551f9e9c-9de0-498f-81ae-da36046569ab target id: f701dd74-49c5-4a6c-a241-ea403ddb1589 removed 1 action(s) from queue.

cc @cmacknz @kpollich

[juliaElastic]

I tried to fix locally by clearing the upgrade_details of the agent doc in kibana when the cancel API is called. It doesn't seem to work, because the upgrade_details with scheduled state keeps coming back on every checkin. So I think this has to be fixed on agent side.

[cmacknz]

The Cancel action id: seen above are coming from

elastic-agent/internal/pkg/agent/application/actions/handlers/handler_action_cancel.go

Lines 34 to 44 in bdd885c

	// Handle will cancel any actions in the queue that match target_id.
	func (h *Cancel) Handle(ctx context.Context, a fleetapi.Action, acker acker.Acker) error {
	action, ok := a.(*fleetapi.ActionCancel)
	if !ok {
	return fmt.Errorf("invalid type, expected ActionCancel and received %T", a)
	}
	n := h.c.Cancel(action.TargetID)
	if n == 0 {
	h.log.Debugf("Cancel action id: %s target id: %s found no actions in queue.", action.ActionID, action.TargetID)
	return nil
	}

The action queue is registered as the canceller in

elastic-agent/internal/pkg/agent/application/managed_mode.go

Lines 380 to 387 in bdd885c

	m.dispatcher.MustRegister(
	&fleetapi.ActionCancel{},
	handlers.NewCancel(
	m.log,
	m.actionQueue,
	),
	)

The actual cancel implementation is in

elastic-agent/internal/pkg/queue/actionqueue.go

Lines 136 to 149 in bdd885c

	// Cancel will remove any actions in the queue with a matching actionID and return the number of entries cancelled.
	// Complexity: O(n*log n)
	func (q *ActionQueue) Cancel(actionID string) int {
	items := make([]*item, 0)
	for _, item := range *q.q {
	if item.action.ID() == actionID {
	items = append(items, item)
	}
	}
	for _, item := range items {
	heap.Remove(q.q, item.index)
	}
	return len(items)
	}

There is no notification that an upgrade is cancelled when it is removed from the queue like this, so I think we will stay in the upgrade scheduled state until an upgrade is eventually completed by a separate action. We likely need to update this cancel implementation to have some special handling for upgrade actions, specifically it needs to clear the upgrade details.

The upgrade is marked as scheduled in

elastic-agent/internal/pkg/agent/application/dispatcher/dispatcher.go

Lines 358 to 361 in bdd885c

	upgradeDetails := details.NewDetails(
	nextUpgrade.Version,
	details.StateScheduled,
	nextUpgrade.ID())

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)