[cmd/opampsupervisor]: Supervisor waits for configurable healthchecks to report remote config status

[srikanthccv]

Description:

This pull request addresses the remote config status reporting issue discussed in #21079. Currently, there's no way to determine if the agent terminated due to an incorrect configuration. While we can use the dry-run option, it only performs basic validation. This update introduces the following options to the Agent section of the supervisor config:

1. successful_health_checks: The number of consecutive successful checks needed before reporting an APPLIED status.
2. config_apply_timeout: If neither success nor failure is reported within this time frame, a FAILED status is reported.

Link to tracking Issue: #21079

Testing: Added e2e test

Documentation:

[github-actions]

This PR was marked stale due to lack of activity. It will be closed in 14 days.

[github-actions]

Closed as inactive. Feel free to reopen if this PR is still being worked on.

[BinaryFissionGames]

@srikanthccv I don't think I saw this was open 😞 - Could you re-open this/make a new PR? I'll give it a review.

[Asarew]

I don't think that while the supervisor is applying the new config it should report Healthy = false. We should report RemoteConfigStatus.Status = APPLYING together with Healthy = true as it's following a desired flow and nothing unusual is happening.

[srikanthccv]

@BinaryFissionGames, I have been meaning to submit a new PR but have had a tight work schedule for quite some time. I promise to get back to this soon.

[tigrannajaryan]

I don't think that while the supervisor is applying the new config it should report Healthy = false. We should report RemoteConfigStatus.Status = APPLYING together with Healthy = true as it's following a desired flow and nothing unusual is happening.

I would suggest a slight modification from this logic. While applying the new config the Supervisor should continue reporting the value of Healthy as it was known before we started applying the new config. If the agent was previously healthy then keep reporting it is healthy, if it was unhealthy keep reporting it is unhealthy, until we get a new health status.

I agree we should report RemoteConfigStatus.Status = APPLYING.