Implement a mechanism that to enable a NOS running SAI and SDK, to detect SDK and FW failures

[amandyuk]

Add new health event, for SAI to notify severity, timestamp and description of a SDK/FW health event

Signed-off-by: Andrii Mandiuk [email protected]

[mikeberesford]

@JaiOCP this looks to be partially overlapping with #1307

[rlhui]

Can we please add example usage?

[rlhui]

Intel/ashutosh: no way for nos to subscribe to specific ones they're interested in

[rlhui]

mike: this seems to take away control from NOS.

[rlhui]

better if nos doesn't need to parse string but the info is in a defined format. Please also checkout https://github.com/sonic-net/SONiC/blob/5677242770e3774a5eab0f951dc14642d81422ec/doc/event-alarm-framework/events-producer.md

[itaibaz]

Community discussion comments :

1. Add category, to allow the NOS to act based on category and not have to parse the free text description
   Review
   https://github.com/sonic-net/SONiC/blob/5677242770e3774a5eab0f951dc14642d81422ec/doc/event-alarm-framework/events-producer.md

2. Add a knob, to control the minimum severity of event NOS wants to receive, for example receive just warning or above

3. Consider rate limitation, so NOS isn't flooded with events

[reshmaintel]

@itaibaz I like this proposal that this SDK failure detection and reporting is generic and gives the control to the vendor to push specific failure logs with related severity that help user to debug the given vendor SDK/HW. I think that if this feature is only severity level based it may resort to being an alternative to syslog. If there is additional categorization for the failures it may be more valuable to the user to monitor certain categories. Certain broad categories can be suggested. Given that using enums for categories seem restrictive and not having to add new enums for additional categories is one of your goals, please explore if we can make it extensible by adding the category as a field just like the description field.

[itaibaz]

opened #1777 instead