API driven noticed errors display as expected: true, and observed errors should too

[fallwith]

Problem: Observed exceptions are reported as expected: false even when they are treated as expected by the agent.

Explanation: There are 2 ways to inform the agent about an error. The public notice_error API can be called with an optional boolean based expected option, or the agent can observe an exception being raised by the Ruby application it monitors.

One a noticed error is in play, its origin does not matter when it comes to the logic the agent uses to process the error. When it comes to determining if the error is expected, the agent will make use of the :'error_collector.expected_classes', :'error_collector.expected_messages', and :'error_collector.expected_status_codes' configuration settings. And once an error is considered expected, that error won't negatively impact apdex. There will be no difference in agent behavior for API driven errors or observed errors.

But when it comes to reporting the error to the New Relic errors inbox, the origin of the error does make a difference when it comes to the expected attribute on the noticed error. API driven noticed errors that had the expected: true key/value pair present at API method call time will have that key/value pair persist all the way through to the errors inbox UI. But observed errors - despite effectively being treated as expected by the agent - will show up as expected: false in the UI.

From my interpretation of our agent specs, particularly the errors guide, I feel that this difference between API driven errors and observed errors is an oversight and therefore a bug. I believe we should set the expected: false key/value pair on observed errors once we have conditionally determined the error to be expected and before we report the error to the errors inbox.

Definition of done: if an ActiveRecord::RecordNotFound error is raised and results in an HTTP 404 status for a web request and the 404 code exists in the :'error_collector.expected_status_codes' configuration array, then the instance of the ActiveRecord::RecordNotFound error that appears in the errors inbox should display an attribute of "expected" with a value of "true".

[workato-integration]

https://new-relic.atlassian.net/browse/NR-279908

[fallwith]

Upon further inspection, it looks like most observed errors DO in fact set expected: true on the noticed error.

It is only those observed errors that are behaviorally treated as expected as a result of their HTTP status code that do not receive the expected: true setting because the HTTP status code is not in scope when the expected setting is updated.

This will require a smaller fix than originally thought.