[LintDiff] AggregateError is not showing constituent errors. Original bug report: error encountered: Error: Error running Nimma

[konrad-jamrozik]

• https://dev.azure.com/azure-sdk/internal/_build/results?buildId=3024142&view=logs&j=688669d0-441c-57c3-cf6d-f89a22ccfa5d&t=b91b1e88-b042-5e18-36d8-34e4fb3a9b3b&l=427
• Teams discussion
• https://github.com/Azure/azure-rest-api-specs-pr/pull/14406/checks?check_run_id=16112028648

Update 9/20/2023

Known cases from Teams discussion:

• https://github.com/Azure/azure-rest-api-specs-pr/pull/14727
  • https://dev.azure.com/azure-sdk/internal/_build/results?buildId=3088336&view=logs&j=0574a2a6-2d0a-5ec6-40e4-4c6e2f70bea2&t=80c3e782-49f0-5d1c-70dd-cbee57bdd0c7
• ARG Query Generation API in 2023-09-01-preview azure-rest-api-specs#25699
  • https://dev.azure.com/azure-sdk/internal/_build/results?buildId=3088163&view=logs&j=0574a2a6-2d0a-5ec6-40e4-4c6e2f70bea2&t=80c3e782-49f0-5d1c-70dd-cbee57bdd0c7
• Adding patching scenario ARM API azure-rest-api-specs#25446
  • https://dev.azure.com/azure-sdk/internal/_build/results?buildId=3096221&view=logs&j=0574a2a6-2d0a-5ec6-40e4-4c6e2f70bea2&t=80c3e782-49f0-5d1c-70dd-cbee57bdd0c7

Getting more info from the error

Investigation trail, chronologically:

• Improve logging output from Nimma when a custom function errors stoplightio/spectral#2053
  • https://github.com/stoplightio/spectral/blob/develop/packages/cli/src/commands/lint.ts
  • Getting an AggregateError when using JSON custom ruleset stoplightio/spectral#2242
    • Resolve mac agent failure #2242 has tip on logging errors property of the aggregate error

Where the AggregateError.errors property is defined?

• We can see based on example source code of spectral that the AggregateError comes from es-aggregate-error.

About Spectral and Nimma

Per this doc:

Spectral uses nimma as its main implementation, and sometimes resorts to jsonpath-plus to ensure backwards-compatibility.

• https://github.com/P0lip/nimma
• https://docs.stoplight.io/docs/spectral/eb68e7afd463e-spectral-in-java-script
• https://docs.stoplight.io/docs/spectral/f8921361d3f05-error-handling

[konrad-jamrozik]

Conclusion

OK, so the problem is as follows: the error error encountered: Error: Error running Nimma is in fact an es-aggregate-error thrown from Spectral (see above for details). LintDiff currently is not printing out the details of the errors aggregated in it. This PR I just opened fixes this:

• Print out details of AggregateError originating from Spectral (@microsoft.azure/openapi-validator v2.1.4 -> v2.1.5) azure-openapi-validator#595

Looks to me that Spectral is using nimma to query JSON elements (a'la JSONPath) and runs rules from our spectral ruleset on them. If they fail with exception, it results in such aggregate exception, hiding the details of the failures.

Bug in patchPropertiesCorrespondToPutProperties rule uncovered during my local repro

My local repro of the problem on the PR:

• Adding patching scenario ARM API azure-rest-api-specs#25446

with a call like this one:

autorest --v3 --spectral --azure-validator --use=./packages/azure-openapi-validator/autorest --tag=package-preview-2023-08 <local clone of azure-openapi-validator>/specs/specification/app/resource-manager/readme.md

has revealed all occurrences of AggregateError: Error running Nimma originate from a problem with patchPropertiesCorrespondToPutProperties.

For example:

fatal   | spectralPluginFunc: Failed validating: '<local clone>/specification/app/resource-manager/Microsoft.App/preview/2023-08-01-preview/Usages.json'. Errors encountered:
AggregateError: Error running Nimma
Error 1: $[paths,'x-ms-paths'].* threw: ErrorWithCause("Function \"patchPropertiesCorrespondToPutProperties\" threw an exception: Cannot read properties of undefined (reading 'length')")
Error 2: $[paths,'x-ms-paths'].* threw: ErrorWithCause("Function \"patchPropertiesCorrespondToPutProperties\" threw an exception: Cannot read properties of undefined (reading 'length')")

Appears to me the bug is here:

• https://github.com/Azure/azure-openapi-validator/blob/452b964de0e7722c92e01fa08a121c429a8ba57a/packages/rulesets/src/spectral/functions/patch-properties-correspond-to-put-properties.ts#L29

That is, patchBodyProperties may be undefined, resulting in the reported exception of Cannot read properties of undefined (reading 'length').

Note on error occurrence

The error logs of form error encountered: Error: Error running Nimma go at least as far as 2023-04-03, increasing over time (query link):

// Logs confirmed present, selected days:
// before 2023-04-03: did not check
// 2023-04-03 - 64,
// 2023-04-06 - 120,
// 2023-05-02 - 150,
// 2023-06-08 - 182,
// 2023-06-15 - 64,
// 2023-06-21 - 240,
// 2023-07-03 - 144,
// 2023-07-06 - 144,
// 2023-07-11 - 680,
// 2023-08-01 - 5k,
// 2023-09-19 - 12.4k
// 2023-09-20 - 16.6k
SpecsPipelinesBuildsJobLogs("LintDiff", ago(2d), now())
| where Message has "Error running Nimma"
| summarize count() by bin(Timestamp, 1d)

FYI @rkmanda @TimLovellSmith @bdefoy

[konrad-jamrozik]

Closing as completed.

Relevant artifacts:

• Print out details of AggregateError originating from Spectral (@microsoft.azure/openapi-validator v2.1.4 -> v2.1.5) azure-openapi-validator#595
• https://devdiv.visualstudio.com/DevDiv/_git/openapi-alps/pullrequest/499394
• https://devdiv.visualstudio.com/DevDiv/_build/results?buildId=8431926&view=results

Follow-up work:

• Update PatchPropertiesCorrespondToPutProperties to run in stagingOnly azure-openapi-validator#596