[Breaking Change] (Cross Version) should not compare against old preview

[mikekistler]

In PR 21606 the Breaking Change(Cross Version) checks compared the new api version to the most recent stable, 2022-05-01, and to an older preview version, 2021-04-01-preview.

There is no point in reporting differences to a preview that is older than the most recent stable api version. This clutters the report with issues that are not relevant and potentially wastes reviewers time investigating these issues.

This issue belongs to a set of related (and possibly at least partially duplicate) issues:

• [Breaking Change][PR Workflow] pipeline can auto-detect a prior approval #4850
• [Breaking Change] (Cross Version) should not compare against old preview #5024
• [Breaking Change][PR Workflow] Breaking Changes should not flag breaks from prior preview as errors #6396
• [Breaking Change] Breaking changes PR check does not check against latest GA version #6567

[raych1]

@jianyexi , looks like the preview version earlier than last GA can be excluded from comparison as per the guidance https://github.com/Azure/azure-rest-api-specs/blob/main/documentation/ci-fix.md#breaking-change-check.

[jianyexi]

This is by design, as the breaking change (cross-version) will select the latest version which exists in the main branch to compare with the new version in the PR

[mikekistler]

It looks to me like 2022-05-01 exists in the main branch and is newer than 2021-04-01-preview, which would make 2022-05-01 the latest version in the main branch. So what is the reason for comparing against 2021-04-01-preview?

[mikekistler]

Discussed in REST API Tooling meeting. Concluded that we want to compare against the most recent GA version (if one exists). We also compare against the latest preview version but only if it is more recent than the latest GA, and these reported as warnings. This rule applies for both new GA version and new preview version.

[heaths]

I'd argue it's worth reporting, but not as a required check. I've seen cases where a couple previews in flight should have both gotten fixes but didn't, and the breaking change was a good indication of that.

[konrad-jamrozik]

@mikekistler could you clarify what is the required behavior change here? Not exactly clear to me upon reading through the discussion. Thank you!

[mikekistler]

The desired behavior is described in this comment and we should make whatever changes are needed to make this be the actual behavior.

[konrad-jamrozik]

There exists an open PR that is supposed to potentially address this:

• https://devdiv.visualstudio.com/DevDiv/_git/openapi-alps/pullrequest/450147

[konrad-jamrozik]

Another case:

• https://github.com/Azure/azure-rest-api-specs/pull/25002/checks?check_run_id=15921551415
• Email thread Re: Service Fabric Managed Clusters - api 2023-07-01-preview Cross-Version Breaking Changes

[konrad-jamrozik]

This work item was mentioned by @mikekistler in recent discussion on comparing to RPSaaSMaster. The discsusion on Teams is here.

[konrad-jamrozik]

Additional context in Teams, here.