Add tool to generate changelog for Azure SDK for Dotnet #6033

RodgeFu · 2023-04-23T07:45:19Z

This is a tool to update changelog.md with generated release notes for Azure SDK for Dotnet.

In detail, following release notes will be generated automatically:

Obsoleted API in the stable release (i.e. Method/Property/Model/...)
The change in the api-version tag used when generating the SDK (i.e. tag: package-preview-2023-03)
The change in the version of dependencies (i.e. Azure.Core, Azure.ResourceManager)

Usage:

> ChangelogGen.exe apiFile version releaseDate(xxxx-xx-xx)

Example:

> ChangelogGen.exe ...\azure-sdk-for-net\sdk\compute\Azure.ResourceManager.Compute\api\Azure.ResourceManager.Compute.netstandard2.0.cs 1.2.3 2099-02-03

Remark:

For preview release, the previous release will be used as baseline to detect changes.
For GA release, the previous GA release will be used as baseline to detect changes.
The generated release notes will be merged into the last release (1.1.0-beta.1 (Unreleased)) in changelog.md.
Only Management Plane is supported now and the Data Plane will be supported in next step.

…re SDK for Dotnet Mgmt plane)

konrad-jamrozik · 2023-04-24T17:57:26Z

@RodgeFu can you provide some elaboration on this work in the PR description? Please include link to the relevant issue and/or project.

In addition, please include a README.md for the added tool.

Currently I can only guess what this is about.

RodgeFu · 2023-04-25T06:14:30Z

@konrad-jamrozik, thanks for looking at the PR. The issue is: Azure/azure-sdk-for-net#35186

following is a brief of the tool.
This is a tool to generate release note in changelog for Azure SDK for Dotnet. Now following info is included:

The change in the API interface for the release (i.e. Added/Remove/Obsoleted Model/Method/Property)
The change in the tag of api-version used in codegen (i.e. tag: package-preview-2023-03)
The change in the version of dependencies (i.e. Azure.Core, Azure.ResourceManager)

The tool takes the api file as argument (i.e. ChangelogGen ...\azure-sdk-for-net\sdk\compute\Azure.ResourceManager.Compute\api\Azure.ResourceManager.Compute.netstandard2.0.cs) and would merge the generated release notes into the changelog.md file like the "## 1.0.2 (2023-04-27)" release in https://github.com/RodgeFu/azure-sdk-for-net/blob/main/sdk/websites/Azure.ResourceManager.AppService/CHANGELOG.md

I dont see readme file in other tools under Dotnet. Is there any template I should use for the Readme.md? Or a brief description like above is good enough? Thanks.

RodgeFu · 2023-04-27T04:52:10Z

@konrad-jamrozik, please help to review the PR. Feel free to let me know if there is anything else missing. thanks.

konrad-jamrozik · 2023-04-27T05:56:51Z

adding @hallipr and @jsquire as azure-sdk-for-net repo owners

@RodgeFu apologizes for the delay! I am wondering - did you had a chance to collaborate with anyone from the Redmond Azure SDK EngSys team in preparation for making this PR? For example, @hallipr or @weshaggard ? Just trying to figure out if there is anybody on my team that has additional context for this work and as such should be aware of this PR.

Re this:

I dont see readme file in other tools under Dotnet. Is there any template I should use for the Readme.md? Or a brief description like above is good enough? Thanks.

I think if you copy-paste your PR comment into the README with slight adjustments to make the context more appropriate, this would work.

RodgeFu · 2023-04-27T06:46:03Z

@konrad-jamrozik, thanks for the reply. I haven't collaborated with anyone on your team for the work item. Include @ArthurMa1978 in case there is more background before the work item is assigned to me.

As for the README, i have added a readme.md to the PR. feel free to let me know if more adjustments are needed. thanks.

jsquire · 2023-04-27T13:34:11Z

I haven't had a chance to review yet, but I'd like to understand the goals and intended use of this before we merge the PR. I'd also like to request that we add meaningful descriptions to pull requests when opening them. Someone unfamiliar should be able to look at a PR and understand the context.

//cc: @ArthurMa1978, @RodgeFu

weshaggard · 2023-04-27T21:16:00Z

@RodgeFu I'd like to better understand to goals of this tool. In general, some of the other languages have generated changelogs with these library diffs and I'd say for the most part they have contained more noise than value for the consumers of the changelog. So much so that we have filtered them out in our release notes and release blog posts.

RodgeFu · 2023-04-28T05:23:10Z

@weshaggard, the goal of this tool is to generate the common part in changelog so that people don't need to do it manually every time including list the change in API interface, upgrade in the dependency version of Swagger, Azure.Core and Azure.ResourceManager now. The information are needed for each release and have to be checked and written manually every time now. So having a tool to generate them automatically will be helpful to save the dup manual cost as well as avoid possible mistakes when writing them manually. I am not sure about the noise you mentioned in the changelog in other language. It will be helpful if you can provide more detail or example about the noise so that we can see whether it also applies here. thanks.

Agree to below and will make sure more description will be added next time. Furthermore, I am interested in whether there is any recommended way that we can contact your team first when we want to add more tools to help our work in Azure SDK so that we may collaborate or leverage your team's knowledge/resource in some way and benefit both sides. thanks.

I haven't had a chance to review yet, but I'd like to understand the goals and intended use of this before we merge the PR. I'd also like to request that we add meaningful descriptions to pull requests when opening them. Someone unfamiliar should be able to look at a PR and understand the context.

//cc: @ArthurMa1978, @RodgeFu

jsquire

Flagging to prevent accidental merges until I have a chance to review and we understand the intended scenario and usage for this tool.

jsquire · 2023-04-28T14:30:59Z

@RodgeFu: Are you intending this to be something manually run by developers to populate the change log that you'll manually review/trim or are you intending this to be integrated with CI?

Generally speaking, we do NOT want to list out every single API change in the changelog - that's not something that is contextually helpful for customers and creates a lot of noise that hides information that they actually need. There are some exceptional cases (compilation breaking changes, specifically) where we do want to call out the exact member and point customers at a file that shows migration examples, but not for every addition/change and especially not for preview releases which we expect to have churn.

The goal of a change log is to highlight for customers changes that may impact them or they may wish to take advantage of in the context for how they would use the SDK. We want to curate this and summarize at a high level, not provide an API diff. Please read the Azure SDK policy for release notes - our change logs may go into a bit deeper detail than the release notes, but should generally follow this guidance.

To highlight the point that Wes is making, consider the following contrived examples:

Poor change log

5.10.0-beta.1 (Unreleased)

Features Added

Added GetThatThing to SomeClient
Added FooClient

Breaking Changes

None

Bugs Fixed

Added CancellationToken parameter to SomeActionAsync for BarClient

Other Changes

Renamed OldName to NewName for OtherClient
Changed SomeValue for OtherClient from int to long

Good change log

5.10.0-beta.1 (Unreleased)

Features Added

It is now possible to query for ThatThing using SomeClient
Foo resources can now be managed using the new FooClient

Bugs Fixed

SomeActionAsync may now be cancelled when using BarClient

src/dotnet/Azure.SDK.ChangelogGen/Azure.SDK.ChangelogGen.Tests/TestApiComparer.cs

src/dotnet/Azure.SDK.ChangelogGen/Azure.SDK.ChangelogGen/Azure.SDK.ChangelogGen.csproj

src/dotnet/Azure.SDK.ChangelogGen/Azure.SDK.ChangelogGen/ChangeLogResult.cs

jsquire · 2023-05-01T14:46:04Z

Looking through the code, I'm not comfortable with this automatically generating change logs. If we're going to use it as a diff tool to make authors aware of what changed so that they can craft a good quality change log, we should output to something that is obviously temporary.

weshaggard · 2023-05-02T17:58:44Z

As an example, take a look at https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/network/azure-mgmt-network/CHANGELOG.md#2100-2022-08-05 which is automatically generated but we end up filtering this out when we publish the release notes at https://azure.github.io/azure-sdk/releases/2022-08/python.html. That is because it flooded the entire notes and we couldn't find the more interesting changes. I've found that a blind api diff in changelogs doesn't end up adding a lot of value.

RodgeFu · 2023-05-04T10:37:39Z

This tool is expected to be triggered manually and the generated and merged changelog will be reviewed through standard PR review process which should be able to make sure they are reviewed properly before check-in.
As for the API diff list, I believe the breaking change, obsoleted API and the upgrade of dependencies are worthwhile to be included in the release notes. As for others, I am fine to exclude them, @ArthurMa1978 to confirm the DCR.

jsquire · 2023-05-04T14:14:59Z

obsoleted API and the upgrade of dependencies are worthwhile to be included in the release notes.

If we're talking about a GA package, then I agree. If we're talking changes from a beta, then no. We market the Azure SDK as having a very strong focus on backwards compatibility and NOT making breaking changes to GA packages. We are clear that the same guarantees do not apply to betas and that beta packages should not be used in production applications.

We generally want to avoid having change logs that list API tweaks that are breaking for betas. It makes it more difficult for customers to find breaking changes to GA versions that are impactful for production applications by introducing a lot of visual noise. It also presents poor optics, as a quick scan of a change log with a large list of breaking changes communicates that these are not, in fact, rare occurrences and weakens perceptions of our guarantees.

//cc: @czubair and @KrzysztofCwalina for additional perspectives.

ArthurMa1978 · 2023-05-04T15:44:42Z

This tool is expected to be triggered manually and the generated and merged changelog will be reviewed through standard PR review process which should be able to make sure they are reviewed properly before check-in. As for the API diff list, I believe the breaking change, obsoleted API and the upgrade of dependencies are worthwhile to be included in the release notes. As for others, I am fine to exclude them, @ArthurMa1978 to confirm the DCR.

This tool can assist us to capture all changes from a new release.
And I agree that listing all new methods and properties for a new release would be noisy. Therefor I think the auto-gened changelog should contain:

Service api version change
Dependency change
Obsoleted change, include class, method and property

ArthurMa1978 · 2023-05-04T15:49:30Z

obsoleted API and the upgrade of dependencies are worthwhile to be included in the release notes.

If we're talking about a GA package, then I agree. If we're talking changes from a beta, then no. We market the Azure SDK as having a very strong focus on backwards compatibility and NOT making breaking changes to GA packages. We are clear that the same guarantees do not apply to betas and that beta packages should not be used in production applications.

We generally want to avoid having change logs that list API tweaks that are breaking for betas. It makes it more difficult for customers to find breaking changes to GA versions that are impactful for production applications by introducing a lot of visual noise. It also presents poor optics, as a quick scan of a change log with a large list of breaking changes communicates that these are not, in fact, rare occurrences and weakens perceptions of our guarantees.

//cc: @czubair and @KrzysztofCwalina for additional perspectives.

Yes, the phrase "breaking change" is incorrect because we always fix them.

czubair · 2023-05-04T22:35:13Z

I agree with @jsquire that change logs should not include automatically generated list of every API change. As per these carefully thought-out guidelines (Azure SDK policy for release notes)- they should be limited to developer impacting changes. Customers do not need to know about an API change unless there is an action they need to take or it impacts a functionality they may already be using.

KrzysztofCwalina · 2023-05-04T22:58:02Z

If we wanted to provide a listing of all API changes, I think we would need a better format for that. Maybe something like our API view diff. I totally agree with @jsquire and @weshaggard that the format in this PR is more of a noise than helpful.

RodgeFu · 2023-05-05T10:07:08Z

Thanks for all the inputs. So as a summary, instead of listing all the API changes, we will only have following info generated in the changelog (assume we won't have any breaking change in API)

Service api version change
Dependency change
Obsoleted change, include class, method and property

These notes will be under the 'Other Changes' section. I will update the code accordingly if there is no more comment. thanks.

jsquire · 2023-05-05T13:49:32Z

Obsoleted change, include class, method and property

For GA versions only, please. We don't want to track beta-only breaking changes and obsoleted members.

RodgeFu · 2023-05-09T14:03:24Z

Yes. The obsoleted changes will only be generated for GA version. thanks.

2. Change merge mode from group to line 3. for GA release, the previous GA release will be used as baseline

RodgeFu · 2023-05-15T11:22:08Z

I have updated the code and comments. Please help to have a look again and let me know if there are more comments. thanks.

tools/Azure.SDK.ChangelogGen/Azure.SDK.ChangelogGen/Azure.SDK.ChangelogGen.csproj

tools/Azure.SDK.ChangelogGen/Azure.SDK.ChangelogGen.sln

jsquire · 2023-05-16T15:07:47Z

tools/Azure.SDK.ChangelogGen/Azure.SDK.ChangelogGen/ChangeLogResult.cs

+
+            if (ApiChange?.GetBreakingChanges().Count() > 0)
+            {
+                ReleaseNoteGroup breakingGroup = new ReleaseNoteGroup("Breaking Changes");


What is the plan for putting together information in a breaking-changes.txt file that demonstrate how to migrate code?

Not quite sure I understand the comment correct here. Do you mean we will have a doc to demonstrate how to mitigate breaking change?

This part code is to handle the case when breaking change detected just in case though we dont expect any breaking change. In case there are breaking changes, the tool will log an error as well as adding these breaking change into changelog under "Breaking Changes" section anyway to make sure people are aware of these unexpected breaking changes and fix them when reviewing PR.

Do you mean we will have a doc to demonstrate how to mitigate breaking change?

Yes. That is the standard for Azure SDK packages. [Example]. Here, again, I don't know if there was an exception granted to management libraries. @ArthurMa1978?

Once the management plane SDK reaches GA, no new releases will introduce breaking changes. Any breaking changes in the swagger spec will be handled by using customization code (or autogen code in the future).
So just skip this part here.

@jsquire , kindly ping..., please let us know if you have further questions about this. thanks.

Once the management plane SDK reaches GA, no new releases will introduce breaking changes. Any breaking changes in the swagger spec will be handled by using customization code (or autogen code in the future).
So just skip this part here.

We generally do not document API breaking changes from beta-to-beta. I thought we agreed to only list these for GA libraries - so if this section does not apply to GA libraries, then let's not include it or, let's at least have some minimal plan for how we'll communicate to partner teams that they need to put together a breaking changes guide.

I misunderstood that we will fix breaking change for beta too which seems not true. fixed to ignore breaking changes for beta version. Thanks.

Can you help me understand where that logic is? I don't see a filter for breaking changes or obsolete against beta versions.

the original logic was when calling GenerateReleaseNote method in ChangeLogResult.cs, it took parameters 'ignoreBreakingChange' and 'filter'. the 'ignoreBreakingChange' would be true for preview release to skip them while only ObsoleteChange would be set in filter to only log obsoleted changes. Just now, I realize that the api comparison part is just not needed at all for preview version, so i have updated the code to skip that part for preview version to make the logic more clean and bring better perf and stability. thx.

tools/Azure.SDK.ChangelogGen/Azure.SDK.ChangelogGen/ChangeLogResult.cs

tools/Azure.SDK.ChangelogGen/Azure.SDK.ChangelogGen/Compare/ApiComparer.cs

tools/Azure.SDK.ChangelogGen/Azure.SDK.ChangelogGen/Context.cs

tools/Azure.SDK.ChangelogGen/Azure.SDK.ChangelogGen/Program.cs

weshaggard · 2023-05-16T16:25:45Z

/azp run tools - Azure.SDK.ChangelogGen - ci

azure-pipelines · 2023-05-16T16:25:53Z

Azure Pipelines successfully started running 1 pipeline(s).

RodgeFu · 2023-05-17T10:13:55Z

/azp run tools - Azure.SDK.ChangelogGen - ci

azure-pipelines · 2023-05-17T10:14:05Z

Azure Pipelines successfully started running 1 pipeline(s).

tools/Azure.SDK.ChangelogGen/Azure.SDK.ChangelogGen.Tests/Azure.SDK.ChangelogGen.Tests.csproj

jsquire · 2023-05-30T16:18:30Z

tools/changelog-gen-for-net/Azure.SDK.ChangelogGen/Azure.SDK.ChangelogGen.csproj

Let's please rename this project to include a reference to "management" or "mgmt" or "resource manager"

renamed to Azure.SDK.Management...

jsquire · 2023-05-30T16:48:55Z

tools/changelog-gen-for-net/Azure.SDK.ChangelogGen/Compare/Change.cs

+        }
+
+        public abstract string Description { get; }
+        public virtual bool IsBreakingChange


This isn't the only thing that can cause a breaking change. I am concerned that we're oversimplifying this check and other breaking changes will be overlooked because people assume this tool is doing deeper analysis than it is.

As a couple of examples:

The order of parameters was changed

A parameter was renamed

A new overload was added with a subset of existing parameters and different behavior

A default value was changed which changes behavior

A breaking change means an original interface no longer available (removed) in the end. all the examples you listed above will trigger two changes detected, one new interface added and one old interface removed. and all removed interfaces will be treated as breaking change.

The other examples are also breaking changes and will need to be documented and treated that way. As mentioned, my fear is that people will rely on this generator to identify breaking changes for them and miss those that are not "we removed a thing."

the breaking changes from other examples will also be detected by the tool because the original interface no longer exists after the change. For example, if Method(string a, int b) is changed to Method(int b, string a). the tool will find the Method (string a, int b) can't be found in new version (removed) and log a breaking change for it.
If we think a breaking change in a general way, it's just because an interface no longer available in the new version (treated as removed by the tool) no matter it's because the interface is deleted or renamed or modified to a different signature...

That's fair. I overlooked that the signature comparison would catch the two structural cases:

The order of parameters was changed

A parameter was renamed

The other two cases would require semantic analysis to catch. Since we're scoping this to just management libraries and Arthur already signed-off, I won't block on this but I do still want to register my concerns about it - particularly for partner teams.

tools/changelog-gen-for-net/Azure.SDK.ChangelogGen/ChangeLogResult.cs

2. improve log 3. support same version of last release as the given one to improve the re-run user experience

Blocking issues have been addressed.

weshaggard · 2023-06-02T19:22:37Z

/azp run tools - net-changelog-gen-mgmt - ci

azure-pipelines · 2023-06-02T19:22:47Z

Azure Pipelines successfully started running 1 pipeline(s).

weshaggard · 2023-06-02T19:23:45Z

I updated the pipelines with the new location of the ci.yml based on the file path change.

2. fix a bug in method default value check

RodgeFu · 2023-06-03T06:41:32Z

/azp run tools - net-changelog-gen-mgmt - ci

azure-pipelines · 2023-06-03T06:41:40Z

Azure Pipelines successfully started running 1 pipeline(s).

RodgeFu added 2 commits April 23, 2023 15:34

Initial check-in for ChangelogGen (Tool to generate changelog for Azu…

6d472f6

…re SDK for Dotnet Mgmt plane)

Merge remote-tracking branch 'upstream/main' into changeloggen

a8e0f62

RodgeFu requested a review from a team as a code owner April 23, 2023 07:45

RodgeFu added 3 commits April 25, 2023 18:41

add support of api-version tag

b05d995

Update test for api-version tag

faf7645

add a simple readme.md

6e4720a

weshaggard requested a review from ArthurMa1978 April 27, 2023 21:13

jsquire requested changes Apr 28, 2023

View reviewed changes

jsquire reviewed May 1, 2023

View reviewed changes

1. Only list obsoleted API change

08661c4

2. Change merge mode from group to line 3. for GA release, the previous GA release will be used as baseline

RodgeFu added 2 commits May 15, 2023 15:41

fix ci after moving to tools folder

cd94206

Merge remote-tracking branch 'upstream/main' into changeloggen

66754b8

ArthurMa1978 mentioned this pull request May 16, 2023

Update the changelog.md automatically after generating new SDK Azure/azure-sdk-for-net#35186

Closed

4 tasks

RodgeFu added 3 commits May 16, 2023 15:47

Update readme file for supporting DataPlane in the next step

362e13a

Update markdown format of readme.md

e1c52e9

Update markdown format of readme.md2

2859f6a

jsquire previously requested changes May 16, 2023

View reviewed changes

Azure deleted a comment from azure-pipelines bot May 16, 2023

RodgeFu added 2 commits May 17, 2023 13:44

Update markdig to signed version

a42e1b2

include static method/property/type in the api compare

78857ba

ArthurMa1978 approved these changes May 24, 2023

View reviewed changes

weshaggard reviewed May 24, 2023

View reviewed changes

tools/Azure.SDK.ChangelogGen/Azure.SDK.ChangelogGen.Tests/Azure.SDK.ChangelogGen.Tests.csproj Outdated Show resolved Hide resolved

rename folder to changelog-gen-for-net under tools\

3597ad8

jsquire reviewed May 30, 2023

View reviewed changes

RodgeFu added 2 commits May 31, 2023 17:09

remove breaking changes from preview version

1525253

1. rename folder name to net-changelog-gen-mgmt

73f1641

2. improve log 3. support same version of last release as the given one to improve the re-run user experience

1. shortcut api comparison for preview version which is not needed

c4a1970

2. fix a bug in method default value check

RodgeFu merged commit fbd523b into Azure:main Jun 5, 2023

Add tool to generate changelog for Azure SDK for Dotnet #6033

Add tool to generate changelog for Azure SDK for Dotnet #6033

Conversation

RodgeFu commented Apr 23, 2023 • edited Loading

Usage:

Example:

Remark:

konrad-jamrozik commented Apr 24, 2023 • edited Loading

RodgeFu commented Apr 25, 2023 • edited Loading

RodgeFu commented Apr 27, 2023

konrad-jamrozik commented Apr 27, 2023 • edited Loading

RodgeFu commented Apr 27, 2023

jsquire commented Apr 27, 2023 • edited Loading

weshaggard commented Apr 27, 2023

RodgeFu commented Apr 28, 2023 • edited Loading

jsquire left a comment

Choose a reason for hiding this comment

jsquire commented Apr 28, 2023 • edited Loading

Poor change log

5.10.0-beta.1 (Unreleased)

Features Added

Breaking Changes

Bugs Fixed

Good change log

5.10.0-beta.1 (Unreleased)

Features Added

Bugs Fixed

jsquire commented May 1, 2023

weshaggard commented May 2, 2023

RodgeFu commented May 4, 2023 • edited Loading

jsquire commented May 4, 2023

ArthurMa1978 commented May 4, 2023

ArthurMa1978 commented May 4, 2023

czubair commented May 4, 2023

KrzysztofCwalina commented May 4, 2023

RodgeFu commented May 5, 2023

jsquire commented May 5, 2023

RodgeFu commented May 9, 2023

RodgeFu commented May 15, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

weshaggard commented May 16, 2023

azure-pipelines bot commented May 16, 2023

RodgeFu commented May 17, 2023

azure-pipelines bot commented May 17, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

weshaggard commented Jun 2, 2023

azure-pipelines bot commented Jun 2, 2023

weshaggard commented Jun 2, 2023

RodgeFu commented Jun 3, 2023

azure-pipelines bot commented Jun 3, 2023

RodgeFu commented Apr 23, 2023 •

edited

Loading

konrad-jamrozik commented Apr 24, 2023 •

edited

Loading

RodgeFu commented Apr 25, 2023 •

edited

Loading

konrad-jamrozik commented Apr 27, 2023 •

edited

Loading

jsquire commented Apr 27, 2023 •

edited

Loading

RodgeFu commented Apr 28, 2023 •

edited

Loading

jsquire commented Apr 28, 2023 •

edited

Loading

RodgeFu commented May 4, 2023 •

edited

Loading