[?w] changelog process for collections

[gundalow]

• How do we notify users of (potentially) breaking changes (aka porting_guide)
• How do we notify users of new features?
• Do we need to document bug fixes

Once we've answered the above, we can look at tooling that exists.

[samccann]

Release Note / Changelog Fragment Ideas for Collections

We have a continuing need for changelogs (or release notes) to cover:

• ansible/ansible - currently has a working solution using changelog fragments
• collections - two options mentioned so far are reno and towncrier
• Ansible Community Distribution (ACD) - a bundled package that includes the trimmed down ansible/ansible and a set of curated collections that will mimic the current user experience of Ansible with all batteries included (2.9 level of content). ACD has no proposed solution yet, but whatever is used by collections has to provide fragments that can be drawn up into an ACD changelog/release note.

Reviewing similar projects

We've looked at how RHEL, Fedora, and OpenStack handle this scenario. These projects are built from a subset of projects/components that are similar to how ACD will be created.

RHEL

RHEL is a release that combines a set of subprojects (e.g. gnome, linux kernel, qemu, etc). Looking at hte RHEL release, it includes Release Notes and Red Hat Errata. The Release Note focuses on new features, bug fixes, and known issues since the last RHEL release. It does not include a combined list of changelogs etc from all the subprojects upstream that make up RHEL. The errata is a more detailed list of bugfixes and upstream subprojects included in the RHEL release, but again, does not include details on all the changes that happened in all the subprojects. RHEL uses internal tools the generate the release notes and errata.

Fedora

Fedora is upstream of RHEL and is also built from a set of upstream subprojects. Fedora includes release notes and ChangeSet. One package example (RPM) shows a link to the upstream release notes that are a summary of changes from one major release to another (e.g. RPM 4.14x to RPM 4.15.0) That RPM release note appears to be manually created. Since Fedora upgraded from RMP 4.14 to RPM 4.15, the release notes upstream show the full set of changes that were manually added to the release note. Another example, the gnu c library - the Fedora release notes have high-level changes and point to the upstream NEWS section for the gnu c project for details. This news section seems to be partly manually written, and partly a script-generated list of bugfixes.

OpenStack

Openstack uses the reno to create a release note for each individual subproject. There is a main page that lists all the individual projects and links to their release notes. Openstack subprojects can release on a slower cycle than the Openstack release so in that sense, they are different from ACD, which is a compilation of collections that may have released multiple times between ACD versions. That said, reno may support creating a subset of release notes between versions. See the ironic release note that goes from 12.2 to 13.0 for the current Openstack train release.

[samccann]

ansible-collections/community.kubernetes#40 is also discussing this...

[samccann]

Changelog tool options

This comment summarizes the changelog tools under investigation. All tools must:

• work with git
• provide changelogs or porting_guide to be written in fragments committed with the code change.
• automate the creation of the changelog summary file for the collection.

OpenStack reno

The reno project:

• uses yaml files stored in a subdirectory (the yaml accepts .rst format inside). These are never removed from the repo.
• installable with pip
• customizable 'sections' - so we might be able to recreate some of our current changelog sections (e.g. new modules, deprecated features, etc)
• includes a section called upgrade that could be used to pull out just porting-guide like content with a separate script.
• fragment type is determined by the content of the yaml file
• includes a Sphinx extension to allow the release note html file to be generated from the fragments.
• possible Travis CI integration?
• notes are output in the order they appear in git history for each section.
• includes a reno lint option to check the syntax of the fragments to help enforce correctness.
• can configure the earliest_version that reno will include within the release note. I think this uses the git release tags.
• example fragments
• example release note

towncrier

The towncrier project:

• uses rst fragment files stored in a subdirectory.
• installed with pip.
• deletes all fragments (git rm) once the news.rst file is generated.
• fragment type seems determined by the fragment filename.
• example fragments
• example News.rst output

[mattclay]

We should also consider updating the current changelog tool we use to work with collections. We used reno originally, but built our own tool for additional features we needed:

1. Automatically document new plugins and modules. No fragment creation required.
2. Does not require git to function.
3. Supports sub-sections in the table of contents and in the details for each released version.
4. Provides linting for both the yaml and the embedded rst in fragments.

That last one is critical, as it's embedded into ansible-test sanity testing (currently only for Ansible), to prevent invalid fragments from being merged. Without that, invalid fragments are not discovered until release time when build failures occur due to invalid syntax.

I'm sure there are other features I'm forgetting as well, but I think I covered most of the major ones.

[felixfontein]

I prefer Ansible's tool and towncrier over reno, since they don't enforce a specific git workflow.

Ansible's tool also has the advantage that we can easily adjust it to do exactly what we want. And people working with Ansible really should be familiar with YAML files :)

[samccann]

@felixfontein can you clarify the git workflow you are talking about? I think in order for use to be able to gather fragments from multiple collections (and multiple releases within the collection) to get a complete ACD release note, we will need those git release tags within the collections.

[mattclay]

@samccann The current tool tracks every fragment that goes into a release in a JSON YAML file, so it's trivial to find out which fragments belong in which release without using git.

[mattclay]

For those not familiar with the current tool, here's a basic idea of how it works in ansible/ansible:

1. Contributors add changelog fragments in PRs. These are tested with ansible-test before merging. This prevents syntax errors in the yaml and embedded rst, as well as any issues with parsing or generating the release notes at the time of release.

2. When releasing a new version of the code, the changelog tool is executed. It uses a YAML file to keep track of which changelog fragment belongs in which release. This file is committed as part of the release process. This avoids depending on git or any other SCM to work with changelog fragments. The file just needs to exist in the source.

3. To generate a human readable changelog, the changelog tool is executed. It reads the YAML file which assigns fragments to releases, as well as the fragments, and generates a readable changelog.

Since steps two and three are separate, it allows for things like generating a detailed changelog of pre-releases and then collapsing the pre-release entries for a GA release.

This could also be used to facilitate processes such as combining changelog fragments from multiple collections and generating a comprehensive changelog. The original fragments would be used, yet the resulting changelog output could be, if desired, organized and formatted differently.

[mattclay]

Step number two is also where the tool creates entries to track new plugins and modules in the YAML file, which is why fragments do not need to be manually created for those. No need to worry about forgetting to document a new plugin or module.

[resmo]

Ansible's tool (let's give it a name) because it has all we need and we used it before.

[felixfontein]

ansichlog? :)

[geerlingguy]

Whatever we choose, I just want it to be simple, and ideally also have some way of linting (if possible) so whoever does releases for collections doesn't have to sit around manually cleaning up stuff at release time. It's already a larger burden pushing out new releases for collections than roles, and would be especially bad if we can start automating releases but those releases get broken due to bad fragments.

[sshnaidm]

Will it be a must and be forced for all collections in ACD? In Openstack Ansible Collection we consider to use reno, as it's Openstack native tool for changes.
If Ansible changelog tool will be chosen here, does it mean all others must accept it? Or we still can use whatever we prefer.

[webknjaz]

• uses rst fragment files stored in a subdirectory.

Not just RST. It can be any text file and the changelog rendering can be templated w/ jinja2.

[webknjaz]

also have some way of linting

FYI for reno fragments, ansible/ansible does this in CI. For towncrier, I have a GitHub App enforcing the presence of the fragments in PRs.

[samccann]

Changelog Idea Comparison

Goal: provide a unified changelog for ACD

NOTE: Current thinking is that the porting guide for ACD would be handled the same as we handle it today for ansible (core) - that is, we provide the shell of a porting guide and each collection owner is responsible for putting in the details necessary for an ACD user to upgrade from the current ACD version to the new ACD version (for their respective collection).

#1 Require ansible/ansible changelog process

Pros:

• Devs used to it
• Minimal extra coding

Cons:

• Limits collection owner flexibility
• Requires turning ansible/ansible solution into separate package that collection owners can use
• Current implementation works best when old changelog entries are cleared out each time a new stable release branch gets cut; may be tricky if changelog entries stay forever

#2 Recommend ansible/ansible approach or collection owner must provide changelog file per release in ansible-defined format.

Pros:

• Provides some flexibility for collection owners

Cons:

• requires coding from repo: mongodb #1 plus coding to fetch each collection's changelog files
• Collection maintainers would need to transform whatever format they use into the ansible-defined format

#3 Same as #2 but includes individual changelog fragments

#4 Link to changelogs within collections

Pros:

• Minimal extra work for ACD dev & collection owners
• Provides most flexibility for collection owners to do what they want with changelogs

Cons:

• Users have to click the links and read each collection changelog separately

#5 Same as #4 but have a manual release note for ACD that collection owners can add important changes to.

Pros:

• Provides a way for users to see just the important changes vs all changes within a changelog.

Cons:

• Users who want more detail still have to follow all the links for their collections
• Possible duplication between the manual release notes and the manual porting guides

[felixfontein]

My peronal ranking (best to worst): 2, 5, 4, 1 (I intentionally left 3 away)

• 2 is most flexible for collection authors (they aren't forced to use a specific tool), but results in usable ACD changelogs
• unfortunately it requires some work; for collections it's no work if they use ACD's tool, but some work if they use another tool
• if we can't do 2, it's probably better to simply link to changelogs instead of forcing a tool upon the people
• if we do 4 or 5, I would prefer 5 to allow to report "porting guide"-like entries
• requiring the ansible/ansible changelog process is in my opinion the worst choice, since it forces collection authors to use ansible/ansible's workflow with release branches and they cannot use anything else (such as reno, release-drafter, ...)

[pabelanger]

I'm a big fan of reno. And have wanted to enable it on network collections for some time.

[danielmellado]

+1 for reno, it'll be also a straighforward change for network collections/tox

[jimi-c]

3, 2, 4, 5, 1 (best to worst).

I always figured it'd be the same way changelogs in RPMs are handled - the changes are in the collection and can be viewed by viewing the collection's information. I prefer #3 to #2 (fragments vs. monolithic).

[webknjaz]

I prefer # 2

[webknjaz]

@samccann you may want to create a poll via https://github.com/apps/polls next time

[abadger]

@pabelanger @danielmellado Note: reno is only sort of an option. #2 would require collection owners to create some code so that reno can output a machine parsable changelog that our tools can then pull in to make a unified changelog/porting guide. #3 would require the same but at the fragment level instead of a single output file (there would probably need to be an additional metadata file as well that pulled out what fragments belong to which release.). #4 would let you use reno and you'd manually tell us what the link to your reno-generated changelog is so that we can add the link to our changelog/release notes. #5 Would be the same but docs would merge manual requests from collection owners to highlight things that collection owners considered important enough to show up in the ACD changelog.

[felixfontein]

@jimi-c as I understand it, 3 vs. 2 only means whether the fragments have to be there for ACD to consume them. For creating changelogs, I really prefer fragments as well, but I don't mind if collections using other tools provide the compiled changelog as a combined YAML file and not as a set of changelog fragment files.

[samccann]

Summary of voting/comments

Based on the comments and actual votes, the top vote-getter is:

2 - recommending the ansible/ansible approach (and the assumption that we'd turn this into a tool that collection owners could use).

• The collection owner can also use any tool they want (reno, towncrier, etc), but need to provide a per-collection changelog file that matches the current format of ansible/ansible changelogs.
• In the event that the collection owner doesn't provide a properly-formatted file per collection release, we would default to adding a link to their changelog in whatever human-readable format it is in.

[apple4ever]

My vote is 5 first, then 4 then 2.

I much prefer direct linking so I can see just what I want. I don't need a giant list that is hard to scan about changes. Having important changes noted would be nice, but if I don't care about AWS but do care about Azure, I should just be able to dive into their logs.

[samccann]

This is a mockup of what an ACD level changelog file might look like - https://github.com/samccann/ansible/blob/test-changelog/docs/docsite/rst/changelog_acd.rst

[Andersson007]

BTW

- get_certificate - Fix cryptography backend when pyopenssl is unavailable (https://github.com/ansible/ansible/issues/67900)
- openssh_keypair - add logic to avoid breaking password protected keys.

Would be good to officially determine a proper changelog message format:

• should it start from uppercase letter or lowercase?
• should it contain a link to a pr itself if it does not have an issue?
• should it have a dot at the end?
  Mix of things like above looks a bit sloppy:)

[felixfontein]

I tried to generalize Ansible's changelog generator to collections; here's my current state: https://github.com/felixfontein/ansible-changelog/

I've ran it for community.general (with version_added: 1.0.0 for all new content); here's the result: https://github.com/felixfontein/community.general/tree/changelog-1.0/changelogs

In particular:

• the changelog YAML container file: https://github.com/felixfontein/community.general/blob/changelog-1.0/changelogs/changelog.yaml
• the generated .rst file: https://github.com/felixfontein/community.general/blob/changelog-1.0/changelogs/CHANGELOG-v1.rst

[samccann]

reno/towncrier etc users - can you please evaluate the changelog YAML container file in the above comment? That is similar to what we would want collection owners to produce if they don't use the proposed changelog generator based on ansible/ansible. (with the final fallback of just giving us a link to whatever the collection owner has)

[felixfontein]

Update: I moved the code for ACD changelog build here: https://github.com/felixfontein/ansibulled/tree/acd-changelog/acd_changelog The resulting changelog: https://github.com/felixfontein/ansibulled/blob/acd-changelog/acd_changelog/acd.rst It now only contains one collection (the only one on galaxy with the proper changelog.yaml AFAIK).

Old: Here is a code-generated version of @samccann's mockup: https://github.com/felixfontein/ansible-changelog/blob/14570defd37aa5fd1047842a2ac77b1cf10bd84f/acd/acd.rst

And here another version which has ACD versions as top-level index, and below that sections for Ansible Base and the collections: https://github.com/felixfontein/ansible-changelog/blob/14570defd37aa5fd1047842a2ac77b1cf10bd84f/acd/acd2.rst

The changelog is parameterized by the following config file: https://github.com/felixfontein/ansible-changelog/blob/14570defd37aa5fd1047842a2ac77b1cf10bd84f/acd/acd.yml, and generated by https://github.com/felixfontein/ansible-changelog/blob/14570defd37aa5fd1047842a2ac77b1cf10bd84f/acd/acd.py

[Andersson007]

ansible/ansible#69268 , it's a PR about changelog entry's format i mentioned above

[felixfontein]

I started writing specification of the changelog.yaml format here, and I included a linter for such files in the ansible-changelog tool.

[felixfontein]

I also created a test collection with a changelog spanning multiple changelog.yaml files and a non-trivial versioning history.

[felixfontein]

ansibulled has been renamed to antsibull, and can now be installed via pip. Part of antsibull is antsibull-changelog, the changelog generator! Here is some documentation for collection maintainers on using it.

If you are using another changelog generator and want to output a ACD-compatible changelog.yaml file, the format is documented here, and antsibull includes a validator (antsibull-lint changelog-yaml).

[felixfontein]

The changelog generator was moved to https://github.com/ansible-community/antsibull-changelog/ and can be installed via pip (https://pypi.org/project/antsibull-changelog/). The docs are now here, and the changelog.yaml syntax is here.

[felixfontein]

ansible-test sanity now includes a changelog fragment linting sanity check: if changelogs/fragments/ exists and contains files, these will be linted with antsibull-changelog. (If changelogs/config.yaml does not exist when fragments are found, the test will fail.)

[samccann]

@felixfontein - seems like this issue can be resolved now that https://pypi.org/project/antsibull-changelog/ is live?

[felixfontein]

@samccann I think so! Thanks for reminding us :)