Rethink the Collection Removal process

[gotmax23]

Summary

We discussed a bit in the meeting how we could simplify the process. It feels like a lot of work (filing a community-topics issue, filing a collection issue, starting a vote, filing an ansible-build-data issue, adding a deprecation changelog entry) if all it does is kludge maintainers into fixing issues and we add back the collection.

(I'll write a less terse summary after the meeting. Putting this here so I don't forget.)

[mariolenz]

Sorry, I couldn't attend the last meeting. But I had a look at the discussion.

I agree, the removal process is a lot of (manual) work. It would be great to automate this.

So we would need some kind of CI job to test the collections. @Andersson007 suggested to use passing sanity tests as criteria. That's a good starting point, although maybe someone has a good idea for additional tests. Of course, those have to be tests we can do automatically.

How could this look like? Well, the current Ansible 7.2.0 is based on ansible-core 2.14.2. So the CI job would pull in all collections that it contains (from source or from Galaxy?) and do the sanity tests with 2.14.2.

Collections that fail those tests in three consecutive Ansible releases are automatically tagged as unmaintained. I think we should still announce this in the collection's issue tracker and maybe also The Bullhorn. Can we automate this?

Removal should also be automated. Say, if a collection fails the tests in another three consecutive Ansible releases?

Important: In those rather simple cases, we wouldn't vote on removing the collection, it would just happen.

A lot to discuss and probably also a lot of fine detail to add but: Thoughts so far?

@gotmax23 Did you have something like this in mind?

[felixfontein]

Please don't forget that Ansible uses semantic versioning, so we cannot remove anything during a release cycle (except in emergencies, but I don't think this would be one). We can only remove collections from the next release cycle on.

Regarding automatic sanity tests: @dmsimard set up something like that earlier (last year? or the year before?), and one problem with it is that it takes a very long time to run though. In particular pylint is very slow if you have large files, and there is at least one collection with very large module files. So there are some practical limits to this we shouldn't ignore.

[mariolenz]

Please don't forget that Ansible uses semantic versioning, so we cannot remove anything during a release cycle (except in emergencies, but I don't think this would be one). We can only remove collections from the next release cycle on.

Sorry, I expressed myself badly. With Removal should also be automated I didn't mean to remove the collection automatically from the next minor release. It should be deprecated there and removed from the next major release. But, if at all possible, automatically and without voting.

Regarding automatic sanity tests: @dmsimard set up something like that earlier (last year? or the year before?)

ansible-community/antsibull-build#419 mentioned in #96?

and one problem with it is that it takes a very long time to run though. In particular pylint is very slow if you have large files, and there is at least one collection with very large module files. So there are some practical limits to this we shouldn't ignore.

I read something there about The sanity tests timed out after 6h, it looks like there's something wrong with fortinet.fortimanager. But if we have problems running the sanity tests, should they have also?

But I agree, we shouldn't ignore the practical limits.

[gotmax23]

So we would need some kind of CI job to test the collections. @Andersson007 suggested to use passing sanity tests as criteria. That's a good starting point, although maybe someone has a good idea for additional tests. Of course, those have to be tests we can do automatically.

Yeah, I think we need tooling to run sanity tests across collections and report failures. I believe #96 is where we previously discussed this.

It might also be an interesting idea to publish an activity ranking, amount of active contributors, or some sort of statistics about the development of the various collections in the ansible package. I believe @GregSutcliffe has done similar work with metrics in the past.

How could this look like? Well, the current Ansible 7.2.0 is based on ansible-core 2.14.2. So the CI job would pull in all collections that it contains (from source or from Galaxy?) and do the sanity tests with 2.14.2.

For now, it should be possible to download the ansible sdist from PyPI and loop over the directories.

Collections that fail those tests in three consecutive Ansible releases are automatically tagged as unmaintained. I think we should still announce this in the collection's issue tracker and maybe also The Bullhorn. Can we automate this?

Both Matrix (for the Bullhorn) and Github (for the issues) have APIs (and ansible modules that interface with them IIRC), but someone would need to write that automation :).

Removal should also be automated. Say, if a collection fails the tests in another three consecutive Ansible releases?

Important: In those rather simple cases, we wouldn't vote on removing the collection, it would just happen.

I'm not sure. I think we should try and work with collection maintainers and evaluate the individual situations, but we need to ensure that we aren't too lenient and let these sanity test issues stall (this has happened in the past).

@gotmax23 Did you have something like this in mind?

I guess I have two main discussion questions:

1. Is the removal process always the best way to get collections to fix issues?
2. Are there ways in which we can automate the removal process or make it more efficient?

Re Question #1:

The Removal Process is good kludge to get people to fix problems, but I wish there were other ways to proactively identify unmaintained collections or collections with a single active maintainer and encourage more community participation in collection maintenance. FWIW, Fedora's Engineering Steering Committee has an official policy for encouraging package comaintainers. The structure of our two projects is quite different, but I think we can take inspiration :).

Re Question #2:

Thinking out loud here:

We could automate more parts of the process, but someone has to do that work. Personally, I'm spread a bit thin right now :). @mariolenz started on this in ac040f2. I was thinking about adding this on to the antsibull project. We could have an antsibull-manager command that can automate announcements or filing issues, but I'm not convinced the antsibull project right place for this type of thing or if this is worth the effort.

We could also teach antsibull-build to automatically insert changelog entries about deprecated collections. Perhaps it can get the data from collection_exclusion tickets in this repository.

We could handle identifying and voting on inactive collections all at once at a specific point in the release cycle to avoid repetitive work. We'd file issues and announce the unmaintained collections in the Bullhorn anytime, but at this specific point, we'd reevaluate collections that said they needed more time, hold votes all at once, and then announce the removals all at once in the Bullhorn and changelog.

[mariolenz]

@gotmax23 @felixfontein Is this something we should close, move to the forum or keep open?

[mariolenz]

@gotmax23 Please close this issue if done, or open a new forum topic and then close the issue with a pointer to the new discussion: Community-topics: Archiving the repo

[felixfontein]

From my side we can close this.

[Andersson007]

from my too, so let's close it, @gotmax23 please consider creating a forum topic later if needed