CD pipeline no longer works because of PyPi attestations enforcement for reusable workflow

[amaltaro]

Impact of the bug
WMCore in general

Describe the bug
We made a new tag yesterday and noticed that the CD pipeline failed to authenticate our package against PyPi, as seen in these logs:
https://github.com/dmwm/WMCore/actions/runs/11956233634/job/33390661595#step:8:182

This seems to be a recent (~3 weeks) change in PyPi; and we need to update our github action workflow to take this into consideration.

How to reproduce it
Just make a new WMCore tag with the current (reusable) CD workflow.

Expected behavior
Upon a new tag, all the expected WMCore packages should be built and uploaded to PyPi without any errors (in addition to the remaining CD pipeline actions).

We have tried many things so far, so probably what is left now is to transform our PyPi workflow from reusable to non-reusable.

Additional context and error message
History of events to try to resolve it:

• 2.3.8rc1: we spotted this issue
• 2.3.8rc2: we had a new trusted publisher configured in every single project in PyPi
• 2.3.8rc3: we adopted an "attestation" step in our CD pipeline.
• 2.3.8rc4: we enabled write access to store the attestation (which should have been provided in the previous tag).

[mapellidario]

We are not the only ones to face this problem [1].

Apparently it is a combination of many factors:

• we use trusted publishing to pypi
• pypi started supporting digital attestations
• we use the github action github.com/pypa/gh-action-pypi-publish , which started enforcing digital attestations
• we have a reusable workflow in our github actions to build and publish our packages to pypi
• pypi does not properly support trusted publishing in reusable workflows [2] and it is suggested to use only non-reusable workflows [3]

Alan made some experiments with his own repo [4], but we are still not sure if we can

I would suggest to stop using the reusable workflow pypi_build_publish_template.yaml and move those steps in the parent workflow, replacing this uses:. Our github actions would be less clean and tidy, but at least they would be more compatible with the external tools that we use (aka pypi). [edit] thanks alan for #12180, it is exactly what i had in mind :)

---

[1] pypa/gh-action-pypi-publish#283
[2] pypi/warehouse#11096
[3] pypa/gh-action-pypi-publish#283 (comment)
[4] https://github.com/amaltaro/TestActions/actions