Trusted publishing: Support for GitHub reusable workflows

[woodruffw]

Our MVP implementation (#10753) assumes that the workflow is in the same repository, which is not necessarily true.

We should support reusable workflows, specifically via the job_workflow_ref claim.

[woodruffw]

Just as a note: this will have to interoperate with #11263.

[GergelyKalmar]

Any chance to expedite this issue? It seems there's many who would be eager to move over to trusted publishing now that it is the recommended approach, but not being able to do the flow from a reusable workflow is a pretty major blocker. This issue also does not seem to be documented anywhere so we only find out about it once we changed everything to use the new approach.

[woodruffw]

Any chance to expedite this issue?

I can't personally promise a timeline here, but it is on my backlog of things to do.

This issue also does not seem to be documented anywhere so we only find out about it once we changed everything to use the new approach.

Thank you for pointing this out; I'll make some changes to the docs to emphasize that reusable workflows are not currently supported.

[woodruffw]

Thank you for pointing this out; I'll make some changes to the docs to emphasize that reusable workflows are not currently supported.

I've opened #13592 for this. Thanks again for bringing it to our attention!

[webknjaz]

@woodruffw I just stumbled upon an example of a reusable workflow actually working with OIDC here jorisroovers/gitlint#486 — could you confirm this is because of secrets: inherit at https://github.com/jorisroovers/gitlint/blob/5247839/.github/workflows/ci.yml#L225 ?

[GergelyKalmar]

That workflow is in the same repository though, I think the issue is with reusable workflows that are in a different repository (e.g. in a separate public repo).

[woodruffw]

Yeah, I think that's because it's in the same repo, not because of secrets: inherit.

In other words: reusable workflows do work with the current implementation, just not reusable workflows that are external to the repository that the trusted publisher was configured with.

[webknjaz]

That workflow is in the same repository though, I think the issue is with reusable workflows that are in a different repository (e.g. in a separate public repo).

Yes, you can't use inherit across orgs. The issue was that the OIDC context has a reusable workflow reference and that's currently not checked IIUC. So my concern is that it might be possible to fool the verification logic in PyPI with this...

[GergelyKalmar]

You can get around the inherit issue between organizations via forking, which is what we would usually do. So same org, different repo would be fine. The point of reusable workflows in this case would be that you store them in a central repository and you can re-use them in other repositories.

[mhils]

True, but I would argue that a parent-level workflow will be overscoped almost by definition, whereas a non-reusable workflow can be made minimally scoped.

Reusable workflows can be made minimally scoped just as well. You may be right that folks are more likely to overscope here, but I don't have any data on this. 🙂

This might be an option. IME though, most projects don't use environments, so making it required would (arguably) be harder for users since they would have to set it up. I'm open to it though, since I might be wrong here.

FWIW I think security-wise environments would be the superior solution here. For example, we have (relatively speaking) lots of folks with write access to the main branch, but only a few selected individuals can approve deployments (and thus push to PyPI).

The one major downside I can think of is that environments are not available for free accounts in private repositories. So that makes it hard to enforce as a strict requirement. If that's a major concern, job_workflow_ref may be the better solution.

Thank you for pushing this entire thing forward! 🎉

[GergelyKalmar]

I'd definitely prefer the implementation where we can scope on the reusable workflow level, I think that the few optional fields in the setup UI would be totally fine and it's unlikely to confuse anyone.

Environments are great, but there are lots of organizations that already have an established CI/CD setup that does not use environments and have no need for it otherwise, so requiring it just to use this feature would be probably suboptimal.

[GergelyKalmar]

With attestation becoming a more and more important part of the publication process to PyPI I wanted to ask if it would be possible to expedite this work together with #13911.

This issue makes it less convenient to use trusted publishing compared to the token-based approach (which works fine with reusable workflows already), while the lack of tag pattern matching described in #13911 makes it difficult to achieve the same level of control and security over publishing as what the token-based approach provides.

[woodruffw]

With attestation becoming a more and more important part of the publication process to PyPI I wanted to ask if it would be possible to expedite this work together with #13911.

We're currently working on reusable workflow support. There are no short-term plans (on my part, at least) for #13911, however. That's going to require a larger re-thinking of the data model, since every additional optional field geometrically increases the number of states PyPI needs to check.

[thepwagner]

Is there a reason why we cannot match on workflow_ref (the top-level workflow) only?
The scenario we're trying to avoid is having overscoped Trusted Publishers

Could this just be a boolean + warning?

• Allow reusable workflows. ⚠ Only enable this if you're using a dedicated release workflow.

Behind the scenes, the checked box would mean only workflow_ref is verified, per _mhils suggestion.

---

Or a really out there idea: decode workflow_ref and fetch the associated workflow file contents from GitHub.
If you know that contents of workflow_ref only specify a single job, then you've mitigated the concern of overscoping without caring about job_workflow_ref.
OSSF scorecards use that technique to make assertions about the workflow that is publishing - https://github.com/ossf/scorecard-webapp/blob/main/app/server/verify_workflow.go

[woodruffw]

Could this just be a boolean + warning?

The main reason we can't do this is because, long term, we want to allow both states to enable hermetic CI configurations: a user should be able to configure pypi.yml as their "top-level" workflow (i.e. the one in workflow_ref), and then a specific pinned reusable workflow.