Using a few OSSF scorecard checks to show more information to Actions consumers

[svemu]

We at GitHub Actions are thinking of surfacing the security posture of a Repo that hosts Actions code on the Actions listing pages.
While there is information we can get by making API calls(E.g: Branch protection enabled or not), for the others (E.g:Binary artifacts, dangerous workflows )we are looking at using the OSSF checks (use the CLI in a workflow and get the score) and start to display that information on the Actions listing page. Since this is a new change, there will be enough time for the Action creators to fix before the information is shown on the listing page.

We looked at all the OSSF checks and using the following checks in the critical, high and medium categories as something that we need to check for and surface to the consumer of the Action.

Dangerous-Workflow
Binary-Artifacts
Branch-Protection
Code-Review
Dependency-Update-Tool
Vulnerabilities
Pinned-Dependencies
SAST

We will use the same scoring process for each of the checks and display the results

CC: @azeemshaikh38 , @laurentsimon

[azeemshaikh38]

@ossf/scorecard-maintainers fyi.

Thanks @svemu. The list of checks looks good to me. What help can we offer here to get this effort going?

[anuraag016]

Hi @azeemshaikh38, I have a couple of questions I wanted some help with -

1. We are currently thinking of using the OSSF command line tool instead of the scorecard-action. This is mainly due to two reasons:
   1. Scorecard action currently does not support running the action for selected checks. Our use case, as mentioned above, is to run only 8 selected checks for repos containing GitHub Actions.
   2. Scorecard action does not support windows runners.
      Is there a plan in place to support the above features for the scorecard action? If so, we would be able to start using the action itself.
2. We wanted to check if there is a way to provide a custom tool name in the SARIF output - ( Scorecard-Lite/Scorecard-Actions,etc) so that it is easy to filter alerts based on the tool name in the Code Scanning UI. Currently the tool name in the SARIF output is Scorecard - we want to remove any confusion for users who already use/ might use the OSSF Scorecard Action.
3. For our current choice of approach of using the command line tool, we have few queries -
   1. We tested the tool with windows runners and were able to run it without issue. It is mentioned in the docs that windows users may experience some issues, I wanted to know if there is some documentation/information about what those issues might be?
   2. We were able get the SARIF output from the command line tool by setting an env variable, but this method isn’t documented anywhere. Is this approach safe to use or can you help us with any problems we might face?
4. Not a query, but can you point us to some documentation/information of common errors faced while using the command line tool?

cc: @shruti-shukla

[azeemshaikh38]

We are currently thinking of using the OSSF command line tool instead of the scorecard-action.

Could you clarify how you plan to do this at scale (i.e for all GitHub Action providers)? Would every GitHub Action repo be required to include scorecard-action in their project or would GitHub internally run Scorecard for these repos for all triggers (commit push, branch protection update etc.) ?

Scorecard action currently does not support running the action for selected checks. Our use case, as mentioned above, is to run only 8 selected checks for repos containing GitHub Actions.

If GitHub plans to re-create something similar to scorecard-action with support for running selected checks, could I suggest that you folks could directly contribute to ossf/scorecard-action to get this feature added in? It would improve scorecard-action and also reduce workload on your end.

Scorecard action does not support windows runners.

AFAIK, we don't test scorecard-action against windows runners so the confidence there is not high. Its completely likely that it would be supported on windows without any extra code so we would just need extra tests to productionize it. @laurentsimon can you think of any reason we may not be able to support windows?

We wanted to check if there is a way to provide a custom tool name in the SARIF output - ( Scorecard-Lite/Scorecard-Actions,etc) so that it is easy to filter alerts based on the tool name in the Code Scanning UI. Currently the tool name in the SARIF output is Scorecard - we want to remove any confusion for users who already use/ might use the OSSF Scorecard Action.

This may not be needed if we can use scorecard-action to support your usecase directly?

We tested the tool with windows runners and were able to run it without issue. It is mentioned in the docs that windows users may experience some issues, I wanted to know if there is some documentation/information about what those issues might be?

Mostly testing. All our tests are heavily Ubuntu/Linux focused.

We were able get the SARIF output from the command line tool by setting an env variable, but this method isn’t documented anywhere. Is this approach safe to use or can you help us with any problems we might face?

Yes it is safe and this is how scorecard-action also does it. The only problem I anticipate is that this is expected to be an internal only feature (hence the lack of documentation). Even though this will be supported in the long-term the usage may change.

Not a query, but can you point us to some documentation/information of common errors faced while using the command line tool?

Hmm, not sure we have something like that. What specifically were you looking for here?

[laurentsimon]

re: windows runner. The go binary should run on Windows so I don't expect problems. But as @azeemshaikh38 said, most our tests run on ubuntu runner. I think we could add some tests in the scorecard-action to run on windows runners, if we want more certainty. We can definitely do that if needed.

[anuraag016]

Could you clarify how you plan to do this at scale (i.e for all GitHub Action providers)? Would every GitHub Action repo be required to include scorecard-action in their project or would GitHub internally run Scorecard for these repos for all triggers (commit push, branch protection update etc.) ?

We plan to run it internally on a schedule and for new releases of the action. Users will not have to do anything actively to run this.

If GitHub plans to re-create something similar to scorecard-action with support for running selected checks, could I suggest that you folks could directly contribute to ossf/scorecard-action to get this feature added in? It would improve scorecard-action and also reduce workload on your end.

Would love to contribute and get this feature in, just want to be sure if this is already in scope or planned, if not we can definitely look into contributing and raising a PR and would be great if any approvals on this can be prioritized so our timelines aren't at risk

This may not be needed if we can use scorecard-action to support your usecase directly?

Even with the action it would be great if we can possibly pass in a custom value for the tool name in the SARIF output, as it still helps to show user filtered results based on different runs. (Custom run added by user itself vs Run that GitHub would trigger).

The only problem I anticipate is that this is expected to be an internal only feature (hence the lack of documentation). Even though this will be supported in the long-term the usage may change.

If we can get the scorecard-action itself to run selected checks then we wont have to worry about it, till then we can keep a lookout on this.

Hmm, not sure we have something like that. What specifically were you looking for here?

Just wanted to see if there are known common errors that we can put safeguards for, since we are trying to run this internally we want to ensure that it runs without a hitch and does not show up as a failed attempt to the users.

AFAIK, we don't test scorecard-action against windows runners so the confidence there is not high. Its completely likely that it would be supported on windows without any extra code so we would just need extra tests to productionize it. @laurentsimon can you think of any reason we may not be able to support windows?

re: windows runner. The go binary should run on Windows so I don't expect problems. But as @azeemshaikh38 said, most our tests run on ubuntu runner. I think we could add some tests in the scorecard-action to run on windows runners, if we want more certainty. We can definitely do that if needed.

It would be great if we can get certain tests added, it would give us more confidence in running on windows based runners.

[azeemshaikh38]

Would love to contribute and get this feature in, just want to be sure if this is already in scope or planned, if not we can definitely look into contributing and raising a PR and would be great if any approvals on this can be prioritized so our timelines aren't at risk

Not planned, but this is definitely within scope. If you could provide a brief description of the design/solution for configuring the checks in scorecard-action we can make sure we are all in agreement before you spend your time on coding it. And yes, I can review these PRs for you on priority.

Even with the action it would be great if we can possibly pass in a custom value for the tool name in the SARIF output, as it still helps to show user filtered results based on different runs. (Custom run added by user itself vs Run that GitHub would trigger).

Not sure I understood this - IIUC the plan is to show Scorecard info to Action consumers and not Action repo devs? How would Actions consumers have access to custom run added by user itself?

[laurentsimon]

@anuraag016 if GitHub runs the Action themselves, is there anything specific to your environment for the Action to detect it's being run by you and not the user? I ask because we could add some logic into the Action to detect it's running on your environment, and allow you to pass certain parameters, but without releasing this functionality to end-users.

Just something to consider to reduce the amount of maintenance and support on the Action side. Once there is a proper design we're comfortable releasing for end-users (selection of checks, etc), we would then migrate your "special" integration to the "public" version of it (in the future).

[anuraag016]

Not sure I understood this - IIUC the plan is to show Scorecard info to Action consumers and not Action repo devs? How would Actions consumers have access to custom run added by user itself?

@azeemshaikh38 The plan is to show the scorecard info to both the Action repo owners and the consumers of the actions. For the Repo owners we want to remove the confusion of custom run vs the Run that GitHub would trigger. For the customers we would ensure that we pick only the results that were created by our initiated runs.

@anuraag016 if GitHub runs the Action themselves, is there anything specific to your environment for the Action to detect it's being run by you and not the user?

@laurentsimon yes there is a way we can identify if the workflow is run by us or by user. The event_name in the github context will have a reserved event name that can not be used by users.

I ask because we could add some logic into the Action to detect it's running on your environment, and allow you to pass certain parameters, but without releasing this functionality to end-users.

Just something to consider to reduce the amount of maintenance and support on the Action side. Once there is a proper design we're comfortable releasing for end-users (selection of checks, etc), we would then migrate your "special" integration to the "public" version of it (in the future).

This would be great if it means we can get these functionalities faster. We would be able to directly use the action instead of relying on the command line tool.

[azeemshaikh38]

Sounds reasonable to me. To summarize, the plan is to:

• update SARIF generation logic to output tool name based event_name context.
• Add capability to scorecard-action to choose which checks to run.

Does that sound right @anuraag016 ? @laurentsimon @ossf/scorecard-maintainers fyi.

[laurentsimon]

Let's also keep these changes "private", ie we will make the changes for the integration and don't publicize them?
The ability to select which checks to use is useful for public consumption, but I think we need some discussion whether this should be done via a config file, input argument, etc. For the integration, we can select a simple option without committing to it for all our users.

@anuraag016 if we were to implement the check selection via a hidden env variable, and in a few months decide we will provide a config file for public use, do you have a scalable way to patch your workflow across projects (env -> config)?

NOTE: there's also this workaround #1098 (comment) that you could use to select checks, which does not require changes. Would this work?

[laurentsimon]

follow-up question: would you be pushing the results back to our servers? The Action has an option publish_result: true that lets users send the results to our backend servers. We then make these results available via a REST API and BiqQuery.

The setup uses OIDC to prove to the backend that the results originate from the correct repo. We do some verification on the workflow.yml that run scorecard before ingesting the results. I doubt this would work in your setup, since the workflow is not part of the repo's history.

[anuraag016]

Sounds reasonable to me. To summarize, the plan is to:

• update SARIF generation logic to output tool name based event_name context.
• Add capability to scorecard-action to choose which checks to run.

Does that sound right @anuraag016 ? @laurentsimon @ossf/scorecard-maintainers fyi.

@azeemshaikh38 Yes, these seem right. One more thing is, if we can get the action working in Windows environment as well.

Let's also keep these changes "private", ie we will make the changes for the integration and don't publicize them?
The ability to select which checks to use is useful for public consumption, but I think we need some discussion whether this should be done via a config file, input argument, etc. For the integration, we can select a simple option without committing to it for all our users.

@laurentsimon This would work for us perfectly. Our use case is of running a workflow for repos of other users, best approach for us would be argument based selection of checks. We can go with a config file or any other way as well, just that would require a little extra maintenance on our side.

@anuraag016 if we were to implement the check selection via a hidden env variable, and in a few months decide we will provide a config file for public use, do you have a scalable way to patch your workflow across projects (env -> config)?

Yes, since we run the workflow dynamically for repos we can change the workflow file centrally and the new one would be used in all future runs.

NOTE: there's also this workaround #1098 (comment) that you could use to select checks, which does not require changes. Would this work?

We had thought of a similar workaround, but decided against it because it ends up running all checks which would mean using extra compute power than necessary, increasing the cost of running it for all the action repos present in GitHub. But this can be used to change the scorecard tool name without any changes to the action.

follow-up question: would you be pushing the results back to our servers? The Action has an option publish_result: true that lets users send the results to our backend servers. We then make these results available via a REST API and BiqQuery.

The setup uses OIDC to prove to the backend that the results originate from the correct repo. We do some verification on the workflow.yml that run scorecard before ingesting the results. I doubt this would work in your setup, since the workflow is not part of the repo's history.

We are not planning on publishing the results from the workflow that we run. The authors can and will have the possibility of manually adding the OSSF starter workflow to run the checks and upload the results if they wish.

[laurentsimon]

Thanks for all the info.
We can start with a branch for you to try the end-to-end integration. One we have all the feedback from you, we can cut a release. Does this work for you?

When do you ideally need a first PoC to try?

[anuraag016]

When do you ideally need a first PoC to try?

@laurentsimon We plan to start testing the workflow internally by end of this month or start of the next month. An initial PoC by that time would be the most ideal for us.

[laurentsimon]

I will make a branch with the changes.

@laurentsimon yes there is a way we can identify if the workflow is run by us or by user. The event_name in the github context will have a reserved event name that can not be used by users.

What's the event name? I''ll use this in the code to enable this internal feature just for your use case. Thanks

[laurentsimon]

Since the discussion is about the scorecard-action, I have transferred the issue to the Action repo #1107

[laurentsimon]

I send ossf/scorecard#2773. I think it's actually simpler to make changes only to the Scorecard code.

[varunsh-coder]

We at GitHub Actions are thinking of surfacing the security posture of a Repo that hosts Actions code on the Actions listing pages. While there is information we can get by making API calls(E.g: Branch protection enabled or not), for the others (E.g:Binary artifacts, dangerous workflows )we are looking at using the OSSF checks (use the CLI in a workflow and get the score) and start to display that information on the Actions listing page. Since this is a new change, there will be enough time for the Action creators to fix before the information is shown on the listing page.

We looked at all the OSSF checks and using the following checks in the critical, high and medium categories as something that we need to check for and surface to the consumer of the Action.

Dangerous-Workflow Binary-Artifacts Branch-Protection Code-Review Dependency-Update-Tool Vulnerabilities Pinned-Dependencies SAST

We will use the same scoring process for each of the checks and display the results

CC: @azeemshaikh38 , @laurentsimon

I like this idea as it will improve the security posture of the GitHub Actions ecosystem.

Please consider adding Token Permission and Security Policy checks to the list. Token Permission is a high-risk issue as per Scorecard and also recommended by the GitHub Actions security guide - make sure that the GITHUB_TOKEN is granted the minimum required permissions, and Security Policy makes it easier to report vulnerabilities.

[ljharb]

I would be very cautious about indiscriminately showing metrics from the scorecard. There’s a bunch of them that don’t apply to every ecosystem (binary artifacts, for example), and some that imply inactionable recommendations (like, “go get another maintainer”). “Pinned dependencies” would be actively harmful if it applies to npm packages, and it’s not actually a good idea to apply to actions in every case, since you’d be depriving yourself of automatic bug and security fixes.

[laurentsimon]

@ljharb GitHub Actions are applications so it should avoid the problem.

@anuraag016 I've made a test branch for you to try:

- uses: ossf/scorecard-action@9ea03ba18bab6e2ea29983f3b0fabdb8fd20ecbc # gh-integration
   env:
      SCORECARD_INTERNAL_GITHUB_INTEGRATION: 1
      SCORECARD_INTERNAL_GITHUB_SARIF_TOOL_NAME: tool-name
      SCORECARD_INTERNAL_GITHUB_CHECKS: Binary-Artifacts, Security-Policy

Please tell me if it works.

Note: the tool name will have its first letter capitalized automatically https://github.com/ossf/scorecard/blob/main/pkg/sarif.go#L421. In the example above, it would be Tool-Name

[ljharb]

@laurentsimon i'm not sure what you mean by that - I run github actions on hundreds of packages. GHA is in no way unique to applications.

[laurentsimon]

I meant that GitHub Action are not "libraries". The GitHub tutorials ask users to use this template https://github.com/actions/javascript-action which uses a lock file. And to use dependabot to receive updates.

[ljharb]

ahhh ok, so you're talking about providing scores for actions not for the projects that consume those actions. That makes sense, thanks for clarifying!