Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: npm builder updates #1206

Merged
merged 7 commits into from
Nov 7, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -189,8 +189,7 @@ jobs:
###################################################################
build:
outputs:
node-tarball-sha256: ${{ steps.upload.outputs.sha256 }}
node-tarball-name: ${{ steps.tarball.outputs.filename }}
artifacts: ${{ steps.upload.outputs.artifacts }}
runs-on: ubuntu-latest
needs: [builder, rng, detect-env]
steps:
Expand All @@ -202,7 +201,6 @@ jobs:
path: __BUILDER_CHECKOUT_DIR__

- name: Checkout the Node repository
# TODO: set a pin or use at head?
uses: ./__BUILDER_CHECKOUT_DIR__/.github/actions/secure-project-checkout-node
with:
node-version: ${{ inputs.node-version }}
Expand All @@ -220,62 +218,64 @@ jobs:
sha256: "${{ needs.builder.outputs.node-builder-sha256 }}"
set-executable: true

- name: Download dependencies
# TODO(hermeticity)
- name: Build project
working-directory: __PROJECT_CHECKOUT_DIR__
env:
UNTRUSTED_CI_ARGUMENTS: "${{ inputs.ci-arguments }}"
UNTRUSTED_DIR: ${{ inputs.directory }}"
SLSA_WORKFLOW_INPUTS: "${{ inputs }}"
run: |
set -euo pipefail

# npm ci <args>
"$GITHUB_WORKSPACE/$BUILDER_BINARY" ci \
--ci-arguments "$UNTRUSTED_CI_ARGUMENTS" \
--directory "$UNTRUSTED_DIR"
"$GITHUB_WORKSPACE/$BUILDER_BINARY" build

# TODO(hermeticity) Enable OS-level hermeticity.
- name: Upload generated tarball
id: upload
uses: ./__BUILDER_CHECKOUT_DIR__/.github/actions/secure-upload-artifact
with:
name: "<TODO>"
path: "<TODO>"
Comment on lines +235 to +236
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Were you planning on setting these?

Copy link
Collaborator Author

@laurentsimon laurentsimon Nov 7, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not in this PR. The output of the builder can theoretically contain multiple artifacts, so I think we need to write code to process severals artifacts at once. For nodejs, I think it will always be a single artifact. But in general I'm trying to see what's needed to build a "generic workflow" that generalizes for any builder (nodejs, Java, goreleaser, container images, etc)


- name: Build project
working-directory: __PROJECT_CHECKOUT_DIR__
env:
UNTRUSTED_RUN_SCRIPTS: "${{ inputs.run-scripts }}"
UNTRUSTED_DIR: ${{ inputs.directory }}"
run: |
set -euo pipefail
build-dry:
outputs:
provenance-metadata: ${{ steps.build-dry.outputs.provenance-metadata }}
runs-on: ubuntu-latest
needs: [builder, build, rng, detect-env]
steps:
- name: Checkout builder repository
uses: slsa-framework/slsa-github-generator/.github/actions/secure-builder-checkout@main
with:
repository: "${{ needs.detect-env.outputs.repository }}"
ref: "${{ needs.detect-env.outputs.ref }}"
path: __BUILDER_CHECKOUT_DIR__

- name: Checkout the Node repository
uses: ./__BUILDER_CHECKOUT_DIR__/.github/actions/secure-project-checkout-node
with:
node-version: ${{ inputs.node-version }}
scope: ${{ inputs.scope }}
token: ${{ inputs.token }}
always-auth: ${{ inputs.always-auth }}
registry-url: ${{ inputs.registry-url }}
cache: "npm"

# npm run <args>
"$GITHUB_WORKSPACE/$BUILDER_BINARY" run \
--run-scripts "$UNTRUSTED_RUN_SCRIPTS" \
--directory "$UNTRUSTED_DIR"
- name: Download builder
uses: ./__BUILDER_CHECKOUT_DIR__/.github/actions/secure-download-artifact
with:
name: "${{ env.BUILDER_BINARY }}-${{ needs.rng.outputs.value }}"
path: "${{ env.BUILDER_BINARY }}"
sha256: "${{ needs.builder.outputs.node-builder-sha256 }}"
set-executable: true

- name: Create tarball
id: tarball
- name: Build dry project
id: build-dry
working-directory: __PROJECT_CHECKOUT_DIR__
env:
UNTRUSTED_DIR: ${{ inputs.directory }}"
SLSA_WORKFLOW_INPUTS: "${{ inputs }}"
SLSA_BUILD_ARTIFACTS: "${{ needs.build.outputs.artifacts }}"
run: |
set -euo pipefail

# TODO: only run if non-empty.
# Note: pack-destination only supported version 7.x above.
# https://docs.npmjs.com/cli/v7/commands/npm-pack.
# This outputs a .tgz. Before running this command, let's record the .tgz
# files and their hashes, so that we can identify the new file without the need to parse
# the manifest.json.
# echo "npm pack --pack-destination="./out"
"$GITHUB_WORKSPACE/$BUILDER_BINARY" pack \
--directory "$UNTRUSTED_DIR"

# cp output into upper folder to make the tarball accessible to
# next step.
echo "filename=$TARBALL" >> "$GITHUB_OUTPUT"

- name: Upload generated tarball
id: upload
uses: ./__BUILDER_CHECKOUT_DIR__/.github/actions/secure-upload-artifact
with:
name: "${{ steps.tarball.outputs.filename }}"
path: "${{ steps.tarball.outputs.filename }}"
"$GITHUB_WORKSPACE/$BUILDER_BINARY" build --dry-run

###################################################################
# #
Expand All @@ -284,14 +284,13 @@ jobs:
###################################################################
provenance:
runs-on: ubuntu-latest
needs: [builder, build, rng, detect-env]
needs: [builder, build, build-dry, rng, detect-env]
permissions:
id-token: write # Needed to create an OIDC token for keyless signing.
contents: read
actions: read # Needed to read workflow info.
outputs:
node-provenance-name: ${{ steps.sign-prov.outputs.signed-provenance-name }}
node-provenance-sha256: ${{ steps.sign-prov.outputs.signed-provenance-sha256 }}
provenances: ${{ steps.sign-prov.outputs.provenances }}
steps:
- name: Checkout builder repository
uses: slsa-framework/slsa-github-generator/.github/actions/secure-builder-checkout@main
Expand All @@ -311,33 +310,23 @@ jobs:
- name: Create and sign provenance
id: sign-prov
env:
UNTRUSTED_TARBALL_NAME: "${{ needs.build.outputs.node-tarball-name }}"
UNTRUSTED_TARBALL_HASH: "${{ needs.build.outputs.node-tarball-sha256 }}"
UNTRUSTED_CI_ARGUMENTS: "${{ inputs.ci-arguments }}"
UNTRUSTED_RUN_SCRIPTS: "${{ inputs.run-scripts }}"
UNTRUSTED_DIR: ${{ inputs.directory }}"
UNTRUSTED_PROVENANCE_METADATA: "${{ needs.build-dry.outputs.provenance-metadata }}"
UNTRUSTED_ARTIFACTS: "${{ needs.build.outputs.artifacts }}"
UNTRUSTED_WORKFLOW_INPUTS: "${{ inputs }}"
GITHUB_CONTEXT: "${{ toJSON(github) }}"
run: |
set -euo pipefail

echo "provenance generator is $BUILDER_BINARY"

# Create and sign provenance
# This sets signed-provenance-name to the name of the signed DSSE envelope.
# Note: this command outputs the sha256 of the signed provenance, so the upload
# in the next step need not be secure-artifact-upload.
"$GITHUB_WORKSPACE/$BUILDER_BINARY" provenance \
--tarball-name "$UNTRUSTED_TARBALL_NAME" \
--digest "$UNTRUSTED_TARBALL_HASH" \
--ci-arguments "$UNTRUSTED_CI_ARGUMENTS" \
--run-scripts "$UNTRUSTED_RUN_SCRIPTS" \
--directory "$UNTRUSTED_DIR"
"$GITHUB_WORKSPACE/$BUILDER_BINARY" provenance

- name: Upload the signed provenance
uses: actions/upload-artifact@83fd05a356d7e2593de66fc9913b3002723633cb # tag=v3.1.1
with:
name: "${{ steps.sign-prov.outputs.signed-provenance-name }}"
path: "${{ steps.sign-prov.outputs.signed-provenance-name }}"
name: "<TODO>"
path: "<TODO>"
if-no-files-found: error
retention-days: 5

Expand All @@ -358,7 +347,6 @@ jobs:
path: __BUILDER_CHECKOUT_DIR__

- name: Checkout the Node repository
# TODO: set a pin or use at head?
uses: ./__BUILDER_CHECKOUT_DIR__/.github/actions/secure-project-checkout-node
with:
node-version: ${{ inputs.node-version }}
Expand All @@ -375,12 +363,12 @@ jobs:
path: "${{ needs.build.outputs.node-tarball-name }}"
sha256: "${{ needs.build.outputs.node-tarball-sha256 }}"

- name: Download provenance
- name: Download provenances
uses: ./__BUILDER_CHECKOUT_DIR__/.github/actions/secure-download-artifact
with:
name: "${{ needs.provenance.outputs.node-provenance-name }}"
path: "${{ needs.provenance.outputs.node-provenance-name }}"
sha256: "${{ needs.provenance.outputs.node-provenance-sha256 }}"
name: "<TODO>"
path: "<TODO>"
sha256: "<TODO>"

- name: Download builder
uses: ./__BUILDER_CHECKOUT_DIR__/.github/actions/secure-download-artifact
Expand All @@ -393,11 +381,8 @@ jobs:
- name: Publish
working-directory: __PROJECT_CHECKOUT_DIR__
env:
UNTRUSTED_PUBLISH_ARGUMENTS: "${{ inputs.publish-arguments }}"
SLSA_WORKFLOW_INPUTS: "${{ inputs }}"
run: |
set -euo pipefail

# echo "npm publish ${{ inputs.publish-arguments }}"
"$GITHUB_WORKSPACE/$BUILDER_BINARY" publish \
--publish-arguments "$UNTRUSTED_PUBLISH_ARGUMENTS" \
--directory "$UNTRUSTED_DIR"
"$GITHUB_WORKSPACE/$BUILDER_BINARY" publish
13 changes: 0 additions & 13 deletions internal/builders/nodejs/README.md

This file was deleted.

Original file line number Diff line number Diff line change
Expand Up @@ -21,20 +21,28 @@ import (
"github.com/slsa-framework/slsa-github-generator/slsa"
)

// provenanceCmd runs the 'provenance' command.
func provenanceCmd(provider slsa.ClientProvider, check func(error), signer signing.Signer, tlog signing.TransparencyLog) *cobra.Command {
var provenanceDirectory string

// attestCmd runs the 'attest' command.
func attestCmd(provider slsa.ClientProvider, check func(error), signer signing.Signer, tlog signing.TransparencyLog) *cobra.Command {
c := &cobra.Command{
Use: "provenance",
Short: "Run provenance command",
Long: `Run provenance command to generate and sign provenance file.`,
Use: "attest",
Short: "Run attest command",
Long: `Run attest command to generate and sign attest file.`,

Run: func(cmd *cobra.Command, args []string) {
if err := attest(); err != nil {
panic(err)
}
},
}

c.Flags().StringVarP(&provenanceDirectory, "directory", "d", "", "Working directory to issue commands.")

return c
}

func attest() error {
// 1. Retrieve the provenance metadata from env variable UNTRUSTED_PROVENANCE_METADATA.

// 4. Create / sign a provenance file as per UNTRUSTED_PROVENANCE_METADATA.

// 5. Output the list of provenance files generated in a format TBD (sha256sum?).
return nil
}
109 changes: 109 additions & 0 deletions internal/builders/nodejs/build.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,109 @@
// Copyright 2022 SLSA Authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// https://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

package main

import (
"github.com/spf13/cobra"
)

// buildCmd runs the 'build' command.
func buildCmd(check func(error)) *cobra.Command {
var dryRun bool

c := &cobra.Command{
Use: "build",
Short: "build a project",
Long: `build a project. Example: ./binary build --dry-run"`,

Run: func(cmd *cobra.Command, args []string) {
if err := build(dryRun); err != nil {
panic(err)
}
},
}

c.Flags().BoolVarP(&dryRun, "dry-run", "d", false,
"Perform a dry run only. Do not build. Output provenance metadata (steps and provenance filenames)")
return c
}

func build(dryRun bool) error {
// 1. Retrieve the workflow inputs from env variable SLSA_WORKFLOW_INPUTS.

// 2. Install dependencies via `npm ci <inputs.ci-arguments>`

// 3. Build via `npm run <inputs.run-scripts>`

// 4. Create the final package tarball.
/* TODO: only run if non-empty.
Note: pack-destination only supported version 7.x above.
https://docs.npmjs.com/cli/v7/commands/npm-pack.
This outputs a .tgz. Before running this command, let's record the .tgz
files and their hashes, so that we can identify the new file without the need to parse
the manifest.json.
echo "npm pack --pack-destination="./out"
copy tarball to upper folder to make the tarball accessible to next step.
*/
// TODO: output the list of artifacts and their corresponding build steps.
// The tarball name into a step output: echo "filename=$TARBALL" >> "$GITHUB_OUTPUT"

if dryRun {
// 1. Retrieve artifacts and their hases from env variable SLSA_BUILD_ARTIFACTS.

// 2. Output the proveance metadata in a format:
/* METADATA={
"provenance1.intoto.jsonl":{
"artifact-1":{
"digest": {
"sha256": "abcdef"
},
"buildSteps":[]Steps{
"workingDir": string,
"env": map[string]string,
"command": []string
}
},
"artifact-2":{
"digest": {
"sha256": "abcdef"
},
"buildSteps":[]Steps{
"workingDir": string,
"env": map[string]string,
"command": []string
}
}
},
"provenance2.intoto.jsonl":{
"artifact-3":{
"digest": {
"sha256": "abcdef"
},
"buildSteps":[]Steps{
"workingDir": string,
"env": map[string]string,
"command": []string
}
},
}
}
*/
return nil
}

// 4. Output the list of artifacts in a format TBD (sha256sum?).

return nil
}
Loading