From 53e1d583aa16aec111fde10f91c55851501407c1 Mon Sep 17 00:00:00 2001 From: Daniel Mikusa Date: Mon, 22 Aug 2022 16:58:02 -0400 Subject: [PATCH 1/3] Support for monorepos in draft release This PR modifies the draft release handling code to support a monorepo where buildpack.toml is not at the root of the project. The default is still at the root of the project, but the path is configurable via environment variable set for the draft-release action. See the README.md changes for details. Signed-off-by: Daniel Mikusa --- README.md | 16 ++++++- drafts/drafts.go | 84 +++++++++++++++++++++++++----------- integration/drafts_test.go | 4 +- octo/descriptor.go | 7 +-- octo/draft_release.go | 11 +++++ octo/package_dependencies.go | 4 +- 6 files changed, 93 insertions(+), 33 deletions(-) diff --git a/README.md b/README.md index 106a6d7d..985b62e4 100644 --- a/README.md +++ b/README.md @@ -113,6 +113,8 @@ github: mappers: - '|tanzu-buildpacks|paketo-buildpacks|' - '|tanzu-buildpacks\/|pivotal-cf/tanzu-|' + buildpack_toml_paths: + 'tanzu-buildpacks/java-function': buildpacks/java ``` `github` is an object the describes the GitHub identity that the pipeline should use in the many places that GitHub API access is required. The token must be granted the `repo`, `write:packages`, and `workflow` scopes. @@ -121,6 +123,8 @@ github: The `mappers` list is optional and is used by the [Draft Release](#draft-release) workflow. Mappers are used to control how the action looks up metadata in Github for a buildpack. See the action notes for more information. +The `buildpack_toml_paths` map allows you to specify a map of buildpack ids and map them to the location within the Github repository where the buildpack.toml file exists. The path does not need to contain `buildpack.toml`, that is assumed and added automatically (although including it won't break anything). + #### `codeowners` ```yaml @@ -412,7 +416,9 @@ A Github token is strongly recommended. This is for fetching information from Gi The `package` and `version` entries are required. They are required to indiciate the specific buildpack that will be described. -A mapper will add additional permutations of Github project URIs to try. For example, if you have an image URI of `gcr.io/foo/bar:1.2.3` but the Github project is at `github.com/org/foo-bar` then you can add a mapper of `|foo\/bar|org/foo-bar|` to have `compute-artifact-dependencies` try both. You can specify as many mappers as you want, with each mapper generating a new URI for `compute-artifact-dependencies` to try. It will stop when it hits the first URL that successfully pulls back the `buildpack.toml` file. +A mapper will add additional permutations of Github project URIs to try. For example, if you have an image URI of `gcr.io/foo/bar:1.2.3` but the Github project is at `github.com/org/foo-bar` then you can add a mapper of `|foo\/bar|org/foo-bar|` to have `compute-artifact-dependencies` try both. You can specify as many mappers as you want, with each mapper generating a new URI for `compute-artifact-dependencies` to try. It will stop when it hits the first URL that successfully pulls back the `buildpack.toml` file.Mappers may use regexp capture groups and reference the captured value with `$` (ex: `$1`, `$2`, etc) in the regular expression. + +The location of the `buildpack.toml` file in the Github project is assumed to be at `/buildpack.toml`. If you have a monorepo with multiple buildpacks, you may adjust the location by setting an environment variable in the format `BP_TOML_PATH__=actual/path` where `` and `` are the organization and repo pulled out of the image URI. For example, `gcr.io/foo/bar:1.2.3`. The org is `foo` and the repo is `bar`. A leading `/` is not necessary, nor is `buildpack.toml` on the end. You may omit that to shorten the value and the action will automatically add `/buildpack.toml` to the end. For example, with an org of `vmware-tanzu` and a repo of `function-buildpacks-for-knative` the environment variable would be `BP_TOML_PATH_VMWARE_TANZU_FUNCTION_BUILDPACKS_FOR_KNATIVE='buildpacks/java/buildpack.toml`, which would be the same as `BP_TOML_PATH_VMWARE_TANZU_FUNCTION_BUILDPACKS_FOR_KNATIVE='buildpacks/java` since the trailing `buildpack.toml` is optional. The output is `artifact-reference-description`, which can be referenced by other jobs. @@ -432,7 +438,11 @@ The `draft-release` action is used to augment release notes generated by the `re The `draft-release` action needs to be able to look up and remotely fetch buildpack metadata. It does this by obtaining the `buildpack.toml` file for any dependent buildpacks from Github. This requires mapping the image URI to the Github project URI, which is not always one-to-one. As such, the action supports mappers. -A mapper will add additional permutations of Github project URIs to try. For example, if you have an image URI of `gcr.io/foo/bar:1.2.3` but the Github project is at `github.com/org/foo-bar` then you can add a mapper of `|foo\/bar|org/foo-bar|` to have `draft-release` try both. You can specify as many mappers as you want, with each mapper generating a new URI for `draft-release` to try. It will stop when it hits the first URL that successfully pulls back the `buildpack.toml` file. +A mapper will add additional permutations of Github project URIs to try. For example, if you have an image URI of `gcr.io/foo/bar:1.2.3` but the Github project is at `github.com/org/foo-bar` then you can add a mapper of `|foo\/bar|org/foo-bar|` to have `draft-release` try both. You can specify as many mappers as you want, with each mapper generating a new URI for `draft-release` to try. It will stop when it hits the first URL that successfully pulls back the `buildpack.toml` file. Mappers may use regexp capture groups and reference the captured value with `$` (ex: `$1`, `$2`, etc) in the regular expression. + +The location of the `buildpack.toml` file in the Github project is assumed to be at `/buildpack.toml`. If you have a monorepo with multiple buildpacks, you may adjust the location by setting an environment variable for the action in the format `BP_TOML_PATH__=actual/path` where `` and `` are the organization and repo pulled out of the image URI. For example, `gcr.io/foo/bar:1.2.3`. The org is `foo` and the repo is `bar`. + +A leading `/` on the environment variable is not necessary, nor is `buildpack.toml` on the end. You may omit both to shorten the value and the action adjust as required. For example, with an org of `tanzu-buildpacks` and a repo of `java-function` then the environment variable would be `BP_TOML_PATH_TANZU_BUILDPACKS_JAVA_FUNCTION='/buildpacks/java/buildpack.toml`, which would be the same as `BP_TOML_PATH_VMWARE_TANZU_FUNCTION_BUILDPACKS_FOR_KNATIVE='buildpacks/java` since the trailing `buildpack.toml` is optional. ```yaml uses: docker://ghcr.io/paketo-buildpacks/actions/draft-release:main @@ -444,6 +454,8 @@ with: release_id: ${{ steps.release-drafter.outputs.id }} release_name: ${{ steps.release-drafter.outputs.name }} release_tag_name: ${{ steps.release-drafter.outputs.tag_name }} + env: + BP_TOML_PATH_TANZU_BUILDPACKS_JAVA_FUNCTION: buildpacks/java ``` ### Foojay Dependency diff --git a/drafts/drafts.go b/drafts/drafts.go index 85a47bf7..847d2463 100644 --- a/drafts/drafts.go +++ b/drafts/drafts.go @@ -337,39 +337,46 @@ func (g GithubBuildpackLoader) LoadBuildpack(imgUri string) (Buildpack, error) { return Buildpack{}, fmt.Errorf("unable to map URIs\n%w", err) } - for _, uri := range uris { - uriPattern := regexp.MustCompile(`.*\/(.*)\/(.*):(.*)`) + origOrg, origRepo, _, err := parseRepoOrgVersionFromImageUri(imgUri) + if err != nil { + return Buildpack{}, fmt.Errorf("unable to parse original image uri\n%w", err) + } - parts := uriPattern.FindStringSubmatch(uri) - if len(parts) != 4 { - return Buildpack{}, fmt.Errorf("unable to parse %s, found %q", uri, parts) + for _, uri := range uris { + org, repo, version, err := parseRepoOrgVersionFromImageUri(uri) + if err != nil { + return Buildpack{}, fmt.Errorf("unable to parse image uri\n%w", err) } - org := parts[1] - repo := parts[2] - version := parts[3] - if regexp.MustCompile(`\d+\.\d+\.\d+`).MatchString(version) { - version = fmt.Sprintf("v%s", version) + paths, err := g.mapBuildpackTOMLPath(origOrg, origRepo) + if err != nil { + return Buildpack{}, fmt.Errorf("unable to map buildpack toml path\n%w", err) } - tomlBytes, err := g.fetchTOMLFile(org, repo, version, "/buildpack.toml") - if err != nil { - var apiErr *github.ErrorResponse - if errors.As(err, &apiErr) && apiErr.Response.StatusCode == 404 { - fmt.Println("skipping 404", apiErr) - continue - } - return Buildpack{}, fmt.Errorf("unable to fetch toml\n%w", err) + if regexp.MustCompile(`^\d+\.\d+\.\d+$`).MatchString(version) { + version = fmt.Sprintf("v%s", version) } - if len(tomlBytes) > 0 { - bp, err := loadBuildpackTOML(tomlBytes) + for _, path := range paths { + tomlBytes, err := g.fetchTOMLFile(org, repo, version, path) if err != nil { - return Buildpack{}, fmt.Errorf("unable to load buildpack toml from image\n%w", err) + var apiErr *github.ErrorResponse + if errors.As(err, &apiErr) && apiErr.Response.StatusCode == 404 { + fmt.Println("skipping 404", apiErr) + continue + } + return Buildpack{}, fmt.Errorf("unable to fetch toml\n%w", err) } - bp.Info.Version = parts[3] - return *bp, nil + if len(tomlBytes) > 0 { + bp, err := loadBuildpackTOML(tomlBytes) + if err != nil { + return Buildpack{}, fmt.Errorf("unable to load buildpack toml from image\n%w", err) + } + + bp.Info.Version = version + return *bp, nil + } } } @@ -442,8 +449,26 @@ func (g GithubBuildpackLoader) mapURIs(uri string) ([]string, error) { return possibilities, nil } +func (g GithubBuildpackLoader) mapBuildpackTOMLPath(org, repo string) ([]string, error) { + paths := []string{ + "/buildpack.toml", + } + + org = strings.ToUpper(strings.ReplaceAll(org, "-", "_")) + repo = strings.ToUpper(strings.ReplaceAll(repo, "-", "_")) + + if p, found := os.LookupEnv(fmt.Sprintf("BP_TOML_PATH_%s_%s", org, repo)); found { + if !strings.HasSuffix(p, "/buildpack.toml") { + p = fmt.Sprintf("%s/buildpack.toml", p) + } + return []string{p}, nil + } + + return paths, nil +} + func (g GithubBuildpackLoader) fetchTOMLFile(org, repo, version, path string) ([]byte, error) { - fmt.Println("Fetching from org:", org, "repo:", repo, "version:", version) + fmt.Println("Fetching from org:", org, "repo:", repo, "version:", version, "path:", path) body, _, err := g.GithubClient.Repositories.DownloadContents( context.Background(), org, @@ -581,3 +606,14 @@ func readBuildpackTOML(tarFile *os.File) ([]byte, error) { return []byte{}, fmt.Errorf("unable to find buildpack.toml in image") } + +func parseRepoOrgVersionFromImageUri(imgUri string) (string, string, string, error) { + uriPattern := regexp.MustCompile(`.*\/(.*)\/(.*):(.*)`) + + parts := uriPattern.FindStringSubmatch(imgUri) + if len(parts) != 4 { + return "", "", "", fmt.Errorf("unable to parse %s, found %q", imgUri, parts) + } + + return parts[1], parts[2], parts[3], nil +} diff --git a/integration/drafts_test.go b/integration/drafts_test.go index 6741d5f6..47feba7e 100644 --- a/integration/drafts_test.go +++ b/integration/drafts_test.go @@ -96,13 +96,13 @@ func testDrafts(t *testing.T, context spec.G, it spec.S) { GithubClient: github.NewClient(http.DefaultClient), }.LoadBuildpack("lasjdflaksdjfl") Expect(err).To(HaveOccurred()) - Expect(err).To(MatchError(HavePrefix("unable to parse lasjdflaksdjfl, found []"))) + Expect(err).To(MatchError(ContainSubstring("unable to parse lasjdflaksdjfl, found []"))) _, err = drafts.GithubBuildpackLoader{ GithubClient: github.NewClient(http.DefaultClient), }.LoadBuildpack("gcr.io/paketo-buildpacks/does-not-exist:main") Expect(err).To(HaveOccurred()) - Expect(err).To(MatchError(HavePrefix("unable to load buildpack.toml for gcr.io/paketo-buildpacks/does-not-exist:main"))) + Expect(err).To(MatchError(ContainSubstring("unable to load buildpack.toml for gcr.io/paketo-buildpacks/does-not-exist:main"))) }) }) } diff --git a/octo/descriptor.go b/octo/descriptor.go index 4f2d5365..e4ab640e 100644 --- a/octo/descriptor.go +++ b/octo/descriptor.go @@ -42,9 +42,10 @@ type Descriptor struct { } type GitHub struct { - Username string - Token string - Mappers []string + Username string + Token string + Mappers []string + BuildpackTOMLPaths map[string]string `yaml:"buildpack_toml_paths"` } type Action struct { diff --git a/octo/draft_release.go b/octo/draft_release.go index b3629cb4..74fe9866 100644 --- a/octo/draft_release.go +++ b/octo/draft_release.go @@ -19,6 +19,7 @@ package octo import ( "fmt" "path/filepath" + "strings" "github.com/paketo-buildpacks/pipeline-builder/octo/actions" "github.com/paketo-buildpacks/pipeline-builder/octo/actions/event" @@ -132,6 +133,11 @@ func ContributeDraftRelease(descriptor Descriptor) ([]Contribution, error) { draftReleaseContext[fmt.Sprintf("mapper_%d", i+1)] = mapper } + draftReleaseEnv := map[string]string{} + for key, val := range descriptor.GitHub.BuildpackTOMLPaths { + draftReleaseEnv[fmt.Sprintf("BP_TOML_PATH_%s", toEnvVar(key))] = val + } + j.Steps = append(j.Steps, actions.Step{ Uses: "actions/checkout@v3", @@ -140,6 +146,7 @@ func ContributeDraftRelease(descriptor Descriptor) ([]Contribution, error) { Name: "Update draft release with buildpack information", Uses: "docker://ghcr.io/paketo-buildpacks/actions/draft-release:main", With: draftReleaseContext, + Env: draftReleaseEnv, }, ) w.Jobs["update"] = j @@ -153,3 +160,7 @@ func ContributeDraftRelease(descriptor Descriptor) ([]Contribution, error) { return contributions, nil } + +func toEnvVar(key string) string { + return strings.ToUpper(strings.ReplaceAll(strings.ReplaceAll(strings.ReplaceAll(key, "-", "_"), "/", "_"), " ", "_")) +} diff --git a/octo/package_dependencies.go b/octo/package_dependencies.go index be3019fa..3fa862de 100644 --- a/octo/package_dependencies.go +++ b/octo/package_dependencies.go @@ -108,11 +108,11 @@ func findIds(bpOrders _package.BuildpackOrderGroups, dep _package.Dependency) (s for _, order := range bpOrders.Orders { for _, group := range order.Groups { endOfId := strings.Split(possibleBpId, "/")[1] - fmt.Println("group.Id", group.ID, "endOfId", endOfId, "group.Version", group.Version, "version", version) + // fmt.Println("group.Id", group.ID, "endOfId", endOfId, "group.Version", group.Version, "version", version) if strings.HasSuffix(group.ID, endOfId) && group.Version == version { pkgId := fmt.Sprintf("%s/%s", registry, possibleBpId) bpId := fmt.Sprintf("%s/%s", registry, group.ID) - fmt.Println("pkgId", pkgId, "bpId", bpId) + // fmt.Println("pkgId", pkgId, "bpId", bpId) return pkgId, bpId, nil } } From f5e34b66d02a5cc3fabb188abe86fbe40c971746 Mon Sep 17 00:00:00 2001 From: Daniel Mikusa Date: Tue, 23 Aug 2022 08:47:19 -0400 Subject: [PATCH 2/3] Update README.md Co-authored-by: David O'Sullivan <31728678+pivotal-david-osullivan@users.noreply.github.com> --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 985b62e4..3abeb530 100644 --- a/README.md +++ b/README.md @@ -418,7 +418,7 @@ The `package` and `version` entries are required. They are required to indiciate A mapper will add additional permutations of Github project URIs to try. For example, if you have an image URI of `gcr.io/foo/bar:1.2.3` but the Github project is at `github.com/org/foo-bar` then you can add a mapper of `|foo\/bar|org/foo-bar|` to have `compute-artifact-dependencies` try both. You can specify as many mappers as you want, with each mapper generating a new URI for `compute-artifact-dependencies` to try. It will stop when it hits the first URL that successfully pulls back the `buildpack.toml` file.Mappers may use regexp capture groups and reference the captured value with `$` (ex: `$1`, `$2`, etc) in the regular expression. -The location of the `buildpack.toml` file in the Github project is assumed to be at `/buildpack.toml`. If you have a monorepo with multiple buildpacks, you may adjust the location by setting an environment variable in the format `BP_TOML_PATH__=actual/path` where `` and `` are the organization and repo pulled out of the image URI. For example, `gcr.io/foo/bar:1.2.3`. The org is `foo` and the repo is `bar`. A leading `/` is not necessary, nor is `buildpack.toml` on the end. You may omit that to shorten the value and the action will automatically add `/buildpack.toml` to the end. For example, with an org of `vmware-tanzu` and a repo of `function-buildpacks-for-knative` the environment variable would be `BP_TOML_PATH_VMWARE_TANZU_FUNCTION_BUILDPACKS_FOR_KNATIVE='buildpacks/java/buildpack.toml`, which would be the same as `BP_TOML_PATH_VMWARE_TANZU_FUNCTION_BUILDPACKS_FOR_KNATIVE='buildpacks/java` since the trailing `buildpack.toml` is optional. +The location of the `buildpack.toml` file in the Github project is assumed to be at `/buildpack.toml`. If you have a monorepo with multiple buildpacks, you may adjust the location by setting an environment variable in the format `BP_TOML_PATH__=actual/path` where `` and `` are the organization and repo pulled out of the image URI. For example, `gcr.io/foo/bar:1.2.3`. The org is `foo` and the repo is `bar`. A leading `/` is not necessary, nor is `buildpack.toml` on the end. You may omit that to shorten the value and the action will automatically add `/buildpack.toml` to the end. For example, with an org of `vmware-tanzu` and a repo of `function-buildpacks-for-knative` the environment variable would be `BP_TOML_PATH_VMWARE_TANZU_FUNCTION_BUILDPACKS_FOR_KNATIVE=buildpacks/java/buildpack.toml`, which would be the same as `BP_TOML_PATH_VMWARE_TANZU_FUNCTION_BUILDPACKS_FOR_KNATIVE=buildpacks/java` since the trailing `buildpack.toml` is optional. The output is `artifact-reference-description`, which can be referenced by other jobs. From f88c8686045cb059c53b5d9b566fa0ac9261f7ac Mon Sep 17 00:00:00 2001 From: Daniel Mikusa Date: Tue, 23 Aug 2022 08:47:38 -0400 Subject: [PATCH 3/3] Update README.md Co-authored-by: David O'Sullivan <31728678+pivotal-david-osullivan@users.noreply.github.com> --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 3abeb530..c4449379 100644 --- a/README.md +++ b/README.md @@ -442,7 +442,7 @@ A mapper will add additional permutations of Github project URIs to try. For exa The location of the `buildpack.toml` file in the Github project is assumed to be at `/buildpack.toml`. If you have a monorepo with multiple buildpacks, you may adjust the location by setting an environment variable for the action in the format `BP_TOML_PATH__=actual/path` where `` and `` are the organization and repo pulled out of the image URI. For example, `gcr.io/foo/bar:1.2.3`. The org is `foo` and the repo is `bar`. -A leading `/` on the environment variable is not necessary, nor is `buildpack.toml` on the end. You may omit both to shorten the value and the action adjust as required. For example, with an org of `tanzu-buildpacks` and a repo of `java-function` then the environment variable would be `BP_TOML_PATH_TANZU_BUILDPACKS_JAVA_FUNCTION='/buildpacks/java/buildpack.toml`, which would be the same as `BP_TOML_PATH_VMWARE_TANZU_FUNCTION_BUILDPACKS_FOR_KNATIVE='buildpacks/java` since the trailing `buildpack.toml` is optional. +A leading `/` on the environment variable is not necessary, nor is `buildpack.toml` on the end. You may omit both to shorten the value and the action adjust as required. For example, with an org of `tanzu-buildpacks` and a repo of `java-function` then the environment variable would be `BP_TOML_PATH_TANZU_BUILDPACKS_JAVA_FUNCTION=/buildpacks/java/buildpack.toml`, which would be the same as `BP_TOML_PATH_TANZU_BUILDPACKS_JAVA_FUNCTION=buildpacks/java` since the trailing `buildpack.toml` is optional. ```yaml uses: docker://ghcr.io/paketo-buildpacks/actions/draft-release:main