Draft release action to support monorepo

README.md

@@ -113,6 +113,8 @@ github:
   mappers:
   - '|tanzu-buildpacks|paketo-buildpacks|'
   - '|tanzu-buildpacks\/|pivotal-cf/tanzu-|'
+  buildpack_toml_paths:
+    'tanzu-buildpacks/java-function': buildpacks/java
 ```
 
 `github` is an object the describes the GitHub identity that the pipeline should use in the many places that GitHub API access is required.  The token must be granted the `repo`, `write:packages`, and `workflow` scopes.
@@ -121,6 +123,8 @@ github:
 
 The `mappers` list is optional and is used by the [Draft Release](#draft-release) workflow. Mappers are used to control how the action looks up metadata in Github for a buildpack. See the action notes for more information.
 
+The `buildpack_toml_paths` map allows you to specify a map of buildpack ids and map them to the location within the Github repository where the buildpack.toml file exists. The path does not need to contain `buildpack.toml`, that is assumed and added automatically (although including it won't break anything).
+
 #### `codeowners`
 
 ```yaml
@@ -412,7 +416,9 @@ A Github token is strongly recommended. This is for fetching information from Gi
 
 The `package` and `version` entries are required. They are required to indiciate the specific buildpack that will be described.
 
-A mapper will add additional permutations of Github project URIs to try. For example, if you have an image URI of `gcr.io/foo/bar:1.2.3` but the Github project is at `github.com/org/foo-bar` then you can add a mapper of `|foo\/bar|org/foo-bar|` to have `compute-artifact-dependencies` try both. You can specify as many mappers as you want, with each mapper generating a new URI for `compute-artifact-dependencies` to try. It will stop when it hits the first URL that successfully pulls back the `buildpack.toml` file.
+A mapper will add additional permutations of Github project URIs to try. For example, if you have an image URI of `gcr.io/foo/bar:1.2.3` but the Github project is at `github.com/org/foo-bar` then you can add a mapper of `|foo\/bar|org/foo-bar|` to have `compute-artifact-dependencies` try both. You can specify as many mappers as you want, with each mapper generating a new URI for `compute-artifact-dependencies` to try. It will stop when it hits the first URL that successfully pulls back the `buildpack.toml` file.Mappers may use regexp capture groups and reference the captured value with `$<num>` (ex: `$1`, `$2`, etc) in the regular expression.
+
+The location of the `buildpack.toml` file in the Github project is assumed to be at `/buildpack.toml`. If you have a monorepo with multiple buildpacks, you may adjust the location by setting an environment variable in the format `BP_TOML_PATH_<org>_<repo>=actual/path` where `<org>` and `<repo>` are the organization and repo pulled out of the image URI. For example, `gcr.io/foo/bar:1.2.3`. The org is `foo` and the repo is `bar`. A leading `/` is not necessary, nor is `buildpack.toml` on the end. You may omit that to shorten the value and the action will automatically add `/buildpack.toml` to the end. For example, with an org of `vmware-tanzu` and a repo of `function-buildpacks-for-knative` the environment variable would be `BP_TOML_PATH_VMWARE_TANZU_FUNCTION_BUILDPACKS_FOR_KNATIVE=buildpacks/java/buildpack.toml`, which would be the same as `BP_TOML_PATH_VMWARE_TANZU_FUNCTION_BUILDPACKS_FOR_KNATIVE=buildpacks/java` since the trailing `buildpack.toml` is optional.
 
 The output is `artifact-reference-description`, which can be referenced by other jobs.
 
@@ -432,7 +438,11 @@ The `draft-release` action is used to augment release notes generated by the `re
 
 The `draft-release` action needs to be able to look up and remotely fetch buildpack metadata. It does this by obtaining the `buildpack.toml` file for any dependent buildpacks from Github. This requires mapping the image URI to the Github project URI, which is not always one-to-one. As such, the action supports mappers.
 
-A mapper will add additional permutations of Github project URIs to try. For example, if you have an image URI of `gcr.io/foo/bar:1.2.3` but the Github project is at `github.com/org/foo-bar` then you can add a mapper of `|foo\/bar|org/foo-bar|` to have `draft-release` try both. You can specify as many mappers as you want, with each mapper generating a new URI for `draft-release` to try. It will stop when it hits the first URL that successfully pulls back the `buildpack.toml` file.
+A mapper will add additional permutations of Github project URIs to try. For example, if you have an image URI of `gcr.io/foo/bar:1.2.3` but the Github project is at `github.com/org/foo-bar` then you can add a mapper of `|foo\/bar|org/foo-bar|` to have `draft-release` try both. You can specify as many mappers as you want, with each mapper generating a new URI for `draft-release` to try. It will stop when it hits the first URL that successfully pulls back the `buildpack.toml` file. Mappers may use regexp capture groups and reference the captured value with `$<num>` (ex: `$1`, `$2`, etc) in the regular expression.
+
+The location of the `buildpack.toml` file in the Github project is assumed to be at `/buildpack.toml`. If you have a monorepo with multiple buildpacks, you may adjust the location by setting an environment variable for the action in the format `BP_TOML_PATH_<org>_<repo>=actual/path` where `<org>` and `<repo>` are the organization and repo pulled out of the image URI. For example, `gcr.io/foo/bar:1.2.3`. The org is `foo` and the repo is `bar`. 
+
+A leading `/` on the environment variable is not necessary, nor is `buildpack.toml` on the end. You may omit both to shorten the value and the action adjust as required. For example, with an org of `tanzu-buildpacks` and a repo of `java-function` then the environment variable would be `BP_TOML_PATH_TANZU_BUILDPACKS_JAVA_FUNCTION=/buildpacks/java/buildpack.toml`, which would be the same as `BP_TOML_PATH_TANZU_BUILDPACKS_JAVA_FUNCTION=buildpacks/java` since the trailing `buildpack.toml` is optional.
 
 ```yaml
 uses: docker://ghcr.io/paketo-buildpacks/actions/draft-release:main
@@ -444,6 +454,8 @@ with:
   release_id:       ${{ steps.release-drafter.outputs.id }}
   release_name:     ${{ steps.release-drafter.outputs.name }}
   release_tag_name: ${{ steps.release-drafter.outputs.tag_name }}
+  env:
+    BP_TOML_PATH_TANZU_BUILDPACKS_JAVA_FUNCTION: buildpacks/java
 ```
 
 ### Foojay Dependency

drafts/drafts.go

@@ -337,39 +337,46 @@ func (g GithubBuildpackLoader) LoadBuildpack(imgUri string) (Buildpack, error) {
 		return Buildpack{}, fmt.Errorf("unable to map URIs\n%w", err)
 	}
 
-	for _, uri := range uris {
-		uriPattern := regexp.MustCompile(`.*\/(.*)\/(.*):(.*)`)
+	origOrg, origRepo, _, err := parseRepoOrgVersionFromImageUri(imgUri)
+	if err != nil {
+		return Buildpack{}, fmt.Errorf("unable to parse original image uri\n%w", err)
+	}
 
-		parts := uriPattern.FindStringSubmatch(uri)
-		if len(parts) != 4 {
-			return Buildpack{}, fmt.Errorf("unable to parse %s, found %q", uri, parts)
+	for _, uri := range uris {
+		org, repo, version, err := parseRepoOrgVersionFromImageUri(uri)
+		if err != nil {
+			return Buildpack{}, fmt.Errorf("unable to parse image uri\n%w", err)
 		}
 
-		org := parts[1]
-		repo := parts[2]
-		version := parts[3]
-		if regexp.MustCompile(`\d+\.\d+\.\d+`).MatchString(version) {
-			version = fmt.Sprintf("v%s", version)
+		paths, err := g.mapBuildpackTOMLPath(origOrg, origRepo)
+		if err != nil {
+			return Buildpack{}, fmt.Errorf("unable to map buildpack toml path\n%w", err)
 		}
 
-		tomlBytes, err := g.fetchTOMLFile(org, repo, version, "/buildpack.toml")
-		if err != nil {
-			var apiErr *github.ErrorResponse
-			if errors.As(err, &apiErr) && apiErr.Response.StatusCode == 404 {
-				fmt.Println("skipping 404", apiErr)
-				continue
-			}
-			return Buildpack{}, fmt.Errorf("unable to fetch toml\n%w", err)
+		if regexp.MustCompile(`^\d+\.\d+\.\d+$`).MatchString(version) {
+			version = fmt.Sprintf("v%s", version)
 		}
 
-		if len(tomlBytes) > 0 {
-			bp, err := loadBuildpackTOML(tomlBytes)
+		for _, path := range paths {
+			tomlBytes, err := g.fetchTOMLFile(org, repo, version, path)
 			if err != nil {
-				return Buildpack{}, fmt.Errorf("unable to load buildpack toml from image\n%w", err)
+				var apiErr *github.ErrorResponse
+				if errors.As(err, &apiErr) && apiErr.Response.StatusCode == 404 {
+					fmt.Println("skipping 404", apiErr)
+					continue
+				}
+				return Buildpack{}, fmt.Errorf("unable to fetch toml\n%w", err)
 			}
 
-			bp.Info.Version = parts[3]
-			return *bp, nil
+			if len(tomlBytes) > 0 {
+				bp, err := loadBuildpackTOML(tomlBytes)
+				if err != nil {
+					return Buildpack{}, fmt.Errorf("unable to load buildpack toml from image\n%w", err)
+				}
+
+				bp.Info.Version = version
+				return *bp, nil
+			}
 		}
 	}
 
@@ -442,8 +449,26 @@ func (g GithubBuildpackLoader) mapURIs(uri string) ([]string, error) {
 	return possibilities, nil
 }
 
+func (g GithubBuildpackLoader) mapBuildpackTOMLPath(org, repo string) ([]string, error) {
+	paths := []string{
+		"/buildpack.toml",
+	}
+
+	org = strings.ToUpper(strings.ReplaceAll(org, "-", "_"))
+	repo = strings.ToUpper(strings.ReplaceAll(repo, "-", "_"))
+
+	if p, found := os.LookupEnv(fmt.Sprintf("BP_TOML_PATH_%s_%s", org, repo)); found {
+		if !strings.HasSuffix(p, "/buildpack.toml") {
+			p = fmt.Sprintf("%s/buildpack.toml", p)
+		}
+		return []string{p}, nil
+	}
+
+	return paths, nil
+}
+
 func (g GithubBuildpackLoader) fetchTOMLFile(org, repo, version, path string) ([]byte, error) {
-	fmt.Println("Fetching from org:", org, "repo:", repo, "version:", version)
+	fmt.Println("Fetching from org:", org, "repo:", repo, "version:", version, "path:", path)
 	body, _, err := g.GithubClient.Repositories.DownloadContents(
 		context.Background(),
 		org,
@@ -581,3 +606,14 @@ func readBuildpackTOML(tarFile *os.File) ([]byte, error) {
 
 	return []byte{}, fmt.Errorf("unable to find buildpack.toml in image")
 }
+
+func parseRepoOrgVersionFromImageUri(imgUri string) (string, string, string, error) {
+	uriPattern := regexp.MustCompile(`.*\/(.*)\/(.*):(.*)`)
+
+	parts := uriPattern.FindStringSubmatch(imgUri)
+	if len(parts) != 4 {
+		return "", "", "", fmt.Errorf("unable to parse %s, found %q", imgUri, parts)
+	}
+
+	return parts[1], parts[2], parts[3], nil
+}

integration/drafts_test.go

@@ -96,13 +96,13 @@ func testDrafts(t *testing.T, context spec.G, it spec.S) {
 				GithubClient: github.NewClient(http.DefaultClient),
 			}.LoadBuildpack("lasjdflaksdjfl")
 			Expect(err).To(HaveOccurred())
-			Expect(err).To(MatchError(HavePrefix("unable to parse lasjdflaksdjfl, found []")))
+			Expect(err).To(MatchError(ContainSubstring("unable to parse lasjdflaksdjfl, found []")))
 
 			_, err = drafts.GithubBuildpackLoader{
 				GithubClient: github.NewClient(http.DefaultClient),
 			}.LoadBuildpack("gcr.io/paketo-buildpacks/does-not-exist:main")
 			Expect(err).To(HaveOccurred())
-			Expect(err).To(MatchError(HavePrefix("unable to load buildpack.toml for gcr.io/paketo-buildpacks/does-not-exist:main")))
+			Expect(err).To(MatchError(ContainSubstring("unable to load buildpack.toml for gcr.io/paketo-buildpacks/does-not-exist:main")))
 		})
 	})
 }

octo/descriptor.go

@@ -42,9 +42,10 @@ type Descriptor struct {
 }
 
 type GitHub struct {
-	Username string
-	Token    string
-	Mappers  []string
+	Username           string
+	Token              string
+	Mappers            []string
+	BuildpackTOMLPaths map[string]string `yaml:"buildpack_toml_paths"`
 }
 
 type Action struct {

octo/draft_release.go

@@ -19,6 +19,7 @@ package octo
 import (
 	"fmt"
 	"path/filepath"
+	"strings"
 
 	"github.com/paketo-buildpacks/pipeline-builder/octo/actions"
 	"github.com/paketo-buildpacks/pipeline-builder/octo/actions/event"
@@ -132,6 +133,11 @@ func ContributeDraftRelease(descriptor Descriptor) ([]Contribution, error) {
 			draftReleaseContext[fmt.Sprintf("mapper_%d", i+1)] = mapper
 		}
 
+		draftReleaseEnv := map[string]string{}
+		for key, val := range descriptor.GitHub.BuildpackTOMLPaths {
+			draftReleaseEnv[fmt.Sprintf("BP_TOML_PATH_%s", toEnvVar(key))] = val
+		}
+
 		j.Steps = append(j.Steps,
 			actions.Step{
 				Uses: "actions/checkout@v3",
@@ -140,6 +146,7 @@ func ContributeDraftRelease(descriptor Descriptor) ([]Contribution, error) {
 				Name: "Update draft release with buildpack information",
 				Uses: "docker://ghcr.io/paketo-buildpacks/actions/draft-release:main",
 				With: draftReleaseContext,
+				Env:  draftReleaseEnv,
 			},
 		)
 		w.Jobs["update"] = j
@@ -153,3 +160,7 @@ func ContributeDraftRelease(descriptor Descriptor) ([]Contribution, error) {
 
 	return contributions, nil
 }
+
+func toEnvVar(key string) string {
+	return strings.ToUpper(strings.ReplaceAll(strings.ReplaceAll(strings.ReplaceAll(key, "-", "_"), "/", "_"), " ", "_"))
+}

octo/package_dependencies.go

@@ -108,11 +108,11 @@ func findIds(bpOrders _package.BuildpackOrderGroups, dep _package.Dependency) (s
 		for _, order := range bpOrders.Orders {
 			for _, group := range order.Groups {
 				endOfId := strings.Split(possibleBpId, "/")[1]
-				fmt.Println("group.Id", group.ID, "endOfId", endOfId, "group.Version", group.Version, "version", version)
+				// fmt.Println("group.Id", group.ID, "endOfId", endOfId, "group.Version", group.Version, "version", version)
 				if strings.HasSuffix(group.ID, endOfId) && group.Version == version {
 					pkgId := fmt.Sprintf("%s/%s", registry, possibleBpId)
 					bpId := fmt.Sprintf("%s/%s", registry, group.ID)
-					fmt.Println("pkgId", pkgId, "bpId", bpId)
+					// fmt.Println("pkgId", pkgId, "bpId", bpId)
 					return pkgId, bpId, nil
 				}
 			}