feat: generate_library.sh with postprocessing (#1951)

* feat: add * generate gapic and proto folder * refactor utilities * add an action to verify * checkout googleapis-gen * setup repo name * add commit hash of googleapis-gen * change secret * change token * change to git clone * change user name * add input list * include resources folder in main * remove grpc version in `*ServiceGrpc.java` * change destination path * compare generation result with googleapis-gen * fix type in diff command * checkout repo using checkout action * checkout repos as nested repo * sparse checkout googleapis * Revert "sparse checkout googleapis" This reverts commit 3d612f8. * change library * change step name * add a readme * make grpc version optional * make protobuf version optional * checkout master branch, rather than a commit hash * allow snapshot version of generator * download snapshot of generator parent pom * update README * download generator and grpc using mvn * change error message * add maven central mirror * add comments in utilities * add comments * add an integration test * fail fast if no file is found * do not delete google/ * get protobuf version from WORKSPACE * add instructions on download `google/` from googleapis * add comments * update description of `destination_path` * update comments * download dependencies using `curl` * increase download time * remove comment * add samples directory in readme * remove prerequisite about `proto_path` * add explanation in prerequisite * add example to generate showcase * add a comment * wip adaptations * add owlbot.py template * run owlbot docker image * fix consolidate config * move owlbot call to its own function * move postprocessing logic * prepare integration test for gh workflow * fix local dev script * post-merge fixes * fix test script and IT * fix parent poms * start fixing samples problem * fix samples folder transfer * cleanup, prepare IT workflow * cleanup ii, sparse clone monorepo * delete preserve script * clean unnecessary lines * infer owlbot sha * add template file * remove newline from owlbot template * chore: newline correction * use stderr for error messages * fix script documentation * function comments * quoting variables * format constant * fix sparse checkout of monorepo * include location to googleapis sparse clone * remove unnecessary parent pom setting * remove consolidate_config.sh * exclude changelog and owlbot copy files from diff check * fixes after merge * include .github in monorepo sparse clone * restore `set_parent_pom.sh` * restore `consolidate_config.sh` * correct parameter resolution * use separate variable for version * postprocessing to use separate versions * remove old IT file * post-merge fixes * enable post-processing by default * post-merge fixes * post-merge fixes * post merge fixes * add script to compare poms * post-merge fixes * post-merge fixes ii * fix pom comparison * include pre-existing poms before running owlbot * change owlbot-staging suffix folder to run owlbot.py * fix newline removal in owlbot.py * split git diff command * enable tests for HW libraries * generate all hw libs except bigtable * all libraries passing * fix unit tests * repo metadata json logic cleanup * remove new library scripts * fix googleapis-gen tests * fix post-processing it * magic empty commit * correct conflict string * use os agnostic string replacement * comments and cleanup on postprocessing * cleanup of IT * temp: use custom gapic library name * use owl-bot-copy * remove api_version logic * remove custom_gapic_name in favor of owl-bot-copy * remove unnecessary new library flag * fix folder name test * remove unnecessary util function * remove unnecessary utils script dir var * rename postprocessing folder, apply_current_versions comment * fix postprocessing comments * correct popd folder name to its variable name * unnecessary sed command * skip generation if more versions coming * do not stage previous versions in owl-bot-staging * do not use custom repo metadatas * reset workspace folder * remove unnecessary owlbot yaml copy * modify readme * expand README instructions * examples for both pre and post processing * exclude new library owlbot.py template * do not process HW libraries * success message, folder navigation fix * set git author * add docker to workflow * lint fix * custom docker step for macos * do not postprocess showcase * os-dependent pom comparison * add python to workflow * explicit python version * add debugging output for compare_poms * correct xargs for macos * remove debug checkpoints * clean compare_poms.py * concise else logic * infer destination_path * add generation times * remove unused transport and include_samples from postprocessing * use versions.txt at root of owlbot postprocessor fs * modify success message * remove unused version processing script * remove owlbot_sha and repo_metadata args * use built-in docker images * manual install of docker ii * manual install of docker iii * manual install of docker iv * manual install of docker v * manual install of docker vi * manual install of docker vii * manual install of docker viii * manual install of docker ix * versions.txt as an argument * fix exit code in time tracking * fix readme * remove unused options * fix macos docker install * do not use cask to install docker * test custom user id in docker run * correct time tracking entry * change postprocessing file structture * move helper postprocess funcs to utilities.sh * add unit tests for postprocess utils * remove repository_path * fix workspace creation logic * fix readme * transfer from workspace to destination path * include folder structure for p.p. libs in readme * omit pre-processed folders * omit package-info.java * fix documentation argument order * fix preparation of copy-code source folder * add unit test for copy_directory_if_exists * fix wrong args to cp * change test monorepo folder names --------- Co-authored-by: JoeWang1127 <[email protected]>
googleapis · Oct 31, 2023 · 39b9f0e · 39b9f0e
1 parent f1ee04d
commit 39b9f0e
Show file tree

Hide file tree

Showing 13 changed files with 704 additions and 66 deletions.
diff --git a/.github/workflows/verify_library_generation.yaml b/.github/workflows/verify_library_generation.yaml
@@ -14,6 +14,7 @@ jobs:
       matrix:
         java: [ 8 ]
         os: [ ubuntu-22.04, macos-12 ]
+        post_processing: [ 'true', 'false' ]
     runs-on: ${{ matrix.os }}
     steps:
     - uses: actions/checkout@v3
@@ -22,11 +23,35 @@ jobs:
         java-version: ${{ matrix.java }}
         distribution: temurin
         cache: maven
+    - uses: actions/setup-python@v4
+      with:
+        python-version: '3.11'
+    - name: install docker (ubuntu)
+      if: matrix.os == 'ubuntu-22.04'
+      run: |
+        set -x
+        # install docker
+        sudo apt install containerd -y
+        sudo apt install -y docker.io docker-compose
+
+        # launch docker
+        sudo systemctl start docker
+    - name: install docker (macos)
+      if: matrix.os == 'macos-12'
+      run: |
+        brew update --preinstall
+        brew install docker docker-compose qemu 
+        brew upgrade qemu
+        colima start
+        docker run --user $(id -u):$(id -g) --rm hello-world
     - name: Run integration tests
       run: |
         set -x
+        git config --global user.email "[email protected]"
+        git config --global user.name "Github Workflow"
         library_generation/test/generate_library_integration_test.sh \
-        --googleapis_gen_url https://cloud-java-bot:${{ secrets.CLOUD_JAVA_BOT_GITHUB_TOKEN }}@github.com/googleapis/googleapis-gen.git
+        --googleapis_gen_url https://cloud-java-bot:${{ secrets.CLOUD_JAVA_BOT_GITHUB_TOKEN }}@github.com/googleapis/googleapis-gen.git \
+        --enable_postprocessing "${{ matrix.post_processing }}"
   unit_tests:
     strategy:
       matrix:

diff --git a/library_generation/README.md b/library_generation/README.md
@@ -21,6 +21,13 @@ In order to generate a GAPIC library, you need to pull `google/` from [googleapi
 and put it into `output` since protos in `google/` are likely referenced by 
 protos from which the library are generated.
 
+In order to generate a post-processed GAPIC library, you need to pull the
+original repository (i.e. google-cloud-java) and pass the monorepo as
+`destination_path` (e.g. `google-cloud-java/java-asset`). 
+This repository will be the source of truth for pre-existing
+pom.xml files, owlbot.py and .OwlBot.yaml files. See the option belows for
+custom postprocessed generations (e.g. custom `versions.txt` file).
+
 ## Parameters to run `generate_library.sh`
 
 You need to run the script with the following parameters.
@@ -40,7 +47,7 @@ Use `-d` or `--destination_path` to specify the value.
 
 Note that you do not need to create `$destination_path` beforehand.
 
-The directory structure of the generated library is
+The directory structure of the generated library _withtout_ postprocessing is
 ```
 $destination_path
   |_gapic-*
@@ -65,7 +72,35 @@ $destination_path
 ```
 You can't build the library as-is since it does not have `pom.xml` or `build.gradle`.
 To use the library, copy the generated files to the corresponding directory
-of a library repository, e.g., `google-cloud-java`.
+of a library repository, e.g., `google-cloud-java` or use the
+`enable_postprocessing` flag on top of a pre-existing generated library to
+produce the necessary pom files.
+
+For `asset/v1` the directory structure of the generated library _with_ postprocessing is
+```
+
+├── google-cloud-asset
+│   └── src
+│       ├── main
+│       │   ├── java
+│       │   └── resources
+│       └── test
+│           └── java
+├── google-cloud-asset-bom
+├── grpc-google-cloud-asset-v*
+│   └── src
+│       └── main
+│           └── java
+├── proto-google-cloud-asset-v*
+│   └── src
+│       └── main
+│           ├── java
+│           └── proto
+└── samples
+    └── snippets
+        └── generated
+
+```
 
 ### gapic_generator_version
 You can find the released version of gapic-generator-java in [maven central](https://repo1.maven.org/maven2/com/google/api/gapic-generator-java/).
@@ -150,8 +185,33 @@ Use `--include_samples` to specify the value.
 Choose the protoc binary type from https://github.com/protocolbuffers/protobuf/releases.
 Default is "linux-x86_64".
 
-## An example to generate a client library
+### enable_postprocessing (optional)
+Whether to enable the post-processing steps (usage of owlbot) in the generation
+of this library
+Default is "true".
+
+### versions_file (optional)
+It must point to a versions.txt file containing the versions the post-processed
+poms will have. It is required when `enable_postprocessing` is `"true"`
+
+
+## An example to generate a non post-processed client library
+```bash
+library_generation/generate_library.sh \
+-p google/cloud/confidentialcomputing/v1 \
+-d google-cloud-confidentialcomputing-v1-java \
+--gapic_generator_version 2.24.0 \
+--protobuf_version 23.2 \
+--grpc_version 1.55.1 \
+--gapic_additional_protos "google/cloud/common_resources.proto google/cloud/location/locations.proto" \
+--transport grpc+rest \
+--rest_numeric_enums true \
+--enable_postprocessing false \
+--include_samples true
 ```
+
+## An example to generate a library with postprocessing
+```bash
 library_generation/generate_library.sh \
 -p google/cloud/confidentialcomputing/v1 \
 -d google-cloud-confidentialcomputing-v1-java \
@@ -161,5 +221,7 @@ library_generation/generate_library.sh \
 --gapic_additional_protos "google/cloud/common_resources.proto google/cloud/location/locations.proto" \
 --transport grpc+rest \
 --rest_numeric_enums true \
+--enable_postprocessing true \
+--versions_file "path/to/versions.txt" \
 --include_samples true
 ```
diff --git a/library_generation/generate_library.sh b/library_generation/generate_library.sh
@@ -1,7 +1,6 @@
 #!/usr/bin/env bash
 
 set -eo pipefail
-set -x
 
 # parse input parameters
 while [[ $# -gt 0 ]]; do
@@ -61,10 +60,18 @@ case $key in
     include_samples="$2"
     shift
     ;;
+  --enable_postprocessing)
+    enable_postprocessing="$2"
+    shift
+    ;;
   --os_architecture)
     os_architecture="$2"
     shift
     ;;
+  --versions_file)
+    versions_file="$2"
+    shift
+    ;;
   *)
     echo "Invalid option: [$1]"
     exit 1
@@ -74,6 +81,7 @@ shift # past argument or value
 done
 
 script_dir=$(dirname "$(readlink -f "$0")")
+# source utility functions
 source "${script_dir}"/utilities.sh
 output_folder="$(get_output_folder)"
 
@@ -117,17 +125,20 @@ if [ -z "${include_samples}" ]; then
   include_samples="true"
 fi
 
+if [ -z "$enable_postprocessing" ]; then
+  enable_postprocessing="true"
+fi
+
 if [ -z "${os_architecture}" ]; then
   os_architecture=$(detect_os_architecture)
 fi
 
-
 mkdir -p "${output_folder}/${destination_path}"
 ##################### Section 0 #####################
 # prepare tooling
 #####################################################
 # the order of services entries in gapic_metadata.json is relevant to the
-# order of proto file, sort the proto files with respect to their name to
+# order of proto file, sort the proto files with respect to their bytes to
 # get a fixed order.
 folder_name=$(extract_folder_name "${destination_path}")
 pushd "${output_folder}"
@@ -137,7 +148,7 @@ case "${proto_path}" in
     find_depth="-maxdepth 1"
     ;;
 esac
-proto_files=$(find "${proto_path}" ${find_depth} -type f  -name "*.proto" | sort)
+proto_files=$(find "${proto_path}" ${find_depth} -type f  -name "*.proto" | LC_COLLATE=C sort)
 # include or exclude certain protos in grpc plugin and gapic generator java.
 case "${proto_path}" in
   "google/cloud")
@@ -280,5 +291,41 @@ popd # output_folder
 #####################################################
 pushd "${output_folder}/${destination_path}"
 rm -rf java_gapic_srcjar java_gapic_srcjar_raw.srcjar.zip java_grpc.jar java_proto.jar temp-codegen.srcjar
-popd
-set +x
+popd # destination path
+##################### Section 5 #####################
+# post-processing
+#####################################################
+if [ "${enable_postprocessing}" != "true" ];
+then
+  echo "post processing is disabled"
+  exit 0
+fi
+if [ -z "${versions_file}" ];then
+  echo "no versions.txt argument provided. Please provide one in order to enable post-processing"
+  exit 1
+fi
+workspace="${output_folder}/workspace"
+if [ -d "${workspace}" ]; then
+  rm -rdf "${workspace}"
+fi
+
+mkdir -p "${workspace}"
+
+bash -x "${script_dir}/postprocess_library.sh" "${workspace}" \
+  "${script_dir}" \
+  "${destination_path}" \
+  "${proto_path}" \
+  "${versions_file}" \
+  "${output_folder}"
+
+# for post-procesed libraries, remove pre-processed folders
+pushd "${output_folder}/${destination_path}"
+rm -rdf "proto-${folder_name}"
+rm -rdf "grpc-${folder_name}"
+rm -rdf "gapic-${folder_name}"
+if [ "${include_samples}" == "false" ]; then
+  rm -rdf "samples"
+fi
+popd # output_folder
+# move contents of the post-processed library into destination_path
+cp -r ${workspace}/* "${output_folder}/${destination_path}"
diff --git a/library_generation/postprocess_library.sh b/library_generation/postprocess_library.sh
@@ -0,0 +1,101 @@
+#!/bin/bash
+#
+# Main functions to interact with owlbot post-processor and postprocessing
+# scripts
+
+
+# Runs the owlbot post-processor docker image. The resulting post-processed
+# library gets stored in `${output_folder}/workspace`
+# Arguments
+# 1 - workspace: the location of the grpc,proto and gapic libraries to be
+# processed
+# 2 - scripts_root: location of the generation scripts
+# 3 - destination_path: used to transfer the raw grpc, proto and gapic libraries
+# 4 - proto_path: googleapis path of the library. This is used to prepare the
+# folder structure to run `owlbot-cli copy-code`
+# 5 - versions_file: path to file containing versions to be applied to the poms
+# 6 - output_folder: main workspace of the generation process
+
+workspace=$1
+scripts_root=$2
+destination_path=$3
+proto_path=$4
+versions_file=$5
+output_folder=$6
+
+source "${scripts_root}"/utilities.sh
+
+repository_root=$(echo "${destination_path}" | cut -d/ -f1)
+repo_metadata_json_path=$(get_repo_metadata_json "${destination_path}" "${output_folder}")
+owlbot_sha=$(get_owlbot_sha "${output_folder}" "${repository_root}")
+
+# read or infer owlbot sha
+
+cp "${repo_metadata_json_path}" "${workspace}"/.repo-metadata.json
+
+# call owl-bot-copy
+owlbot_staging_folder="${workspace}/owl-bot-staging"
+mkdir -p "${owlbot_staging_folder}"
+owlbot_postprocessor_image="gcr.io/cloud-devrel-public-resources/owlbot-java@sha256:${owlbot_sha}"
+
+
+
+# copy existing pom, owlbot and version files if the source of truth repo is present
+# pre-processed folders are ommited
+if [[ -d "${output_folder}/${destination_path}" ]]; then
+  rsync -avm \
+    --include='*/' \
+    --include='*.xml' \
+    --include='owlbot.py' \
+    --include='.OwlBot.yaml' \
+    --exclude='*' \
+    "${output_folder}/${destination_path}/" \
+    "${workspace}"
+fi
+
+echo 'Running owl-bot-copy'
+pre_processed_libs_folder="${output_folder}/pre-processed"
+# By default (thanks to generation templates), .OwlBot.yaml `deep-copy` section
+# references a wildcard pattern matching a folder
+# ending with `-java` at the leaf of proto_path. 
+mkdir -p "${pre_processed_libs_folder}/${proto_path}/generated-java"
+folder_name=$(extract_folder_name "${destination_path}")
+copy_directory_if_exists "${output_folder}/${destination_path}/proto-${folder_name}" \
+  "${pre_processed_libs_folder}/${proto_path}/generated-java/proto-google-cloud-${folder_name}"
+copy_directory_if_exists "${output_folder}/${destination_path}/grpc-${folder_name}" \
+  "${pre_processed_libs_folder}/${proto_path}/generated-java/grpc-google-cloud-${folder_name}"
+copy_directory_if_exists "${output_folder}/${destination_path}/gapic-${folder_name}" \
+  "${pre_processed_libs_folder}/${proto_path}/generated-java/gapic-google-cloud-${folder_name}"
+copy_directory_if_exists "${output_folder}/${destination_path}/samples" \
+  "${pre_processed_libs_folder}/${proto_path}/generated-java/samples"
+pushd "${pre_processed_libs_folder}"
+# create an empty repository so owl-bot-copy can process this as a repo
+# (cannot process non-git-repositories)
+git init
+git commit --allow-empty -m 'empty commit'
+popd # pre_processed_libs_folder
+
+docker run --rm \
+  --user $(id -u):$(id -g) \
+  -v "${workspace}:/repo" \
+  -v "${pre_processed_libs_folder}:/pre-processed-libraries" \
+  -w /repo \
+  --env HOME=/tmp \
+  gcr.io/cloud-devrel-public-resources/owlbot-cli:latest \
+  copy-code \
+  --source-repo-commit-hash=none \
+  --source-repo=/pre-processed-libraries \
+  --config-file=.OwlBot.yaml
+
+
+echo 'running owl-bot post-processor'
+versions_file_arg=""
+if [ -f "${versions_file}" ];then
+  versions_file_arg="-v ${versions_file}:/versions.txt"
+fi
+# run the postprocessor
+docker run --rm \
+  -v "${workspace}:/workspace" \
+  ${versions_file_arg} \
+  --user $(id -u):$(id -g) \
+  "${owlbot_postprocessor_image}"