chore: bake synthtool and owl-bot into library_generation docker image #2615

diegomarquezp · 2024-04-02T18:45:25Z

Context

In this PR

Bake synthtool into the docker image (we won't git clone it anymore)
Bake the owlbot CLI into the docker image (we won't use docker inside docker anymore)
Create a DEVELOPMENT.md file with instructions for local setup (example: install owlbot CLI locally)

Follow ups

modify google-cloud-java's workflows to correct the volume mapping being used

JoeWang1127 · 2024-04-22T13:11:15Z

We also need to modify workflow files in google-cloud-java after this pull request is merged because the docker command is changed.

JoeWang1127 · 2024-04-22T13:14:34Z

[Maybe out of scope of this PR] postprocess_library.sh is ~100 lines in this PR, should we refactor it to python script?

JoeWang1127 · 2024-04-23T15:41:22Z

.cloudbuild/library_generation/library_generation.Dockerfile

+RUN chmod -R a+rx /home/.nvm
+
+WORKDIR /workspace
+ENTRYPOINT [ "python", "/src/cli/entry_point.py", "generate" ]


Can the user run the image with different scripts/methods?

yes, I declared this ENTRYPOINT since it's the "official" one. If we want to run a different script, we can pass the --entrypoint /src/my_script.py flag

blakeli0 · 2024-04-25T21:53:35Z

.cloudbuild/library_generation/library_generation.Dockerfile

+
+# set dummy git credentials for the empty commit used in postprocessing
+# we use system so all users using the container will use this configuration
+RUN git config --system user.email "[email protected]"


I think the only reason we need git config is from the post processing here? If it is, I think we should look into not initialize a Git repo, as a follow up enhancement though.

yes, but the root cause is that the owl-bot CLI assumes the source folder (the one we generate and arrange) is a git repository. Extra points to process owlbot yamls using a custom script.

blakeli0 · 2024-04-25T21:56:15Z

.cloudbuild/library_generation/library_generation.Dockerfile

+
+# install synthtool
+WORKDIR /tools
+RUN git clone https://github.com/googleapis/synthtool


I think the only reason we need synthtool is java.py and its dependencies?

Yes, mainly the common_templates function. We could in theory move java.py and that function to sdk-platform

library_generation/DEVELOPMENT.md

blakeli0 · 2024-04-26T18:37:59Z

.cloudbuild/library_generation/library_generation.Dockerfile

 RUN apt-get update && apt-get install -y \
-	unzip openjdk-17-jdk rsync maven jq \
+	unzip openjdk-17-jdk rsync maven jq less vim \


I don't see less and vim being used, I guess the reason we need them is for testing purpose?

It's just for inspecting the container files in case things go wrong during local development. Do we want to only keep the production dependencies?

For local development, I think you can mention them in the development.md. No need to install it in production.

Makes sense, I removed these tools from the image and added a small section in the development guide.

Co-authored-by: Joe Wang <[email protected]>

diegomarquezp · 2024-04-29T22:24:51Z

We also need to modify workflow files in google-cloud-java after this pull request is merged because the docker command is changed.

Yes, indeed. This is considered in the follow up section of the PR description. I will do it right after we merge this

diegomarquezp · 2024-04-29T22:25:46Z

[Maybe out of scope of this PR] postprocess_library.sh is ~100 lines in this PR, should we refactor it to python script?

Yes, my only concern is that xtrace is really useful for debugging the shell scripts. I may give it a try as well as including different levels of logging for some of the scripts

…ansfer-synthtool-java

sonarqubecloud · 2024-04-30T20:01:36Z

Quality Gate passed for 'gapic-generator-java-root'

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
No data about Duplication

See analysis details on SonarCloud

sonarqubecloud · 2024-04-30T20:04:45Z

Quality Gate passed for 'java_showcase_integration_tests'

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
No data about Duplication

See analysis details on SonarCloud

blakeli0 · 2024-04-30T21:41:35Z

.cloudbuild/library_generation/library_generation.Dockerfile

 RUN apt-get update && apt-get install -y \
 	unzip openjdk-17-jdk rsync maven jq \
 	&& apt-get clean

-COPY library_generation /src
-
+# use python 3.11 (the base image has several python versions; here we define the default one)


Because the base image cloud-devrel-public-resources still uses 3.11 so we have to use 3.11? Who maintains cloud-devrel-public-resources?

so we have to use 3.11?

We could use a higher version too, or even a lower one if it's still compatible with the dependencies

who maintains cloud-devrel-public-resources?

Judging from a documentation entry in the cloud client libraries g3doc, I think it's the Cloud Client Libraries team
Maybe we can extract the base Dockerfile and prepend it to ours to eliminate this dependency.

This intends to adapt the config yamle to the changes made in googleapis/sdk-platform-java#2615

…ig (#10833) This intends to adapt the config yamle to the changes made in googleapis/sdk-platform-java#2615

In this PR: - Remove generated files after running tests. Note that files generated by `docker run` have root permission, thus can't be removed by this PR. This issue will be resolved in #2615.

#2615) ## Context [Milestone 7 of Hermetic Build](https://docs.google.com/document/d/1v-sJBmdNEVBRryL8n90XK8OIiqVphaJemXVhANw14Jk/edit?pli=1&resourcekey=0-QGUb2do-JBlDWKvptWBp_g&tab=t.0#bookmark=id.bkbj04ib2d4n) ## In this PR - Bake synthtool into the docker image (we won't `git clone` it anymore) - Bake the owlbot CLI into the docker image (we won't use docker inside docker anymore) - Create a DEVELOPMENT.md file with instructions for local setup (example: install owlbot CLI locally) ## Follow ups - [ ] modify google-cloud-java's workflows to correct the volume mapping being used --------- Co-authored-by: Joe Wang <[email protected]>

diegomarquezp added 4 commits April 2, 2024 18:24

chore: bake synthtool into library_generation docker image

0b65f15

remove synthtool usage

a6bb064

do not use config file

d4f27e2

bake synthtool in dockerfile

19a0b04

product-auto-label bot added the size: m Pull request size is medium. label Apr 2, 2024

diegomarquezp added 4 commits April 2, 2024 20:18

bake copy-code into the image

bc693b5

remove unnecessary SHELL statement

f335cbb

add info for installing synthtool

aab50ef

modify owl-bot usage in postprocess_library

f97cdca

diegomarquezp changed the title ~~chore: bake synthtool into library_generation docker image~~ chore: bake synthtool and owl-bot into library_generation docker image Apr 2, 2024

diegomarquezp changed the title ~~chore: bake synthtool and owl-bot into library_generation docker image~~ chore: [wip] bake synthtool and owl-bot into library_generation docker image Apr 2, 2024

diegomarquezp added 9 commits April 10, 2024 15:16

Merge remote-tracking branch 'origin/main' into transfer-synthtool-java

d68c14a

fix synthtool installation

eca0d92

change permissions to script folder

6aa8533

assume location of generation_config.yaml

53c5dea

fix permissions

cc4309b

fix path to config yaml

7a4b6b8

use entrypoint for docker image

4838d5b

format

1b01713

finish development guide

c149461

product-auto-label bot added size: l Pull request size is large. and removed size: m Pull request size is medium. labels Apr 11, 2024

remove olwbot_cli usage

4ee1692

JoeWang1127 mentioned this pull request Apr 16, 2024

chore: remove generated files in tests #2644

Merged

diegomarquezp added 6 commits April 17, 2024 15:22

add entrypoint for docker image

e77bacc

manually set HOME var in owlbot

9120c94

Merge remote-tracking branch 'origin/main' into transfer-synthtool-java

3d18426

post-merge cleanup

427c564

fix parameters

7b79763

fix synthtool sha

240742a

JoeWang1127 reviewed Apr 23, 2024

View reviewed changes

blakeli0 reviewed Apr 25, 2024

View reviewed changes

blakeli0 reviewed Apr 26, 2024

View reviewed changes

blakeli0 requested review from suztomo and chingor13 April 26, 2024 20:11

Update library_generation/DEVELOPMENT.md

7ec0f4c

Co-authored-by: Joe Wang <[email protected]>

diegomarquezp added 4 commits April 29, 2024 23:28

docker instructions, pwd blurb, unit tests instructions

4399bfe

Merge remote-tracking branch 'origin/main' into transfer-synthtool-java

ef105e7

Merge remote-tracking branch 'origin/transfer-synthtool-java' into tr…

b1d3fc3

…ansfer-synthtool-java

restore generate repo

1df1aa1

diegomarquezp mentioned this pull request Apr 30, 2024

chore: bake protoc into the hermetic build docker image #2707

Merged

move text editor installation as a readme note

bf95938

diegomarquezp requested review from blakeli0 and JoeWang1127 April 30, 2024 19:54

JoeWang1127 approved these changes Apr 30, 2024

View reviewed changes

blakeli0 approved these changes Apr 30, 2024

View reviewed changes

diegomarquezp merged commit 16aec03 into main May 1, 2024
44 checks passed

diegomarquezp deleted the transfer-synthtool-java branch May 1, 2024 17:14

diegomarquezp mentioned this pull request May 1, 2024

chore: update hermetic build docker volume usage googleapis/google-cloud-java#10770

Closed

diegomarquezp added a commit to googleapis/google-cloud-java that referenced this pull request May 13, 2024

chore: remove synthtool_committish and owlbot_cli_image from gen config

b52c556

This intends to adapt the config yamle to the changes made in googleapis/sdk-platform-java#2615

diegomarquezp mentioned this pull request May 13, 2024

chore: remove synthtool_committish and owlbot_cli_image from gen config googleapis/google-cloud-java#10833

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

chore: bake synthtool and owl-bot into library_generation docker image #2615

chore: bake synthtool and owl-bot into library_generation docker image #2615

diegomarquezp commented Apr 2, 2024 •

edited

Loading

JoeWang1127 commented Apr 22, 2024

JoeWang1127 commented Apr 22, 2024 •

edited

Loading

JoeWang1127 Apr 23, 2024

diegomarquezp Apr 29, 2024

blakeli0 Apr 25, 2024

diegomarquezp Apr 29, 2024

blakeli0 Apr 25, 2024

diegomarquezp Apr 29, 2024

blakeli0 Apr 26, 2024

diegomarquezp Apr 29, 2024

JoeWang1127 Apr 30, 2024

diegomarquezp Apr 30, 2024

diegomarquezp commented Apr 29, 2024

diegomarquezp commented Apr 29, 2024

sonarqubecloud bot commented Apr 30, 2024

sonarqubecloud bot commented Apr 30, 2024

blakeli0 Apr 30, 2024

diegomarquezp May 1, 2024 •

edited

Loading

chore: bake synthtool and owl-bot into library_generation docker image #2615

chore: bake synthtool and owl-bot into library_generation docker image #2615

Conversation

diegomarquezp commented Apr 2, 2024 • edited Loading

Context

In this PR

Follow ups

JoeWang1127 commented Apr 22, 2024

JoeWang1127 commented Apr 22, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

diegomarquezp commented Apr 29, 2024

diegomarquezp commented Apr 29, 2024

sonarqubecloud bot commented Apr 30, 2024

Quality Gate passed for 'gapic-generator-java-root'

sonarqubecloud bot commented Apr 30, 2024

Quality Gate passed for 'java_showcase_integration_tests'

Choose a reason for hiding this comment

diegomarquezp May 1, 2024 • edited Loading

Choose a reason for hiding this comment

diegomarquezp commented Apr 2, 2024 •

edited

Loading

JoeWang1127 commented Apr 22, 2024 •

edited

Loading

diegomarquezp May 1, 2024 •

edited

Loading