Skip to content

Commit

Permalink
Unpin beam version and use default beam container images
Browse files Browse the repository at this point in the history
Requires that we explicitly specify *everything*
required by our pipelines in requirements.txt, but means
we no longer have to maintain a complex image here. The
complex image also leads to complex hard to debug problems,
like #90 (comment)
  • Loading branch information
yuvipanda committed Aug 19, 2023
1 parent 0013582 commit 01fc2a1
Show file tree
Hide file tree
Showing 3 changed files with 6 additions and 4 deletions.
6 changes: 4 additions & 2 deletions pangeo_forge_runner/commands/bake.py
Original file line number Diff line number Diff line change
Expand Up @@ -98,12 +98,14 @@ def _validate_job_name(self, proposal):
return proposal.value

container_image = Unicode(
# Provides apache_beam 2.48, which we pin to in setup.py
"quay.io/pangeo/forge:554675c",
"",
config=True,
help="""
Container image to use for this job.
Defaults to letting beam automatically figure out the image to use,
based on version of beam and python in use.
Should be accessible to whatever Beam runner is being used.
Note that some runners (like the local one) may not support this!
Expand Down
2 changes: 1 addition & 1 deletion setup.py
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@
"pangeo-forge-recipes>=0.9.2",
"escapism",
"traitlets",
"apache-beam[gcp]==2.48.0",
"apache-beam[gcp]",
],
entry_points={
"console_scripts": ["pangeo-forge-runner=pangeo_forge_runner.cli:main"]
Expand Down
2 changes: 1 addition & 1 deletion tests/integration/test_dataflow_integration.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@
def test_dataflow_integration():
pfr_version = parse_version(version("pangeo-forge-recipes"))
if pfr_version >= parse_version("0.10"):
recipe_version_ref = "0.10.x"
recipe_version_ref = "beam-bump"
else:
recipe_version_ref = "0.9.x"
bucket = "gs://pangeo-forge-runner-ci-testing"
Expand Down

0 comments on commit 01fc2a1

Please sign in to comment.