Skip to content

Commit

Permalink
Update Lambda and GCF docs for runtime inference (#21266)
Browse files Browse the repository at this point in the history 
This updates the Lambda and GCF docs (very belatedly) for the "new"
behaviours related to choosing platform specific code, like inference of
the `runtime` field (#19314), and associated complete platforms
(#19253).

This includes stripping out the `runtime` field from the "main" flow,
and adding an separate subsection for how to specify the runtime
explicitly.
  • Loading branch information
@huonw
huonw authored Aug 7, 2024
1 parent 6c41074 commit fa583d6
Show file tree
Hide file tree
Showing 2 changed files with 92 additions and 11 deletions.
53 changes: 46 additions & 7 deletions docs/docs/python/integrations/aws-lambda.mdx
View file
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ First, add your lambda function in a Python file like you would [normally do wit

Then, in your BUILD file, make sure that you have a `python_source` or `python_sources` target with the handler file included in the `sources` field. You can use [`pants tailor ::`](../../getting-started/initial-configuration.mdx#5-generate-build-files) to automate this.

Add a `python_aws_lambda_function` target and define the `runtime` and `handler` fields. The `runtime` should be one of the values from [https://docs.aws.amazon.com/lambda/latest/dg/lambda-python.html](https://docs.aws.amazon.com/lambda/latest/dg/lambda-python.html). The `handler` has the form `handler_file.py:handler_func`, which Pants will convert into a well-formed entry point. Alternatively, you can set `handler` to the format `path.to.module:handler_func`.
Add a `python_aws_lambda_function` target and define the `handler` fields. The `handler` has the form `handler_file.py:handler_func`, which Pants will convert into a well-formed entry point. Alternatively, you can set `handler` to the format `path.to.module:handler_func`.

For example:

Expand All @@ -43,7 +43,6 @@ python_sources(name="lib")

python_aws_lambda_function(
name="lambda",
runtime="python3.8",
# Pants will convert this to `project.lambda_example:example_handler`.
handler="lambda_example.py:example_handler",
)
Expand All @@ -66,6 +65,49 @@ Use [layout](../../../reference/targets/python_aws_lambda_function.mdx#layout) t
`file` / `files` targets will not be included in the built AWS Lambda artifacts because filesystem APIs like `open()` would not load them as expected. Instead, use the `resource` and `resources` target. See [Assets and archives](../../using-pants/assets-and-archives.mdx) for further explanation.
:::

### Specifying a runtime explicitly

When building an Lambda artifact, Pants and the underlying Pex tool need to know details about target runtime to be able to choose appropriate artifacts for third-party dependencies that have native code. These details can be inferred or provided in three ways, from highest precedence to lowest precedence:

1. An explicit value for [the `complete_platforms` field](../../../reference/targets/python_aws_lambda_function.mdx#complete_platforms). The "complete platforms" are the underlying source of truth.
```python title="BUILD"
file(name="lambda-platform", source="lambda-platform.json")

python_aws_lambda_function(
name="lambda",
handler="lambda_example.py:example_handler",
# Explicit complete platforms:
complete_platforms=[":lambda-platform"],
)
```
2. An explicit value for [the `runtime` field](../../../reference/targets/python_aws_lambda_function.mdx#runtime) and, optionally, [the `architecture` field](../../../reference/targets/python_aws_lambda_function.mdx#architecture): Pants uses these to pick an appropriate "complete platforms" value, from options that Pants has pre-packaged. These are static exports from docker images provided by AWS, relying on the environment being relatively stable. (If Pants doesn't have an appropriate "complete platforms" default built-in, you will be prompted to use option 1 above.)
```python title="BUILD"
python_aws_lambda_function(
name="lambda",
handler="lambda_example.py:example_handler",
# Explicit runtime, `complete_platforms` taken from Pants' built-in defaults:
runtime="python3.12",
# Override the default x86_64 architecture:
architecture="arm64",
)
```
3. Inferred from [the relevant interpreter constraints](../overview/interpreter-compatibility.mdx): the interpreter constraints may unambiguously imply a value for the `runtime` and thus `complete_platforms` fields. For example, `interpreter_constaints = ["==3.12.*"]` implies `runtime="python3.12"`. This only works with interpreter constraints that cover all patch versions of a given minor release series: `>=3.11,<3.13` is too wide (it covers both 3.11 and 3.12), while `==3.12.0` is too specific (AWS's `python3.12` runtime may not use that exact patch version). As with option 2, the architecture is `x86_64` by default, but can changed using the `architecture` field.

```toml tab={"label":"pants.toml"}
[python]
interpreter_constraints = ["==3.12.*"]
```
```python tab={"label":"project/BUILD"}
python_aws_lambda_function(
name="lambda",
handler="lambda_example.py:example_handler",
# `runtime` inferred and `complete_platforms` from built-in defaults,
# `architecture` defaults to x86_64, but can be overridden
)
```

This guide is written using the last option, with the default `x86_64` architecture, but you can add `runtime`, `complete_platforms` and/or `architecture` to any examples using the `python_aws_lambda_function` or `python_aws_lambda_layer` targets.

## Step 3: Run `package`

Now run `pants package` on your `python_aws_lambda_function` target to create a zipped file.
Expand Down Expand Up @@ -118,7 +160,6 @@ python_sources()

python_aws_lambda_function(
name="lambda",
runtime="python3.8",
handler="main.py:lambda_handler"
)

Expand Down Expand Up @@ -148,15 +189,13 @@ python_sources(name="lib")

python_aws_lambda_function(
name="function",
runtime="python3.8",
handler="lambda_example.py:example_handler",
# only include the sources, the boto3 requirement is packaged in `:layer`
include_requirements=False,
)

python_aws_lambda_layer(
name="layer",
runtime="python3.8"
# specify the handler file, and pants will automatically find its transitive dependencies
dependencies=["./lambda_example.py"],
# only include the boto3 requirement, any sources are packaged in `:function`
Expand Down Expand Up @@ -198,8 +237,8 @@ python_sources()
pex_binary(
name="lambda",
entry_point="lambda_example.py",
# specify an appropriate platform(s) for the targeted Lambda runtime (complete_platforms works too)
platforms=["linux_x86_64-cp39-cp39"],
# specify an appropriate platform for the targeted Lambda runtime:
complete_platforms=["path/to:platform-json-target"],
)
```

Expand Down
50 changes: 46 additions & 4 deletions docs/docs/python/integrations/google-cloud-functions.mdx
View file
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ First, add your Cloud function in a Python file like you would [normally do with

Then, in your BUILD file, make sure that you have a `python_source` or `python_sources` target with the handler file included in the `sources` field. You can use [`pants tailor ::`](../../getting-started/initial-configuration.mdx#5-generate-build-files) to automate this.

Add a `python_google_cloud_function` target and define the `runtime`, `handler`, and `type` fields. The `type` should be either `"event"` or `"http"`. The `runtime` should be one of the values from [https://cloud.google.com/functions/docs/concepts/python-runtime](https://cloud.google.com/functions/docs/concepts/python-runtime). The `handler` has the form `handler_file.py:handler_func`, which Pants will convert into a well-formed entry point. Alternatively, you can set `handler` to the format `path.to.module:handler_func`.
Add a `python_google_cloud_function` target and define `handler` and `type` fields. The `type` should be either `"event"` or `"http"`. The `handler` has the form `handler_file.py:handler_func`, which Pants will convert into a well-formed entry point. Alternatively, you can set `handler` to the format `path.to.module:handler_func`.

For example:

Expand All @@ -43,7 +43,6 @@ python_sources(name="lib")

python_google_cloud_function(
name="cloud_function",
runtime="python38",
# Pants will convert this to `project.google_cloud_function_example:example_handler`.
handler="google_cloud_function_example.py:example_handler",
type="event",
Expand All @@ -67,6 +66,49 @@ Use [layout](../../../reference/targets/python_google_cloud_function.mdx#layout)
`file` / `files` targets will not be included in the built Cloud Function because filesystem APIs like `open()` would not load them as expected. Instead, use the `resource` / `resources` target. See [Assets and archives](../../using-pants/assets-and-archives.mdx) for further explanation.
:::

### Specifying a runtime explicitly

When building an Cloud function artifact, Pants and the underlying Pex tool need to know details about target runtime to be able to choose appropriate artifacts for third-party dependencies that have native code. These details can be inferred or provided in three ways, from highest precedence to lowest precedence:

1. An explicit value for [the `complete_platforms` field](../../../reference/targets/python_google_cloud_function.mdx#complete_platforms). The "complete platforms" are the underlying source of truth.
```python title="BUILD"
file(name="gcf-platform", source="gcf-platform.json")

python_google_cloud_function(
name="cloud_function",
handler="google_cloud_function_example.py:example_handler",
type="event",
# Explicit complete platforms:
complete_platforms=[":gcf-platform"],
)
```
2. An explicit value for [the `runtime` field](../../../reference/targets/python_google_cloud_function.mdx#runtime): Pants uses this to pick an appropriate "complete platforms" value, from options that Pants has pre-packaged. These are static exports from docker images provided by GCP, relying on the environment being relatively stable. (If Pants doesn't have an appropriate "complete platforms" default built-in, you will be prompted to use option 1 above.)
```python title="BUILD"
python_google_cloud_function(
name="cloud_function",
handler="google_cloud_function_example.py:example_handler",
type="event",
# Explicit runtime, `complete_platforms` taken from Pants' built-in defaults:
runtime="python312",
)
```
3. Inferred from [the relevant interpreter constraints](../overview/interpreter-compatibility.mdx): the interpreter constraints may unambiguously imply a value for the `runtime` and thus `complete_platforms` fields. For example, `interpreter_constaints = ["==3.12.*"]` implies `runtime="python312"`. This only works with interpreter constraints that cover all patch versions of a given minor release series: `>=3.11,<3.13` is too wide (it covers both 3.11 and 3.12), while `==3.12.0` is too specific (GCF's `python312` runtime may not use that exact patch version).

```toml tab={"label":"pants.toml"}
[python]
interpreter_constraints = ["==3.12.*"]
```
```python tab={"label":"project/BUILD"}
python_google_cloud_function(
name="cloud_function",
handler="google_cloud_function_example.py:example_handler",
type="event",
# `runtime` inferred and `complete_platforms` from built-in defaults
)
```

This guide is written using the last option, but you can add `runtime` or `complete_platforms` to any examples using the `python_google_cloud_function` target.

## Step 3: Run `package`

Now run `pants package` on your `python_google_cloud_function` target to create a zipped file.
Expand Down Expand Up @@ -119,8 +161,8 @@ python_sources()
pex_binary(
name="gcf",
entry_point="gcf_example.py",
# specify an appropriate platform(s) for the targeted GCF runtime (complete_platforms works too)
platforms=["linux_x86_64-cp39-cp39"],
# specify an appropriate platform for the targeted GCF runtime:
complete_platforms=["path/to:platform-json-target"],
)
```

Expand Down

0 comments on commit fa583d6

Please sign in to comment.