Call for testing/feedback: Improvements to AWS Lambda and Google Cloud Functions support in 2.17 and 2.18

[huonw]

In Pants 2.17 and 2.18 (and 2.19), there's a bunch of improvements to Pants' support for building AWS Lambda and Google Cloud Function artifacts for Python code.

I've mostly tested these locally and in my own work repo, but it'd be great for others to experiment with them: the earlier we identify issues and improvements, the better!

The changes are:

1. 2.17:
   1. a new static layout is available for both python_awslambda and python_google_cloud_function, with dependencies and code all at the top level, without requiring the dynamic PEX initialisation (that is, no use of Lambdex/PEX at runtime, only build time)
   2. the old layout is deprecated but still the default
2. 2.18:
   1. the new layout is the default
   2. support for building Layers, via the new python_aws_lambda_layer target
   3. python_awslambda target renamed to python_aws_lambda_function, for consistency with the layer target and python_google_cloud_function
   4. the runtime=... field can be inferred, if the interpreter constraints are unambiguous, e.g. CPython==3.10.* will imply runtime="python3.10"
   5. if the runtime field value is known to pants (including when inferred, above), a static complete platform JSON file is used, for more precise/useful dependency selection (cf. Some recipes for AWS Lambda and Docker Python artefacts (including cross-building from macOS) #18756). NB. this is currently only for AWS Lambda (Include appropriate complete_platforms for serverless/FaaS environments runtime #18195 (comment)).
3. 2.19:
   1. the old layout will be removed, and the new layout will be the only one supported

For testing, the usual approach for pants is upgrading one major release at a time:

1. make sure you're on 2.16
2. upgrade to 2.17 (pants_version = "2.17.0rc3" at the time of writing), resolve the warnings and validate it
3. upgrade to 2.18 (pants_version = "2.18.0a0"), resolve the warnings and validate it. Both of these are pre-releases, so testing locally or in a dev environment is fine, but a reasonably accurate simulation of your prod environment(s) will find the most bugs 😄

The 2.17 docs are at https://www.pantsbuild.org/v2.17/docs/awslambda-python and https://www.pantsbuild.org/v2.17/docs/google-cloud-function-python.

The 2.18 docs are at https://github.com/pantsbuild/pants/blob/bff43abdc632b6e196b9e97882353eee23baaf61/docs/markdown/Python/python-integrations/awslambda-python.md and https://github.com/pantsbuild/pants/blob/bff43abdc632b6e196b9e97882353eee23baaf61/docs/markdown/Python/python-integrations/google-cloud-function-python.md

Let me know here any issues/hitches/questions, e.g. with errors/warnings, with runtime behaviour, with how to configure BUILD files, anything else!

[cponfick]

Feedback python_awslambda (2.17)

Included Requirements

First I deployed the lambda with the inclusion of the requirements, everything worked as expected.

Excluded Requirements

Excluding the requirements did also work as expected.

Hybrid

When using dependencies like awswrangler users have to include a pre packed layer provided by AWS, because the pypi version is too big (https://aws-sdk-pandas.readthedocs.io/en/stable/install.html). Hence, it would be nice if the target would allow certain 3rd party dependencies to be excluded. I tried using !. This was ignored by pants. pants dependencies also showed the library being included. It would be a good addition to the target, in my opinion.

I think an exclusion of individual third party dependencies should be possible in python_aws_lambda_layer, if I understood the documentation correctly. I will try this in the following days.

[huonw]

Thanks!

When using dependencies like awswrangler users have to include a pre packed layer provided by AWS, because the pypi version is too big (https://aws-sdk-pandas.readthedocs.io/en/stable/install.html). Hence, it would be nice if the target would allow certain 3rd party dependencies to be excluded. I tried using !. This was ignored by pants. pants dependencies also showed the library being included. It would be a good addition to the target, in my opinion.

The target should behave like other targets (especially pex_binary and other Python ones) with respect to dependency exclusion. You may need to use the stronger !! to exclude dependencies transitively (see the "Ignore dependencies with ! and !!" call out in https://www.pantsbuild.org/docs/targets#dependencies-field for more detail).

Additional limitation: even !! doesn't stop third-party dependencies from forcing a package to be included, e.g. if your code depends on some-package that depends on awswrangler, PEX deciding to package some_package will also package the transitive dep awswrangler, because that's a 3rd-party-python-package dependencies which Pants doesn't yet see/manage, rather than a Pants-managed one. #12733 and pex-tool/pex#2097 are relevant issues.

There's also #19256 that discusses a more automatic way to handle this sort of exclusion for layers. (I've written #19256 (comment) that discusses a possibility here more explicitly)

[cponfick]

Ok, I try !! I did not know that ! does not work on transitive dependencies. #19256 seems like a good way of automating layer specific exclusions.

[huonw]

@cponfick-bhs noted on slack that the python_aws_lambda_layer target doesn't exist in 2.18.0.dev6. This is fixed in #19550

[huonw]

Still not right: #19562 now properly works for me locally, with PANTS_SOURCE=... pants package ...

[cponfick]

I still does not work for me on 2.18.0.dev7 was the PR merged into that version? The target exists, but pants package raises an error.

15:00:28.01 [WARN] No applicable files or targets matched. The `package` goal works with these target types:

  * archive
  * docker_image
  * pex_binary
  * python_aws_lambda_function
  * python_distribution

Seems like the package goal is not exported correctly.

[huonw]

Yeah unfortunately #19562 isn't in dev7, so we'll have to wait for the next release.

[huonw]

Okay, layers finally work in 2.18.0a0! Yay. Sorry for the delay. I'll update the original post.

[cponfick]

Review Python AWS Lambda Layer (2.18.0rc5)

I finally found the time to try out the lambda layer packaging. I tried following configuration.

python_aws_lambda_layer(
    name="layer",
    dependencies=[
        "//src/python/a/lambda_a.py",
        "//src/python/b/lambda_b.py",
        "//src/python/c/lambda_c.py",",
        "!!//src/python:reqs#awswrangler", # part of my requirement.txt, but has a own layer provided by AWS.
    ],
    include_sources=False,
    runtime="python3.10",
)

The layer was created as I expected it. I deployed the lambdas and the layer using AWS CDK. The lambdas could be executed without any issues.

[huonw]

Yay! Thanks for testing.