Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature request: Pydantic v2 managed layer #2939

Closed
1 of 2 tasks
lmmx opened this issue Aug 9, 2023 · 7 comments
Closed
1 of 2 tasks

Feature request: Pydantic v2 managed layer #2939

lmmx opened this issue Aug 9, 2023 · 7 comments
Labels
feature-request feature request need-customer-feedback Requires more customers feedback before making or revisiting a decision v3 Features that will be included in Powertools v3.

Comments

@lmmx
Copy link

lmmx commented Aug 9, 2023

Use case

I saw that there was support coming for Pydantic v2 in lambda powertools:

but it seems like it is only support, not incorporation into the library dependencies (i.e. the library is not bumping to v2). This conflicts with the desired workflow of using new versions of the layer in combination with up to date Pydantic models.

Solution/User Experience

Will you consider publishing managed layers with v2 of Pydantic as the dependency?

This seems in line with the linked 'Tenet' of adopting best practices:

Eases the adoption of best practices. The main priority of the utilities is to facilitate best practices adoption, as defined in the AWS Well-Architected Serverless Lens; all other functionality is optional.

It could also be done in a backward compatible way if it was published with a separate ARN

We strive for backwards compatibility. New features and changes should keep backwards compatibility. If a breaking change cannot be avoided, the deprecation and migration process should be clearly defined.

It also relates to 'following language idioms and community common practices':

Progressive. Utilities are designed to be incrementally adoptable for customers at any stage of their Serverless journey. They follow language idioms and their community’s common practices.

Alternative solutions

No response

Acknowledgment

@lmmx lmmx added feature-request feature request triage Pending triage from maintainers labels Aug 9, 2023
@boring-cyborg
Copy link

boring-cyborg bot commented Aug 9, 2023

Thanks for opening your first issue here! We'll come back to you as soon as we can.
In the meantime, check out the #python channel on our Powertools for AWS Lambda Discord: Invite link

@leandrodamascena
Copy link
Contributor

Hello @lmmx! Thank you for opening this issue. I am currently reading this to respond to your points properly.

@leandrodamascena leandrodamascena moved this from Triage to Pending review in Powertools for AWS Lambda (Python) Aug 10, 2023
@leandrodamascena leandrodamascena added revisit-in-3-months Requires more customers feedback before making or revisiting a decision need-customer-feedback Requires more customers feedback before making or revisiting a decision and removed triage Pending triage from maintainers labels Aug 10, 2023
@leandrodamascena
Copy link
Contributor

Hi @lmmx , thank you so much for bringing this issue to discussion, this is an important point.

Here in Powertools we do our best to make clear customer communication in every release we make and new features we add. We follow semantic versioning and therefore breaking changes are scheduled for major releases and even then with early and clear communication to our customers. That said, we've kept our layer with Pydanticv1 so that our users can upgrade to Pydanticv2 as often as they like, without forcing it and causing supply chain problems, as we experienced during the transition.

We are looking to add Pydantic V2 as our default in Powertools v3 - tentatively end of the year (https://docs.powertools.aws.dev/lambda/python/latest/roadmap/). Creating a new layer at this moment with unique support for Pydantic v2 adds more complexity to our process for maintaining and updating that layer. At this time, we are working hard to implement new features such as external observability providers and sensitive data masking.

To keep our eyes on the needs of the community, I added the need-customer-feedback and revisit-in-3-months labels - who knows, this might just be what other customers are looking for too.

Thanks.

@leandrodamascena leandrodamascena moved this from Pending review to Backlog in Powertools for AWS Lambda (Python) Aug 11, 2023
@lmmx
Copy link
Author

lmmx commented Aug 12, 2023

Hey Leandro that's cool, I built a layer myself (it didn't take much effort) so this isn't urgent. I found a link to an AWS repo which linked to another one which had a Dockerfile, and I got a general idea of how to use rustup which is needed for Pydantic v2.

I suspect I actually end up installing Rust twice but it works so it's all good.

Python 3.10 layer Dockerfile is below, with apologies for any mess, in case it helps anyone else but I know you have your own superior CDK approach 😺

Click to show Dockerfile
FROM mlupin/docker-lambda:python3.10-build AS build

USER root

ENV SCRATCH_DIR=/var/task/scratch
ENV LAYER_ZIP="pydantic_layer.zip"

WORKDIR /var/task

# https://towardsdatascience.com/how-to-shrink-numpy-scipy-pandas-and-matplotlib-for-your-data-product-4ec8d7e86ee4
ENV CFLAGS "-g0 -Wl,--strip-all -DNDEBUG -Os -I/usr/include:/usr/local/include -L/usr/lib64:/usr/local/lib64:/usr/lib:/usr/local/lib"

# Rust up https://github.com/rust-serverless/lambda-rust/blob/master/Dockerfile
# via https://github.com/awslabs/aws-lambda-rust-runtime#25-docker
# via https://docs.aws.amazon.com/lambda/latest/dg/lambda-rust.html
ENV RUST_VERSION=1.70.0
RUN yum install -y jq openssl-devel gcc zip
RUN set -o pipefail && curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs \
    | CARGO_HOME=/cargo RUSTUP_HOME=/rustup sh -s -- -y --profile minimal --default-toolchain $RUST_VERSION
# ENV PATH="/root/.cargo/bin:${PATH}"
ENV PATH="/cargo/bin:${PATH}"

RUN rustup default $RUST_VERSION

RUN ls /cargo/bin && \
    rustc --version

# ------------------- END OF SYSTEM BUILD DEPENDENCY SETUP ------------------

RUN mkdir -p $SCRATCH_DIR/python && \
    mkdir -p $SCRATCH_DIR/lib

# Install build dependencies for the wheels
RUN python3.10 -m pip install --upgrade pip && \
    python3.10 -m pip --version

# ------------------- END OF PACKAGE BUILD DEPENDENCY SETUP ------------------

ENV PYDANTIC_VERSION=2.1.1

# Download pydantic source distribution
RUN python3.10 -m pip download --no-binary=:all: pydantic==$PYDANTIC_VERSION

# Extract the pydantic package and build the wheel
RUN ls && tar xzf pydantic-$PYDANTIC_VERSION.tar.gz 

WORKDIR /var/task/pydantic-$PYDANTIC_VERSION

RUN python3.10 -m pip install build
RUN python3.10 -m build

WORKDIR /var/task

ENV BUILT_WHEEL=pydantic-$PYDANTIC_VERSION/dist/*-$PYDANTIC_VERSION-*.whl

RUN ls $BUILT_WHEEL

# ------------------------ END OF PYDANTIC BUILD --------------------------------

# Install the wheels with pip
# (Note: previously this used --compile but now we already did the wheel compilation)
RUN python3.10 -m pip install --no-compile --no-cache-dir \
    -t $SCRATCH_DIR/python \
    $BUILT_WHEEL && \
    ls $SCRATCH_DIR/python

# -------------------- END OF PACKAGE INSTALLATION -----------------------------

# Clean up the sdist and wheel
RUN rm pydantic-$PYDANTIC_VERSION.tar.gz && \
    rm -r pydantic-$PYDANTIC_VERSION

RUN cd $SCRATCH_DIR \
    && zip -r9 $LAYER_ZIP python  \
    && zip -r9 $LAYER_ZIP lib
    
FROM mlupin/docker-lambda:python3.10-build
 
COPY --from=build /var/task/scratch /opt
COPY --from=build /var/task /var/task

RUN PYTHONPATH=/opt/python python3 -c 'import pydantic'

Of course there are features in Powertools including Pydantic integration which go beyond Pydantic's base library, so I'm still looking forward to the managed layer.

@leandrodamascena
Copy link
Contributor

Hello @lmmx! Wow, this Dockfile is priceless!! I'm going to copy and save it because I might need it soon to create some tests. Thank you so much for sharing this.

Yes, we are working to merge the existing PR and start working on the V3 plan as next steps.

I'll update this issue in case of news, ok?

Thanks

@leandrodamascena leandrodamascena added the v3 Features that will be included in Powertools v3. label Aug 28, 2023
@leandrodamascena leandrodamascena added this to the Powertools v3 milestone Aug 28, 2023
@heitorlessa heitorlessa removed the revisit-in-3-months Requires more customers feedback before making or revisiting a decision label Feb 5, 2024
@leandrodamascena
Copy link
Contributor

Hello @lmmx! We are working on Powertools V3 and in the v3 branch we have already made Pydantic v2 our default version. We expect to release v3 somewhere in August and we'll publish a complete Upgrade guide to help customers to update this.

I'm closing this issue because there's nothing to do other than wait for V3 to be released.

Thanks and please reopen if you need anything else.

Copy link
Contributor

⚠️COMMENT VISIBILITY WARNING⚠️

This issue is now closed. Please be mindful that future comments are hard for our team to see.

If you need more assistance, please either tag a team member or open a new issue that references this one.

If you wish to keep having a conversation with other community members under this issue feel free to do so.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature-request feature request need-customer-feedback Requires more customers feedback before making or revisiting a decision v3 Features that will be included in Powertools v3.
Projects
Status: Coming soon
Development

No branches or pull requests

3 participants