From d97d1763f96359dd8547736f8e045c3681db9557 Mon Sep 17 00:00:00 2001 From: Leandro Damascena Date: Fri, 28 Jul 2023 15:34:51 +0100 Subject: [PATCH] docs(navigation): remove nofollow attribute for internal links (#2867) --- docs/automation.md | 50 +++++++-------- docs/core/event_handler/api_gateway.md | 30 ++++----- docs/core/event_handler/appsync.md | 18 +++--- docs/core/logger.md | 18 +++--- docs/core/metrics.md | 26 ++++---- docs/core/tracer.md | 16 ++--- docs/index.md | 80 ++++++++++++------------ docs/maintainers.md | 18 +++--- docs/roadmap.md | 18 +++--- docs/security.md | 2 +- docs/tutorial/index.md | 84 +++++++++++++------------- docs/upgrade.md | 20 +++--- docs/utilities/batch.md | 14 ++--- docs/utilities/data_classes.md | 42 ++++++------- docs/utilities/feature_flags.md | 14 ++--- docs/utilities/idempotency.md | 24 ++++---- docs/utilities/jmespath_functions.md | 6 +- docs/utilities/middleware_factory.md | 12 ++-- docs/utilities/parameters.md | 26 ++++---- docs/utilities/parser.md | 4 +- docs/utilities/streaming.md | 2 +- docs/utilities/typing.md | 4 +- docs/utilities/validation.md | 6 +- 23 files changed, 267 insertions(+), 267 deletions(-) diff --git a/docs/automation.md b/docs/automation.md index 06b11525f9d..467df2b9803 100644 --- a/docs/automation.md +++ b/docs/automation.md @@ -21,7 +21,7 @@ This is a snapshot of our automated checks at a glance. ### Pre-commit checks -> [**Pre-commit configuration**](https://github.com/aws-powertools/powertools-lambda-python/blob/0523ff64606514ea3e59c07c8c69c83d751f61fa/.pre-commit-config.yaml){target="_blank" rel="nofollow"}. +> [**Pre-commit configuration**](https://github.com/aws-powertools/powertools-lambda-python/blob/0523ff64606514ea3e59c07c8c69c83d751f61fa/.pre-commit-config.yaml){target="_blank"}. Pre-commit checks are crucial for a fast feedback loop while ensuring security practices at the individual change level. @@ -29,13 +29,13 @@ To prevent scenarios where these checks are intentionally omitted at the client !!! note "These run locally only for changed files" -* [**Merge conflict check**](https://github.com/aws-powertools/powertools-lambda-python/blob/0523ff64606514ea3e59c07c8c69c83d751f61fa/.pre-commit-config.yaml#L10){target="_blank" rel="nofollow"}. Checks for merge strings in each individual change accidentally left unresolved to prevent breakage. -* [**Code linting**](https://github.com/aws-powertools/powertools-lambda-python/blob/0523ff64606514ea3e59c07c8c69c83d751f61fa/Makefile#L22){target="_blank" rel="nofollow"}. Linter checks for industry quality standards and known bad practices that could lead to abuse. -* [**CloudFormation linting**](https://github.com/aws-powertools/powertools-lambda-python/blob/0523ff64606514ea3e59c07c8c69c83d751f61fa/.pre-commit-config.yaml#L34){target="_blank" rel="nofollow"}. `cfn-lint` ensures [best practices](https://github.com/aws-cloudformation/cfn-lint/blob/86f0370bd43b400ed4c485180dbc2697f73367b2/docs/rules.md){target=""_blank"} at our documentation examples. -* [**Markdown linting**](https://github.com/aws-powertools/powertools-lambda-python/blob/0523ff64606514ea3e59c07c8c69c83d751f61fa/.pre-commit-config.yaml#L25){target="_blank}. Primarily [industry markdown practices](https://github.com/DavidAnson/markdownlint/blob/d01180ec5a014083ee9d574b693a8d7fbc1e566d/README.md#rules--aliases){target="_blank" rel="nofollow"} at this stage. +* [**Merge conflict check**](https://github.com/aws-powertools/powertools-lambda-python/blob/0523ff64606514ea3e59c07c8c69c83d751f61fa/.pre-commit-config.yaml#L10){target="_blank" }. Checks for merge strings in each individual change accidentally left unresolved to prevent breakage. +* [**Code linting**](https://github.com/aws-powertools/powertools-lambda-python/blob/0523ff64606514ea3e59c07c8c69c83d751f61fa/Makefile#L22){target="_blank"}. Linter checks for industry quality standards and known bad practices that could lead to abuse. +* [**CloudFormation linting**](https://github.com/aws-powertools/powertools-lambda-python/blob/0523ff64606514ea3e59c07c8c69c83d751f61fa/.pre-commit-config.yaml#L34){target="_blank"}. `cfn-lint` ensures [best practices](https://github.com/aws-cloudformation/cfn-lint/blob/86f0370bd43b400ed4c485180dbc2697f73367b2/docs/rules.md){target=""_blank"} at our documentation examples. +* [**Markdown linting**](https://github.com/aws-powertools/powertools-lambda-python/blob/0523ff64606514ea3e59c07c8c69c83d751f61fa/.pre-commit-config.yaml#L25){target="_blank}. Primarily [industry markdown practices](https://github.com/DavidAnson/markdownlint/blob/d01180ec5a014083ee9d574b693a8d7fbc1e566d/README.md#rules--aliases){target="_blank"} at this stage. * [**GitHub Actions linting**](https://github.com/rhysd/actionlint/blob/main/docs/checks.md){target="_blank" rel="nofollow"}. `actionlint` ensures workflows follow [GitHub Actions security practices](https://docs.github.com/en/actions/security-guides/security-hardening-for-github-actions){target="_blank" rel="nofollow"}. It guards against numerous [leading practices](https://github.com/rhysd/actionlint/blob/main/docs/checks.md){target="_blank" rel="nofollow"} to prevent common configuration mistakes, insecure inline scripts, among many others. -* [**Terraform linting**](https://github.com/aws-powertools/powertools-lambda-python/blob/0523ff64606514ea3e59c07c8c69c83d751f61fa/.pre-commit-config.yaml#L43){target="_blank" rel="nofollow"}. As of now, largely formatting until we increase our Terraform coverage in documentation examples. -* [**Secrets linting**](https://github.com/aws-powertools/powertools-lambda-python/blob/develop/.pre-commit-config.yaml#L49){target="_blank" rel="nofollow"}. Detects industry credentials that might be accidentally leaked in source code. +* [**Terraform linting**](https://github.com/aws-powertools/powertools-lambda-python/blob/0523ff64606514ea3e59c07c8c69c83d751f61fa/.pre-commit-config.yaml#L43){target="_blank"}. As of now, largely formatting until we increase our Terraform coverage in documentation examples. +* [**Secrets linting**](https://github.com/aws-powertools/powertools-lambda-python/blob/develop/.pre-commit-config.yaml#L49){target="_blank"}. Detects industry credentials that might be accidentally leaked in source code. ### Pre-Pull Request checks @@ -43,10 +43,10 @@ For an improved contributing experience, most of our checks can run locally. For !!! note "These are in addition to [pre-commit checks](#pre-commit-checks)." -* [**Static typing analysis**](https://github.com/aws-powertools/powertools-lambda-python/blob/0523ff64606514ea3e59c07c8c69c83d751f61fa/Makefile#L108). `mypy` checks for static typing annotations to prevent common bugs in Python that may or may not lead to abuse. -* [**Tests**](https://github.com/aws-powertools/powertools-lambda-python/blob/0523ff64606514ea3e59c07c8c69c83d751f61fa/Makefile#L31){target="_blank" rel="nofollow"}. We run `unit`, `functional`, and `performance` tests ([_see our definition_](https://github.com/aws-powertools/powertools-lambda-python/blob/0523ff64606514ea3e59c07c8c69c83d751f61fa/CONTRIBUTING.md#testing-definition){target="_blank" rel="nofollow"}). Besides breaking changes, we are investing in mutation testing to find additional sources of bugs and potential abuse. -* [**Security baseline**](https://github.com/aws-powertools/powertools-lambda-python/blob/0523ff64606514ea3e59c07c8c69c83d751f61fa/CONTRIBUTING.md#testing-definition){target="_blank" rel="nofollow"}. `bandit` detects common security issues defined by Python Code Quality Authority (PyCQA). -* [**Complexity baseline**](https://github.com/aws-powertools/powertools-lambda-python/blob/0523ff64606514ea3e59c07c8c69c83d751f61fa/Makefile#L79){target="_blank" rel="nofollow"}. We run a series of maintenability and cyclomatic checks to reduce code and logic complexity. This aids reviewers' cognitive overhead and long-term maintainers revisiting legacy code at a later date. +* [**Static typing analysis**](https://github.com/aws-powertools/powertools-lambda-python/blob/0523ff64606514ea3e59c07c8c69c83d751f61fa/Makefile#L108){target="_blank"}. `mypy` checks for static typing annotations to prevent common bugs in Python that may or may not lead to abuse. +* [**Tests**](https://github.com/aws-powertools/powertools-lambda-python/blob/0523ff64606514ea3e59c07c8c69c83d751f61fa/Makefile#L31){target="_blank"}. We run `unit`, `functional`, and `performance` tests ([_see our definition_](https://github.com/aws-powertools/powertools-lambda-python/blob/0523ff64606514ea3e59c07c8c69c83d751f61fa/CONTRIBUTING.md#testing-definition){target="_blank"}). Besides breaking changes, we are investing in mutation testing to find additional sources of bugs and potential abuse. +* [**Security baseline**](https://github.com/aws-powertools/powertools-lambda-python/blob/0523ff64606514ea3e59c07c8c69c83d751f61fa/CONTRIBUTING.md#testing-definition){target="_blank"}. `bandit` detects common security issues defined by Python Code Quality Authority (PyCQA). +* [**Complexity baseline**](https://github.com/aws-powertools/powertools-lambda-python/blob/0523ff64606514ea3e59c07c8c69c83d751f61fa/Makefile#L79){target="_blank"}. We run a series of maintainability and cyclomatic checks to reduce code and logic complexity. This aids reviewers' cognitive overhead and long-term maintainers revisiting legacy code at a later date. ### Pull Request checks @@ -54,13 +54,13 @@ While we trust contributors and maintainers do go through pre-commit and pre-pul !!! note "Checks described earlier are omitted to improve reading experience." -* [**Semantic PR title**](https://github.com/Ezard/semantic-prs){target="_blank" rel="nofollow"}. We enforce PR titles follow semantic naming, for example `chore(category): change`. This benefits contributors with a lower entry bar, no need for semantic commits. It also benefits everyone looking for an [useful changelog message](https://docs.powertools.aws.dev/lambda/python/latest/changelog/){target="_blank" rel="nofollow"} on **what** changed and **where**. -* [**Related issue check**](https://github.com/aws-powertools/powertools-lambda-python/blob/0523ff64606514ea3e59c07c8c69c83d751f61fa/.github/workflows/on_opened_pr.yml#L44){target="_blank" rel="nofollow"}. [Every change require an issue](https://github.com/aws-powertools/powertools-lambda-python/blob/0523ff64606514ea3e59c07c8c69c83d751f61fa/CONTRIBUTING.md#contributing-via-pull-requests){target="_blank" rel="nofollow"} describing its needs. This enforces a PR has a related issue by blocking merge operations if missing. -* [**Acknowledgment check**](https://github.com/aws-powertools/powertools-lambda-python/blob/0523ff64606514ea3e59c07c8c69c83d751f61fa/.github/workflows/on_opened_pr.yml#L63){target="_blank" rel="nofollow"}. [Ensures PR template](https://github.com/aws-powertools/powertools-lambda-python/blob/0523ff64606514ea3e59c07c8c69c83d751f61fa/.github/PULL_REQUEST_TEMPLATE.md#L36){target="_blank" rel="nofollow"} is used and every contributor is aware of code redistribution. -* [**Code coverage diff**](https://github.com/aws-powertools/powertools-lambda-python/blob/0523ff64606514ea3e59c07c8c69c83d751f61fa/.github/workflows/quality_check.yml#L73){target="_blank" rel="nofollow"}. Educates contributors and maintainers about code coverage differences for a given change. -* [**Contribution size check**](https://github.com/aws-powertools/powertools-lambda-python/blob/0523ff64606514ea3e59c07c8c69c83d751f61fa/.github/workflows/on_label_added.yml#L44){target="_blank" rel="nofollow"}. Suggests contributors and maintainers to break up large changes (100-499 LOC) in smaller PRs. It helps reduce overlooking security and other practices due to increased cognitive overhead. -* [**Dependency vulnerability check**](https://github.com/aws-powertools/powertools-lambda-python/blob/0523ff64606514ea3e59c07c8c69c83d751f61fa/.github/workflows/dependency-review.yml#L22){target="_blank" rel="nofollow"}. Verifies any dependency changes for common vulnerability exposures (CVEs), in addition to our daily check on any dependencies used (e.g., Python, Docker, Go, etc.) -* [**GitHub Actions security check**](https://github.com/aws-powertools/powertools-lambda-python/blob/0523ff64606514ea3e59c07c8c69c83d751f61fa/.github/workflows/secure_workflows.yml#L35){target="_blank" rel="nofollow"}. Enforces use of immutable 3rd-party GitHub Actions (_e.g., `actions/checkout@_`) to prevent abuse. Upgrades are handled by a [separate automated process](https://github.com/aws-powertools/powertools-lambda-python/blob/0523ff64606514ea3e59c07c8c69c83d751f61fa/.github/dependabot.yml#L4){target="_blank" rel="nofollow"} that includes a maintainer review to also prevent unexpected behavior changes. +* [**Semantic PR title**](https://github.com/Ezard/semantic-prs){target="_blank" rel="nofollow"}. We enforce PR titles follow semantic naming, for example `chore(category): change`. This benefits contributors with a lower entry bar, no need for semantic commits. It also benefits everyone looking for an [useful changelog message](https://docs.powertools.aws.dev/lambda/python/latest/changelog/){target="_blank"} on **what** changed and **where**. +* [**Related issue check**](https://github.com/aws-powertools/powertools-lambda-python/blob/0523ff64606514ea3e59c07c8c69c83d751f61fa/.github/workflows/on_opened_pr.yml#L44){target="_blank"}. [Every change require an issue](https://github.com/aws-powertools/powertools-lambda-python/blob/0523ff64606514ea3e59c07c8c69c83d751f61fa/CONTRIBUTING.md#contributing-via-pull-requests){target="_blank"} describing its needs. This enforces a PR has a related issue by blocking merge operations if missing. +* [**Acknowledgment check**](https://github.com/aws-powertools/powertools-lambda-python/blob/0523ff64606514ea3e59c07c8c69c83d751f61fa/.github/workflows/on_opened_pr.yml#L63){target="_blank"}. [Ensures PR template](https://github.com/aws-powertools/powertools-lambda-python/blob/0523ff64606514ea3e59c07c8c69c83d751f61fa/.github/PULL_REQUEST_TEMPLATE.md#L36){target="_blank"} is used and every contributor is aware of code redistribution. +* [**Code coverage diff**](https://github.com/aws-powertools/powertools-lambda-python/blob/0523ff64606514ea3e59c07c8c69c83d751f61fa/.github/workflows/quality_check.yml#L73){target="_blank"}. Educates contributors and maintainers about code coverage differences for a given change. +* [**Contribution size check**](https://github.com/aws-powertools/powertools-lambda-python/blob/0523ff64606514ea3e59c07c8c69c83d751f61fa/.github/workflows/on_label_added.yml#L44){target="_blank"}. Suggests contributors and maintainers to break up large changes (100-499 LOC) in smaller PRs. It helps reduce overlooking security and other practices due to increased cognitive overhead. +* [**Dependency vulnerability check**](https://github.com/aws-powertools/powertools-lambda-python/blob/0523ff64606514ea3e59c07c8c69c83d751f61fa/.github/workflows/dependency-review.yml#L22){target="_blank"}. Verifies any dependency changes for common vulnerability exposures (CVEs), in addition to our daily check on any dependencies used (e.g., Python, Docker, Go, etc.) +* [**GitHub Actions security check**](https://github.com/aws-powertools/powertools-lambda-python/blob/0523ff64606514ea3e59c07c8c69c83d751f61fa/.github/workflows/secure_workflows.yml#L35){target="_blank"}. Enforces use of immutable 3rd-party GitHub Actions (_e.g., `actions/checkout@_`) to prevent abuse. Upgrades are handled by a [separate automated process](https://github.com/aws-powertools/powertools-lambda-python/blob/0523ff64606514ea3e59c07c8c69c83d751f61fa/.github/dependabot.yml#L4){target="_blank"} that includes a maintainer review to also prevent unexpected behavior changes. ### After merge checks @@ -68,18 +68,18 @@ While we trust contributors and maintainers do go through pre-commit and pre-pul We strike a balance in security and contribution experience. These automated checks take several minutes to complete. Failures are reviewed by a maintainer on-call and before a release. -* [**End-to-end tests**](https://github.com/aws-powertools/powertools-lambda-python/blob/0523ff64606514ea3e59c07c8c69c83d751f61fa/.github/workflows/run-e2e-tests.yml#L41){target="_blank" rel="nofollow"}. We run E2E with a [high degree of parallelization](https://github.com/aws-powertools/powertools-lambda-python/blob/0523ff64606514ea3e59c07c8c69c83d751f61fa/MAINTAINERS.md#test-runner-parallelization). While it is designed to also run locally, it may incur AWS charges to contributors. For additional security, all infrastructure is ephemeral per change and per Python version. -* [**SAST check**](https://github.com/aws-powertools/powertools-lambda-python/blob/0523ff64606514ea3e59c07c8c69c83d751f61fa/.github/workflows/codeql-analysis.yml#L20){target="_blank" rel="nofollow"}. GitHub CodeQL runs ~30m static analysis in the entire codebase. -* [**Security posture check**](https://github.com/aws-powertools/powertools-lambda-python/blob/0523ff64606514ea3e59c07c8c69c83d751f61fa/.github/workflows/ossf_scorecard.yml#L14){target="_blank" rel="nofollow"}. OSSF Scorecard runs numerous automated checks upon changes, and raises security alerts if [OSSF security practices](https://bestpractices.coreinfrastructure.org/en/criteria){target="_blank" rel="nofollow"} are no longer followed. -* [**Rebuild Changelog**](https://github.com/aws-powertools/powertools-lambda-python/blob/0523ff64606514ea3e59c07c8c69c83d751f61fa/.github/workflows/build_changelog.yml#L23){target="_blank" rel="nofollow"}. We rebuild our entire changelog upon changes and create a PR for maintainers. This has the added benefit in keeping a [protected branch](https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/managing-protected-branches/about-protected-branches){target="_blank" rel="nofollow"} while keeping removing error-prone tasks from maintainers. -* [**Stage documentation**](https://github.com/aws-powertools/powertools-lambda-python/blob/0523ff64606514ea3e59c07c8c69c83d751f61fa/.github/workflows/on_push_docs.yml#L27C16-L27C16){target="_blank" rel="nofollow"}. We rebuild and deploy changes to the documentation to a [staged version](https://docs.powertools.aws.dev/lambda/python/stage/){target="_blank" rel="nofollow"}. This gives us safety that our docs can always be rebuilt, and ready to release to production when needed. -* [**Update draft release**](https://github.com/aws-powertools/powertools-lambda-python/blob/0523ff64606514ea3e59c07c8c69c83d751f61fa/.github/workflows/release-drafter.yml#L25){target="_blank" rel="nofollow"}. We use [Release Drafter](https://github.com/release-drafter/release-drafter){target="_blank" rel="nofollow"} to generate a portion of our release notes and to always keep a fresh draft upon changes. You can read our [thoughts on a good quality release notes here](https://github.com/aws-powertools/powertools-lambda-python/blob/0523ff64606514ea3e59c07c8c69c83d751f61fa/MAINTAINERS.md#drafting-release-notes){target="_blank" rel="nofollow"} (human readable changes + automation). +* [**End-to-end tests**](https://github.com/aws-powertools/powertools-lambda-python/blob/0523ff64606514ea3e59c07c8c69c83d751f61fa/.github/workflows/run-e2e-tests.yml#L41){target="_blank"}. We run E2E with a [high degree of parallelization](https://github.com/aws-powertools/powertools-lambda-python/blob/0523ff64606514ea3e59c07c8c69c83d751f61fa/MAINTAINERS.md#test-runner-parallelization). While it is designed to also run locally, it may incur AWS charges to contributors. For additional security, all infrastructure is ephemeral per change and per Python version. +* [**SAST check**](https://github.com/aws-powertools/powertools-lambda-python/blob/0523ff64606514ea3e59c07c8c69c83d751f61fa/.github/workflows/codeql-analysis.yml#L20){target="_blank"}. GitHub CodeQL runs ~30m static analysis in the entire codebase. +* [**Security posture check**](https://github.com/aws-powertools/powertools-lambda-python/blob/0523ff64606514ea3e59c07c8c69c83d751f61fa/.github/workflows/ossf_scorecard.yml#L14){target="_blank"}. OSSF Scorecard runs numerous automated checks upon changes, and raises security alerts if [OSSF security practices](https://bestpractices.coreinfrastructure.org/en/criteria){target="_blank" rel="nofollow"} are no longer followed. +* [**Rebuild Changelog**](https://github.com/aws-powertools/powertools-lambda-python/blob/0523ff64606514ea3e59c07c8c69c83d751f61fa/.github/workflows/build_changelog.yml#L23){target="_blank"}. We rebuild our entire changelog upon changes and create a PR for maintainers. This has the added benefit in keeping a [protected branch](https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/managing-protected-branches/about-protected-branches){target="_blank" rel="nofollow"} while keeping removing error-prone tasks from maintainers. +* [**Stage documentation**](https://github.com/aws-powertools/powertools-lambda-python/blob/0523ff64606514ea3e59c07c8c69c83d751f61fa/.github/workflows/on_push_docs.yml#L27C16-L27C16){target="_blank"}. We rebuild and deploy changes to the documentation to a [staged version](https://docs.powertools.aws.dev/lambda/python/stage/){target="_blank"}. This gives us safety that our docs can always be rebuilt, and ready to release to production when needed. +* [**Update draft release**](https://github.com/aws-powertools/powertools-lambda-python/blob/0523ff64606514ea3e59c07c8c69c83d751f61fa/.github/workflows/release-drafter.yml#L25){target="_blank"}. We use [Release Drafter](https://github.com/release-drafter/release-drafter){target="_blank" rel="nofollow"} to generate a portion of our release notes and to always keep a fresh draft upon changes. You can read our [thoughts on a good quality release notes here](https://github.com/aws-powertools/powertools-lambda-python/blob/0523ff64606514ea3e59c07c8c69c83d751f61fa/MAINTAINERS.md#drafting-release-notes){target="_blank"} (human readable changes + automation). ## Continuous deployment practices !!! note "We adhere to industry recommendations from the [OSSF Scorecard project](https://bestpractices.coreinfrastructure.org/en/criteria){target="_blank" rel="nofollow"}, among [others](https://docs.github.com/en/actions/security-guides/security-hardening-for-github-actions){target="_blank" rel="nofollow"}." -Releases are triggered by maintainers along with a reviewer - [detailed info here](https://github.com/aws-powertools/powertools-lambda-python/blob/develop/MAINTAINERS.md#releasing-a-new-version){target="_blank" rel="nofollow"}. In addition to [checks that run for every code change](#continuous-integration-practices), our pipeline requires a manual approval before releasing. +Releases are triggered by maintainers along with a reviewer - [detailed info here](https://github.com/aws-powertools/powertools-lambda-python/blob/develop/MAINTAINERS.md#releasing-a-new-version){target="_blank"}. In addition to [checks that run for every code change](#continuous-integration-practices), our pipeline requires a manual approval before releasing. We use a combination of provenance and signed attestation for our builds, source code sealing, SAST scanners, Python specific static code analysis, ephemeral credentials that last a given job step, and more. diff --git a/docs/core/event_handler/api_gateway.md b/docs/core/event_handler/api_gateway.md index 720432dd3c1..708a9de6855 100644 --- a/docs/core/event_handler/api_gateway.md +++ b/docs/core/event_handler/api_gateway.md @@ -9,19 +9,19 @@ Event handler for Amazon API Gateway REST and HTTP APIs, Application Loader Bala * Lightweight routing to reduce boilerplate for API Gateway REST/HTTP API, ALB and Lambda Function URLs. * Support for CORS, binary and Gzip compression, Decimals JSON encoding and bring your own JSON serializer -* Built-in integration with [Event Source Data Classes utilities](../../utilities/data_classes.md){target="_blank" rel="nofollow"} for self-documented event schema +* Built-in integration with [Event Source Data Classes utilities](../../utilities/data_classes.md){target="_blank"} for self-documented event schema ## Getting started ???+ tip - All examples shared in this documentation are available within the [project repository](https://github.com/aws-powertools/powertools-lambda-python/tree/develop/examples){target="_blank" rel="nofollow"}. + All examples shared in this documentation are available within the [project repository](https://github.com/aws-powertools/powertools-lambda-python/tree/develop/examples){target="_blank"}. ### Required resources -If you're using any API Gateway integration, you must have an existing [API Gateway Proxy integration](https://docs.aws.amazon.com/apigateway/latest/developerguide/set-up-lambda-proxy-integrations.html){target="_blank" rel="nofollow"} or [ALB](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/lambda-functions.html){target="_blank" rel="nofollow"} configured to invoke your Lambda function. +If you're using any API Gateway integration, you must have an existing [API Gateway Proxy integration](https://docs.aws.amazon.com/apigateway/latest/developerguide/set-up-lambda-proxy-integrations.html){target="_blank"} or [ALB](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/lambda-functions.html){target="_blank"} configured to invoke your Lambda function. -In case of using [VPC Lattice](https://docs.aws.amazon.com/lambda/latest/dg/services-vpc-lattice.html){target="_blank" rel="nofollow"}, you must have a service network configured to invoke your Lambda function. +In case of using [VPC Lattice](https://docs.aws.amazon.com/lambda/latest/dg/services-vpc-lattice.html){target="_blank"}, you must have a service network configured to invoke your Lambda function. This is the sample infrastructure for API Gateway and Lambda Function URLs we are using for the examples in this documentation. @@ -105,7 +105,7 @@ When using Amazon Application Load Balancer (ALB) to front your Lambda functions #### Lambda Function URL -When using [AWS Lambda Function URL](https://docs.aws.amazon.com/lambda/latest/dg/urls-configuration.html){target="_blank" rel="nofollow"}, you can use `LambdaFunctionUrlResolver`. +When using [AWS Lambda Function URL](https://docs.aws.amazon.com/lambda/latest/dg/urls-configuration.html){target="_blank"}, you can use `LambdaFunctionUrlResolver`. === "getting_started_lambda_function_url_resolver.py" @@ -121,7 +121,7 @@ When using [AWS Lambda Function URL](https://docs.aws.amazon.com/lambda/latest/d #### VPC Lattice -When using [VPC Lattice with AWS Lambda](https://docs.aws.amazon.com/lambda/latest/dg/services-vpc-lattice.html){target="_blank" rel="nofollow"}, you can use `VPCLatticeResolver`. +When using [VPC Lattice with AWS Lambda](https://docs.aws.amazon.com/lambda/latest/dg/services-vpc-lattice.html){target="_blank"}, you can use `VPCLatticeResolver`. === "getting_started_vpclattice_resolver.py" @@ -210,7 +210,7 @@ If you need to accept multiple HTTP methods in a single function, you can use th ### Accessing request details -Event Handler integrates with [Event Source Data Classes utilities](../../utilities/data_classes.md){target="_blank" rel="nofollow"}, and it exposes their respective resolver request details and convenient methods under `app.current_event`. +Event Handler integrates with [Event Source Data Classes utilities](../../utilities/data_classes.md){target="_blank"}, and it exposes their respective resolver request details and convenient methods under `app.current_event`. That is why you see `app.resolve(event, context)` in every example. This allows Event Handler to resolve requests, and expose data like `app.lambda_context` and `app.current_event`. @@ -268,11 +268,11 @@ We provide pre-defined errors for the most popular ones such as HTTP 400, 401, 4 ### Custom Domain API Mappings -When using [Custom Domain API Mappings feature](https://docs.aws.amazon.com/apigateway/latest/developerguide/rest-api-mappings.html){target="_blank" rel="nofollow"}, you must use **`strip_prefixes`** param in the `APIGatewayRestResolver` constructor. +When using [Custom Domain API Mappings feature](https://docs.aws.amazon.com/apigateway/latest/developerguide/rest-api-mappings.html){target="_blank"}, you must use **`strip_prefixes`** param in the `APIGatewayRestResolver` constructor. **Scenario**: You have a custom domain `api.mydomain.dev`. Then you set `/payment` API Mapping to forward any payment requests to your Payments API. -**Challenge**: This means your `path` value for any API requests will always contain `/payment/`, leading to HTTP 404 as Event Handler is trying to match what's after `payment/`. This gets further complicated with an [arbitrary level of nesting](https://github.com/aws-powertools/powertools-lambda-roadmap/issues/34){target="_blank" rel="nofollow"}. +**Challenge**: This means your `path` value for any API requests will always contain `/payment/`, leading to HTTP 404 as Event Handler is trying to match what's after `payment/`. This gets further complicated with an [arbitrary level of nesting](https://github.com/aws-powertools/powertools-lambda-roadmap/issues/34){target="_blank"}. To address this API Gateway behavior, we use `strip_prefixes` parameter to account for these prefixes that are now injected into the path regardless of which type of API Gateway you're using. @@ -363,7 +363,7 @@ You can use the `Response` class to have full control over the response. For exa Some event sources require headers and cookies to be encoded as `multiValueHeaders`. ???+ warning "Using multiple values for HTTP headers in ALB?" - Make sure you [enable the multi value headers feature](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/lambda-functions.html#multi-value-headers){target="_blank" rel="nofollow"} to serialize response headers correctly. + Make sure you [enable the multi value headers feature](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/lambda-functions.html#multi-value-headers){target="_blank"} to serialize response headers correctly. === "fine_grained_responses.py" @@ -448,7 +448,7 @@ Like `compress` feature, the client must send the `Accept` header with the corre ### Debug mode -You can enable debug mode via `debug` param, or via `POWERTOOLS_DEV` [environment variable](../../index.md#environment-variables){target="_blank" rel="nofollow"}. +You can enable debug mode via `debug` param, or via `POWERTOOLS_DEV` [environment variable](../../index.md#environment-variables){target="_blank"}. This will enable full tracebacks errors in the response, print request and responses, and set CORS in development mode. @@ -602,7 +602,7 @@ _**Benefits**_ _**Downsides**_ -* **Cold starts**. Frequent deployments and/or high load can diminish the benefit of monolithic functions depending on your latency requirements, due to [Lambda scaling model](https://docs.aws.amazon.com/lambda/latest/dg/invocation-scaling.html){target="_blank" rel="nofollow"}. Always load test to pragmatically balance between your customer experience and development cognitive load. +* **Cold starts**. Frequent deployments and/or high load can diminish the benefit of monolithic functions depending on your latency requirements, due to [Lambda scaling model](https://docs.aws.amazon.com/lambda/latest/dg/invocation-scaling.html){target="_blank"}. Always load test to pragmatically balance between your customer experience and development cognitive load. * **Granular security permissions**. The micro function approach enables you to use fine-grained permissions & access controls, separate external dependencies & code signing at the function level. Conversely, you could have multiple functions while duplicating the final code artifact in a monolithic approach. * Regardless, least privilege can be applied to either approaches. * **Higher risk per deployment**. A misconfiguration or invalid import can cause disruption if not caught earlier in automated testing. Multiple functions can mitigate misconfigurations but they would still share the same code artifact. You can further minimize risks with multiple environments in your CI/CD pipeline. @@ -615,13 +615,13 @@ A micro function means that your final code artifact will be different to each f **Benefits** -* **Granular scaling**. A micro function can benefit from the [Lambda scaling model](https://docs.aws.amazon.com/lambda/latest/dg/invocation-scaling.html){target="_blank" rel="nofollow"} to scale differently depending on each part of your application. Concurrency controls and provisioned concurrency can also be used at a granular level for capacity management. +* **Granular scaling**. A micro function can benefit from the [Lambda scaling model](https://docs.aws.amazon.com/lambda/latest/dg/invocation-scaling.html){target="_blank"} to scale differently depending on each part of your application. Concurrency controls and provisioned concurrency can also be used at a granular level for capacity management. * **Discoverability**. Micro functions are easier do visualize when using distributed tracing. Their high-level architectures can be self-explanatory, and complexity is highly visible — assuming each function is named to the business purpose it serves. -* **Package size**. An independent function can be significant smaller (KB vs MB) depending on external dependencies it require to perform its purpose. Conversely, a monolithic approach can benefit from [Lambda Layers](https://docs.aws.amazon.com/lambda/latest/dg/invocation-layers.html){target="_blank" rel="nofollow"} to optimize builds for external dependencies. +* **Package size**. An independent function can be significant smaller (KB vs MB) depending on external dependencies it require to perform its purpose. Conversely, a monolithic approach can benefit from [Lambda Layers](https://docs.aws.amazon.com/lambda/latest/dg/invocation-layers.html){target="_blank"} to optimize builds for external dependencies. **Downsides** -* **Upfront investment**. You need custom build tooling to bundle assets, including [C bindings for runtime compatibility](https://docs.aws.amazon.com/lambda/latest/dg/lambda-runtimes.html){target="_blank" rel="nofollow"}. Operations become more elaborate — you need to standardize tracing labels/annotations, structured logging, and metrics to pinpoint root causes. +* **Upfront investment**. You need custom build tooling to bundle assets, including [C bindings for runtime compatibility](https://docs.aws.amazon.com/lambda/latest/dg/lambda-runtimes.html){target="_blank"}. Operations become more elaborate — you need to standardize tracing labels/annotations, structured logging, and metrics to pinpoint root causes. * Engineering discipline is necessary for both approaches. Micro-function approach however requires further attention in consistency as the number of functions grow, just like any distributed system. * **Harder to share code**. Shared code must be carefully evaluated to avoid unnecessary deployments when that changes. Equally, if shared code isn't a library, your development, building, deployment tooling need to accommodate the distinct layout. diff --git a/docs/core/event_handler/appsync.md b/docs/core/event_handler/appsync.md index b0f8a294ebd..789bf788004 100644 --- a/docs/core/event_handler/appsync.md +++ b/docs/core/event_handler/appsync.md @@ -9,15 +9,15 @@ Event handler for AWS AppSync Direct Lambda Resolver and Amplify GraphQL Transfo * Automatically parse API arguments to function arguments * Choose between strictly match a GraphQL field name or all of them to a function -* Integrates with [Data classes utilities](../../utilities/data_classes.md){target="_blank" rel="nofollow"} to access resolver and identity information +* Integrates with [Data classes utilities](../../utilities/data_classes.md){target="_blank"} to access resolver and identity information * Works with both Direct Lambda Resolver and Amplify GraphQL Transformer `@function` directive * Support async Python 3.8+ functions, and generators ## Terminology -**[Direct Lambda Resolver](https://docs.aws.amazon.com/appsync/latest/devguide/direct-lambda-reference.html){target="_blank" rel="nofollow"}**. A custom AppSync Resolver to bypass the use of Apache Velocity Template (VTL) and automatically map your function's response to a GraphQL field. +**[Direct Lambda Resolver](https://docs.aws.amazon.com/appsync/latest/devguide/direct-lambda-reference.html){target="_blank"}**. A custom AppSync Resolver to bypass the use of Apache Velocity Template (VTL) and automatically map your function's response to a GraphQL field. -**[Amplify GraphQL Transformer](https://docs.amplify.aws/cli/graphql-transformer/function){target="_blank" rel="nofollow"}**. Custom GraphQL directives to define your application's data model using Schema Definition Language (SDL). Amplify CLI uses these directives to convert GraphQL SDL into full descriptive AWS CloudFormation templates. +**[Amplify GraphQL Transformer](https://docs.amplify.aws/cli/graphql-transformer/function){target="_blank"}**. Custom GraphQL directives to define your application's data model using Schema Definition Language (SDL). Amplify CLI uses these directives to convert GraphQL SDL into full descriptive AWS CloudFormation templates. ## Getting started @@ -28,7 +28,7 @@ You must have an existing AppSync GraphQL API and IAM permissions to invoke your This is the sample infrastructure we are using for the initial examples with a AppSync Direct Lambda Resolver. ???+ tip "Tip: Designing GraphQL Schemas for the first time?" - Visit [AWS AppSync schema documentation](https://docs.aws.amazon.com/appsync/latest/devguide/designing-your-schema.html){target="_blank" rel="nofollow"} for understanding how to define types, nesting, and pagination. + Visit [AWS AppSync schema documentation](https://docs.aws.amazon.com/appsync/latest/devguide/designing-your-schema.html){target="_blank"} for understanding how to define types, nesting, and pagination. === "getting_started_schema.graphql" @@ -93,7 +93,7 @@ Here's an example with two separate functions to resolve `getTodo` and `listTodo ### Scalar functions -When working with [AWS AppSync Scalar types](https://docs.aws.amazon.com/appsync/latest/devguide/scalars.html){target="_blank" rel="nofollow"}, you might want to generate the same values for data validation purposes. +When working with [AWS AppSync Scalar types](https://docs.aws.amazon.com/appsync/latest/devguide/scalars.html){target="_blank"}, you might want to generate the same values for data validation purposes. For convenience, the most commonly used values are available as functions within `scalar_types_utils` module. @@ -143,7 +143,7 @@ For Lambda Python3.8+ runtime, this utility supports async functions when you us ### Amplify GraphQL Transformer -Assuming you have [Amplify CLI installed](https://docs.amplify.aws/cli/start/install){target="_blank" rel="nofollow"}, create a new API using `amplify add api` and use the following GraphQL Schema. +Assuming you have [Amplify CLI installed](https://docs.amplify.aws/cli/start/install){target="_blank"}, create a new API using `amplify add api` and use the following GraphQL Schema. @@ -151,7 +151,7 @@ Assuming you have [Amplify CLI installed](https://docs.amplify.aws/cli/start/ins --8<-- "examples/event_handler_graphql/src/amplify_graphql_transformer_schema.graphql" ``` -[Create two new basic Python functions](https://docs.amplify.aws/cli/function#set-up-a-function){target="_blank" rel="nofollow"} via `amplify add function`. +[Create two new basic Python functions](https://docs.amplify.aws/cli/function#set-up-a-function){target="_blank"} via `amplify add function`. ???+ note Amplify CLI generated functions use `Pipenv` as a dependency manager. Your function source code is located at **`amplify/backend/function/your-function-name`**. @@ -192,7 +192,7 @@ Use the following code for `merchantInfo` and `searchMerchant` functions respect ### Custom data models -You can subclass [AppSyncResolverEvent](../../utilities/data_classes.md#appsync-resolver){target="_blank" rel="nofollow"} to bring your own set of methods to handle incoming events, by using `data_model` param in the `resolve` method. +You can subclass [AppSyncResolverEvent](../../utilities/data_classes.md#appsync-resolver){target="_blank"} to bring your own set of methods to handle incoming events, by using `data_model` param in the `resolve` method. === "custom_models.py.py" @@ -215,7 +215,7 @@ You can subclass [AppSyncResolverEvent](../../utilities/data_classes.md#appsync- ### Split operations with Router ???+ tip - Read the **[considerations section for trade-offs between monolithic and micro functions](./api_gateway.md#considerations){target="_blank" rel="nofollow"}**, as it's also applicable here. + Read the **[considerations section for trade-offs between monolithic and micro functions](./api_gateway.md#considerations){target="_blank"}**, as it's also applicable here. As you grow the number of related GraphQL operations a given Lambda function should handle, it is natural to split them into separate files to ease maintenance - That's when the `Router` feature comes handy. diff --git a/docs/core/logger.md b/docs/core/logger.md index 20accabb310..30a43a5817c 100644 --- a/docs/core/logger.md +++ b/docs/core/logger.md @@ -15,7 +15,7 @@ Logger provides an opinionated logger with output structured as JSON. ## Getting started ???+ tip - All examples shared in this documentation are available within the [project repository](https://github.com/aws-powertools/powertools-lambda-python/tree/develop/examples){target="_blank" rel="nofollow"}. + All examples shared in this documentation are available within the [project repository](https://github.com/aws-powertools/powertools-lambda-python/tree/develop/examples){target="_blank"}. Logger requires two settings: @@ -39,7 +39,7 @@ Your Logger will include the following keys to your structured logging: | **message**: `Any` | `Collecting payment` | Unserializable JSON values are casted as `str` | | **timestamp**: `str` | `2021-05-03 10:20:19,650+0200` | Timestamp with milliseconds, by default uses local timezone | | **service**: `str` | `payment` | Service name defined, by default `service_undefined` | -| **xray_trace_id**: `str` | `1-5759e988-bd862e3fe1be46a994272793` | When [tracing is enabled](https://docs.aws.amazon.com/lambda/latest/dg/services-xray.html){target="_blank" rel="nofollow"}, it shows X-Ray Trace ID | +| **xray_trace_id**: `str` | `1-5759e988-bd862e3fe1be46a994272793` | When [tracing is enabled](https://docs.aws.amazon.com/lambda/latest/dg/services-xray.html){target="_blank"}, it shows X-Ray Trace ID | | **sampling_rate**: `float` | `0.1` | When enabled, it shows sampling rate in percentage e.g. 10% | | **exception_name**: `str` | `ValueError` | When `logger.exception` is used and there is an exception | | **exception**: `str` | `Traceback (most recent call last)..` | When `logger.exception` is used and there is an exception | @@ -108,7 +108,7 @@ You can set a Correlation ID using `correlation_id_path` param by passing a [JME #### set_correlation_id method -You can also use `set_correlation_id` method to inject it anywhere else in your code. Example below uses [Event Source Data Classes utility](../utilities/data_classes.md){target="_blank" rel="nofollow"} to easily access events properties. +You can also use `set_correlation_id` method to inject it anywhere else in your code. Example below uses [Event Source Data Classes utility](../utilities/data_classes.md){target="_blank"} to easily access events properties. === "set_correlation_id_method.py" @@ -163,7 +163,7 @@ You can append additional keys using either mechanism: #### append_keys method ???+ warning - `append_keys` is not thread-safe, please see [RFC](https://github.com/aws-powertools/powertools-lambda-python/issues/991){target="_blank" rel="nofollow"}. + `append_keys` is not thread-safe, please see [RFC](https://github.com/aws-powertools/powertools-lambda-python/issues/991){target="_blank"}. You can append your own keys to your existing Logger via `append_keys(**additional_key_values)` method. @@ -242,7 +242,7 @@ You can remove any additional key from Logger state using `remove_keys`. #### Clearing all state -Logger is commonly initialized in the global scope. Due to [Lambda Execution Context reuse](https://docs.aws.amazon.com/lambda/latest/dg/runtimes-context.html){target="_blank" rel="nofollow"}, this means that custom keys can be persisted across invocations. If you want all custom keys to be deleted, you can use `clear_state=True` param in `inject_lambda_context` decorator. +Logger is commonly initialized in the global scope. Due to [Lambda Execution Context reuse](https://docs.aws.amazon.com/lambda/latest/dg/runtimes-context.html){target="_blank"}, this means that custom keys can be persisted across invocations. If you want all custom keys to be deleted, you can use `clear_state=True` param in `inject_lambda_context` decorator. ???+ tip "Tip: When is this useful?" It is useful when you add multiple custom keys conditionally, instead of setting a default `None` value if not present. Any key with `None` value is automatically removed by Logger. @@ -360,7 +360,7 @@ You can use any of the following built-in JMESPath expressions as part of [injec ### Reusing Logger across your code -Similar to [Tracer](./tracer.md#reusing-tracer-across-your-code){target="_blank" rel="nofollow"}, a new instance that uses the same `service` name - env var or explicit parameter - will reuse a previous Logger instance. Just like `logging.getLogger("logger_name")` would in the standard library if called with the same logger name. +Similar to [Tracer](./tracer.md#reusing-tracer-across-your-code){target="_blank"}, a new instance that uses the same `service` name - env var or explicit parameter - will reuse a previous Logger instance. Just like `logging.getLogger("logger_name")` would in the standard library if called with the same logger name. Notice in the CloudWatch Logs output how `payment_id` appeared as expected when logging in `collect.py`. @@ -407,7 +407,7 @@ You can use values ranging from `0.0` to `1` (100%) when setting `POWERTOOLS_LOG Sampling decision happens at the Logger initialization. This means sampling may happen significantly more or less than depending on your traffic patterns, for example a steady low number of invocations and thus few cold starts. ???+ note - Open a [feature request](https://github.com/aws-powertools/powertools-lambda-python/issues/new?assignees=&labels=feature-request%2C+triage&template=feature_request.md&title=){target="_blank" rel="nofollow"} if you want Logger to calculate sampling for every invocation + Open a [feature request](https://github.com/aws-powertools/powertools-lambda-python/issues/new?assignees=&labels=feature-request%2C+triage&template=feature_request.md&title=){target="_blank"} if you want Logger to calculate sampling for every invocation === "sampling_debug_logs.py" @@ -449,7 +449,7 @@ If you prefer configuring it separately, or you'd want to bring this JSON Format !!! note "In this context, an observability provider is an [AWS Lambda Partner](https://go.aws/3HtU6CZ){target="_blank" rel="nofollow"} offering a platform for logging, metrics, traces, etc." -You can send logs to the observability provider of your choice via [Lambda Extensions](https://aws.amazon.com/blogs/compute/using-aws-lambda-extensions-to-send-logs-to-custom-destinations/){target="_blank" rel="nofollow"}. In most cases, you shouldn't need any custom Logger configuration, and logs will be shipped async without any performance impact. +You can send logs to the observability provider of your choice via [Lambda Extensions](https://aws.amazon.com/blogs/compute/using-aws-lambda-extensions-to-send-logs-to-custom-destinations/){target="_blank"}. In most cases, you shouldn't need any custom Logger configuration, and logs will be shipped async without any performance impact. #### Built-in formatters @@ -744,4 +744,4 @@ Here's an example where we persist `payment_id` not `request_id`. Note that `pay ### How do I aggregate and search Powertools for AWS Lambda (Python) logs across accounts? -As of now, ElasticSearch (ELK) or 3rd party solutions are best suited to this task. Please refer to this [discussion for more details](https://github.com/aws-powertools/powertools-lambda-python/issues/460){target="_blank" rel="nofollow"} +As of now, ElasticSearch (ELK) or 3rd party solutions are best suited to this task. Please refer to this [discussion for more details](https://github.com/aws-powertools/powertools-lambda-python/issues/460){target="_blank"} diff --git a/docs/core/metrics.md b/docs/core/metrics.md index 289068aeeeb..63ee307e789 100644 --- a/docs/core/metrics.md +++ b/docs/core/metrics.md @@ -3,9 +3,9 @@ title: Metrics description: Core utility --- -Metrics creates custom metrics asynchronously by logging metrics to standard output following [Amazon CloudWatch Embedded Metric Format (EMF)](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch_Embedded_Metric_Format.html){target="_blank" rel="nofollow"}. +Metrics creates custom metrics asynchronously by logging metrics to standard output following [Amazon CloudWatch Embedded Metric Format (EMF)](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch_Embedded_Metric_Format.html){target="_blank"}. -These metrics can be visualized through [Amazon CloudWatch Console](https://console.aws.amazon.com/cloudwatch/){target="_blank" rel="nofollow"}. +These metrics can be visualized through [Amazon CloudWatch Console](https://console.aws.amazon.com/cloudwatch/){target="_blank"}. ## Key features @@ -22,7 +22,7 @@ If you're new to Amazon CloudWatch, there are two terminologies you must be awar * **Dimensions**. Metrics metadata in key-value format. They help you slice and dice metrics visualization, for example `ColdStart` metric by Payment `service`. * **Metric**. It's the name of the metric, for example: `SuccessfulBooking` or `UpdatedBooking`. * **Unit**. It's a value representing the unit of measure for the corresponding metric, for example: `Count` or `Seconds`. -* **Resolution**. It's a value representing the storage resolution for the corresponding metric. Metrics can be either Standard or High resolution. Read more [here](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/publishingMetrics.html#high-resolution-metrics){target="_blank" rel="nofollow"}. +* **Resolution**. It's a value representing the storage resolution for the corresponding metric. Metrics can be either Standard or High resolution. Read more [here](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/publishingMetrics.html#high-resolution-metrics){target="_blank"}.
@@ -32,7 +32,7 @@ If you're new to Amazon CloudWatch, there are two terminologies you must be awar ## Getting started ???+ tip - All examples shared in this documentation are available within the [project repository](https://github.com/aws-powertools/powertools-lambda-python/tree/develop/examples){target="_blank" rel="nofollow"}. + All examples shared in this documentation are available within the [project repository](https://github.com/aws-powertools/powertools-lambda-python/tree/develop/examples){target="_blank"}. Metric has two global settings that will be used across all metrics emitted: @@ -83,7 +83,7 @@ You can create metrics using `add_metric`, and you can create dimensions for all ### Adding high-resolution metrics -You can create [high-resolution metrics](https://aws.amazon.com/about-aws/whats-new/2023/02/amazon-cloudwatch-high-resolution-metric-extraction-structured-logs/){target="_blank" rel="nofollow"} passing `resolution` parameter to `add_metric`. +You can create [high-resolution metrics](https://aws.amazon.com/about-aws/whats-new/2023/02/amazon-cloudwatch-high-resolution-metric-extraction-structured-logs/){target="_blank"} passing `resolution` parameter to `add_metric`. ???+ tip "When is it useful?" High-resolution metrics are data with a granularity of one second and are very useful in several situations such as telemetry, time series, real-time incident management, and others. @@ -154,7 +154,7 @@ This decorator also **validates**, **serializes**, and **flushes** all your metr * Maximum of 29 user-defined dimensions * Namespace is set, and no more than one - * Metric units must be [supported by CloudWatch](https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_MetricDatum.html){target="_blank" rel="nofollow"} + * Metric units must be [supported by CloudWatch](https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_MetricDatum.html){target="_blank"} #### Raising SchemaValidationError on empty metrics @@ -191,7 +191,7 @@ If it's a cold start invocation, this feature will: This has the advantage of keeping cold start metric separate from your application metrics, where you might have unrelated dimensions. ???+ info - We do not emit 0 as a value for ColdStart metric for cost reasons. [Let us know](https://github.com/aws-powertools/powertools-lambda-python/issues/new?assignees=&labels=feature-request%2C+triage&template=feature_request.md&title=){target="_blank" rel="nofollow"} if you'd prefer a flag to override it. + We do not emit 0 as a value for ColdStart metric for cost reasons. [Let us know](https://github.com/aws-powertools/powertools-lambda-python/issues/new?assignees=&labels=feature-request%2C+triage&template=feature_request.md&title=){target="_blank"} if you'd prefer a flag to override it. ## Advanced @@ -219,7 +219,7 @@ You can add high-cardinality data as part of your Metrics log with `add_metadata CloudWatch EMF uses the same dimensions across all your metrics. Use `single_metric` if you have a metric that should have different dimensions. ???+ info - Generally, this would be an edge case since you [pay for unique metric](https://aws.amazon.com/cloudwatch/pricing){target="_blank" rel="nofollow"}. Keep the following formula in mind: + Generally, this would be an edge case since you [pay for unique metric](https://aws.amazon.com/cloudwatch/pricing){target="_blank"}. Keep the following formula in mind: **unique metric = (metric_name + dimension_name + dimension_value)** @@ -282,17 +282,17 @@ You can use `EphemeralMetrics` class when looking to isolate multiple instances !!! question "Why not changing the default `Metrics` behaviour to not share data across instances?" -This is an intentional design to prevent accidental data deduplication or data loss issues due to [CloudWatch EMF](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch_Embedded_Metric_Format_Specification.html){target="_blank" rel="nofollow"} metric dimension constraint. +This is an intentional design to prevent accidental data deduplication or data loss issues due to [CloudWatch EMF](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch_Embedded_Metric_Format_Specification.html){target="_blank"} metric dimension constraint. -In CloudWatch, there are two metric ingestion mechanisms: [EMF (async)](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch_Embedded_Metric_Format_Specification.html){target="_blank" rel="nofollow"} and [`PutMetricData` API (sync)](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/cloudwatch.html#CloudWatch.Client.put_metric_data){target="_blank" rel="nofollow"}. +In CloudWatch, there are two metric ingestion mechanisms: [EMF (async)](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch_Embedded_Metric_Format_Specification.html){target="_blank"} and [`PutMetricData` API (sync)](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/cloudwatch.html#CloudWatch.Client.put_metric_data){target="_blank"}. The former creates metrics asynchronously via CloudWatch Logs, and the latter uses a synchronous and more flexible ingestion API. !!! important "Key concept" - CloudWatch [considers a metric unique](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/cloudwatch_concepts.html#Metric){target="_blank" rel="nofollow"} by a combination of metric **name**, metric **namespace**, and zero or more metric **dimensions**. + CloudWatch [considers a metric unique](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/cloudwatch_concepts.html#Metric){target="_blank"} by a combination of metric **name**, metric **namespace**, and zero or more metric **dimensions**. -With EMF, metric dimensions are shared with any metrics you define. With `PutMetricData` API, you can set a [list](https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_MetricDatum.html){target="_blank" rel="nofollow"} defining one or more metrics with distinct dimensions. +With EMF, metric dimensions are shared with any metrics you define. With `PutMetricData` API, you can set a [list](https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_MetricDatum.html){target="_blank"} defining one or more metrics with distinct dimensions. This is a subtle yet important distinction. Imagine you had the following metrics to emit: @@ -374,4 +374,4 @@ You can read standard output and assert whether metrics have been flushed. Here' ``` ???+ tip - For more elaborate assertions and comparisons, check out [our functional testing for Metrics utility.](https://github.com/aws-powertools/powertools-lambda-python/blob/develop/tests/functional/test_metrics.py){target="_blank" rel="nofollow"} + For more elaborate assertions and comparisons, check out [our functional testing for Metrics utility.](https://github.com/aws-powertools/powertools-lambda-python/blob/develop/tests/functional/test_metrics.py){target="_blank"} diff --git a/docs/core/tracer.md b/docs/core/tracer.md index 6ffd44d3be0..413434718db 100644 --- a/docs/core/tracer.md +++ b/docs/core/tracer.md @@ -17,19 +17,19 @@ Tracer is an opinionated thin wrapper for [AWS X-Ray Python SDK](https://github. ## Getting started ???+ tip - All examples shared in this documentation are available within the [project repository](https://github.com/aws-powertools/powertools-lambda-python/tree/develop/examples){target="_blank" rel="nofollow"}. + All examples shared in this documentation are available within the [project repository](https://github.com/aws-powertools/powertools-lambda-python/tree/develop/examples){target="_blank"}. !!! note "Tracer relies on AWS X-Ray SDK over [OpenTelememetry Distro (ADOT)](https://aws-otel.github.io/docs/getting-started/lambda){target="_blank" rel="nofollow"} for optimal cold start (lower latency)." ### Install -!!! info "This is not necessary if you're installing Powertools for AWS Lambda (Python) via [Lambda Layer/SAR](../index.md#lambda-layer){target="_blank" rel="nofollow"}" +!!! info "This is not necessary if you're installing Powertools for AWS Lambda (Python) via [Lambda Layer/SAR](../index.md#lambda-layer){target="_blank"}" Add `aws-lambda-powertools[tracer]` as a dependency in your preferred tool: _e.g._, _requirements.txt_, _pyproject.toml_. This will ensure you have the required dependencies before using Tracer. ### Permissions -Before your use this utility, your AWS Lambda function [must have permissions](https://docs.aws.amazon.com/lambda/latest/dg/services-xray.html#services-xray-permissions){target="_blank" rel="nofollow"} to send traces to AWS X-Ray. +Before your use this utility, your AWS Lambda function [must have permissions](https://docs.aws.amazon.com/lambda/latest/dg/services-xray.html#services-xray-permissions){target="_blank"} to send traces to AWS X-Ray. ```yaml hl_lines="9 12" title="AWS Serverless Application Model (SAM) example" --8<-- "examples/tracer/sam/template.yaml" @@ -51,7 +51,7 @@ You can quickly start by initializing `Tracer` and use `capture_lambda_handler` ### Annotations & Metadata -**Annotations** are key-values associated with traces and indexed by AWS X-Ray. You can use them to filter traces and to create [Trace Groups](https://aws.amazon.com/about-aws/whats-new/2018/11/aws-xray-adds-the-ability-to-group-traces/){target="_blank" rel="nofollow"} to slice and dice your transactions. +**Annotations** are key-values associated with traces and indexed by AWS X-Ray. You can use them to filter traces and to create [Trace Groups](https://aws.amazon.com/about-aws/whats-new/2018/11/aws-xray-adds-the-ability-to-group-traces/){target="_blank"} to slice and dice your transactions. ```python hl_lines="8" title="Adding annotations with put_annotation method" --8<-- "examples/tracer/src/put_trace_annotations.py" @@ -107,7 +107,7 @@ You can trace asynchronous functions and generator functions (including context ### Patching modules -Tracer automatically patches all [supported libraries by X-Ray](https://docs.aws.amazon.com/xray/latest/devguide/xray-sdk-python-patching.html){target="_blank" rel="nofollow"} during initialization, by default. Underneath, AWS X-Ray SDK checks whether a supported library has been imported before patching. +Tracer automatically patches all [supported libraries by X-Ray](https://docs.aws.amazon.com/xray/latest/devguide/xray-sdk-python-patching.html){target="_blank"} during initialization, by default. Underneath, AWS X-Ray SDK checks whether a supported library has been imported before patching. If you're looking to shave a few microseconds, or milliseconds depending on your function memory configuration, you can patch specific modules using `patch_modules` param: @@ -172,7 +172,7 @@ You can use `aiohttp_trace_config` function to create a valid [aiohttp trace_con You can use `tracer.provider` attribute to access all methods provided by AWS X-Ray `xray_recorder` object. -This is useful when you need a feature available in X-Ray that is not available in the Tracer utility, for example [thread-safe](https://github.com/aws/aws-xray-sdk-python/#user-content-trace-threadpoolexecutor){target="_blank" rel="nofollow"}, or [context managers](https://github.com/aws/aws-xray-sdk-python/#user-content-start-a-custom-segmentsubsegment){target="_blank" rel="nofollow"}. +This is useful when you need a feature available in X-Ray that is not available in the Tracer utility, for example [thread-safe](https://github.com/aws/aws-xray-sdk-python/#user-content-trace-threadpoolexecutor){target="_blank"}, or [context managers](https://github.com/aws/aws-xray-sdk-python/#user-content-start-a-custom-segmentsubsegment){target="_blank"}. ```python hl_lines="14" title="Tracing a code block with in_subsegment escape hatch" --8<-- "examples/tracer/src/sdk_escape_hatch.py" @@ -181,7 +181,7 @@ This is useful when you need a feature available in X-Ray that is not available ### Concurrent asynchronous functions ???+ warning - [X-Ray SDK will raise an exception](https://github.com/aws/aws-xray-sdk-python/issues/164){target="_blank" rel="nofollow"} when async functions are run and traced concurrently + [X-Ray SDK will raise an exception](https://github.com/aws/aws-xray-sdk-python/issues/164){target="_blank"} when async functions are run and traced concurrently A safe workaround mechanism is to use `in_subsegment_async` available via Tracer escape hatch (`tracer.provider`). @@ -221,4 +221,4 @@ Tracer is disabled by default when not running in the AWS Lambda environment - T * Use annotations on key operations to slice and dice traces, create unique views, and create metrics from it via Trace Groups * Use a namespace when adding metadata to group data more easily -* Annotations and metadata are added to the current subsegment opened. If you want them in a specific subsegment, use a [context manager](https://github.com/aws/aws-xray-sdk-python/#start-a-custom-segmentsubsegment){target="_blank" rel="nofollow"} via the escape hatch mechanism +* Annotations and metadata are added to the current subsegment opened. If you want them in a specific subsegment, use a [context manager](https://github.com/aws/aws-xray-sdk-python/#start-a-custom-segmentsubsegment){target="_blank"} via the escape hatch mechanism diff --git a/docs/index.md b/docs/index.md index e6d262f62e6..d0269260711 100644 --- a/docs/index.md +++ b/docs/index.md @@ -8,15 +8,15 @@ description: Powertools for AWS Lambda (Python) Powertools for AWS Lambda (Python) is a developer toolkit to implement Serverless best practices and increase developer velocity. ???+ tip - Powertools for AWS Lambda (Python) is also available for [Java](https://docs.powertools.aws.dev/lambda/java/){target="_blank" rel="nofollow"}, [TypeScript](https://docs.powertools.aws.dev/lambda/typescript/latest/){target="_blank" rel="nofollow"}, and [.NET](https://docs.powertools.aws.dev/lambda/dotnet/){target="_blank" rel="nofollow"} + Powertools for AWS Lambda (Python) is also available for [Java](https://docs.powertools.aws.dev/lambda/java/){target="_blank"}, [TypeScript](https://docs.powertools.aws.dev/lambda/typescript/latest/){target="_blank" }, and [.NET](https://docs.powertools.aws.dev/lambda/dotnet/){target="_blank"} ??? hint "Support this project by becoming a reference customer, sharing your work, or using Layers/SAR :heart:" You can choose to support us in three ways: - 1) [**Become a reference customer**](https://github.com/aws-powertools/powertools-lambda-python/issues/new?assignees=&labels=customer-reference&template=support_powertools.yml&title=%5BSupport+Lambda+Powertools%5D%3A+%3Cyour+organization+name%3E){target="_blank" rel="nofollow"}. This gives us permission to list your company in our documentation. + 1) [**Become a reference customer**](https://github.com/aws-powertools/powertools-lambda-python/issues/new?assignees=&labels=customer-reference&template=support_powertools.yml&title=%5BSupport+Lambda+Powertools%5D%3A+%3Cyour+organization+name%3E){target="_blank"}. This gives us permission to list your company in our documentation. - 2) [**Share your work**](https://github.com/aws-powertools/powertools-lambda-python/issues/new?assignees=&labels=community-content&template=share_your_work.yml&title=%5BI+Made+This%5D%3A+%3CTITLE%3E){target="_blank" rel="nofollow"}. Blog posts, video, sample projects you used Powertools! + 2) [**Share your work**](https://github.com/aws-powertools/powertools-lambda-python/issues/new?assignees=&labels=community-content&template=share_your_work.yml&title=%5BI+Made+This%5D%3A+%3CTITLE%3E){target="_blank"}. Blog posts, video, sample projects you used Powertools! 3) Use [**Lambda Layers**](#lambda-layer) or [**SAR**](#sar), if possible. This helps us understand who uses Powertools for AWS Lambda (Python) in a non-intrusive way, and helps us gain future investments for other Powertools for AWS Lambda languages. @@ -33,7 +33,7 @@ You can install Powertools for AWS Lambda (Python) using one of the following op !!! question "Looking for Pip signed releases? [Learn more about verifying signed builds](./security.md#verifying-signed-builds)" ??? question "Using Pip? You might need to install additional dependencies." - [**Tracer**](./core/tracer.md){target="_blank" rel="nofollow"}, [**Validation**](./utilities/validation.md){target="_blank" rel="nofollow"} and [**Parser**](./utilities/parser.md){target="_blank" rel="nofollow"} require additional dependencies. If you prefer to install all of them, use [**`pip install "aws-lambda-powertools[all]"`**](#){: .copyMe}:clipboard:. + [**Tracer**](./core/tracer.md){target="_blank"}, [**Validation**](./utilities/validation.md){target="_blank"} and [**Parser**](./utilities/parser.md){target="_blank"} require additional dependencies. If you prefer to install all of them, use [**`pip install "aws-lambda-powertools[all]"`**](#){: .copyMe}:clipboard:. For example: @@ -46,7 +46,7 @@ You can install Powertools for AWS Lambda (Python) using one of the following op !!! info "Using Lambda Layer? Simply add [**`"aws-lambda-powertools[all]"`**](#){: .copyMe}:clipboard: as a development dependency." -Powertools for AWS Lambda (Python) relies on the [AWS SDK bundled in the Lambda runtime](https://docs.aws.amazon.com/lambda/latest/dg/lambda-python.html){target="_blank" rel="nofollow"}. This helps us achieve an optimal package size and initialization. However, when developing locally, you need to install AWS SDK as a development dependency (not as a production dependency): +Powertools for AWS Lambda (Python) relies on the [AWS SDK bundled in the Lambda runtime](https://docs.aws.amazon.com/lambda/latest/dg/lambda-python.html){target="_blank"}. This helps us achieve an optimal package size and initialization. However, when developing locally, you need to install AWS SDK as a development dependency (not as a production dependency): * **Pip**: [**`pip install "aws-lambda-powertools[aws-sdk]"`**](#){: .copyMe}:clipboard: * **Poetry**: [**`poetry add "aws-lambda-powertools[aws-sdk]" --group dev`**](#){: .copyMe}:clipboard: @@ -55,24 +55,24 @@ Powertools for AWS Lambda (Python) relies on the [AWS SDK bundled in the Lambda ??? question "Why is that necessary?" Powertools for AWS Lambda (Python) relies on the AWS SDK being available to use in the target runtime (AWS Lambda). - As a result, it affects your favorite IDE in terms of code auto-completion, or running your tests suite locally with no Lambda emulation such as [AWS SAM CLI](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/install-sam-cli.html){target="_blank" rel="nofollow"}. + As a result, it affects your favorite IDE in terms of code auto-completion, or running your tests suite locally with no Lambda emulation such as [AWS SAM CLI](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/install-sam-cli.html){target="_blank"}. **A word about dependency resolution** In this context, `[aws-sdk]` is an alias to the `boto3` package. Due to dependency resolution, it'll either install: -* **(A)** the SDK version available in [Lambda runtime](https://docs.aws.amazon.com/lambda/latest/dg/lambda-python.html){target="_blank" rel="nofollow"} -* **(B)** a more up-to-date version if another package you use also depends on `boto3`, for example [Powertools for AWS Lambda (Python) Tracer](core/tracer.md){target="_blank" rel="nofollow"} +* **(A)** the SDK version available in [Lambda runtime](https://docs.aws.amazon.com/lambda/latest/dg/lambda-python.html){target="_blank"} +* **(B)** a more up-to-date version if another package you use also depends on `boto3`, for example [Powertools for AWS Lambda (Python) Tracer](core/tracer.md){target="_blank"} ### Lambda Layer ???+ warning "As of now, Container Image deployment (OCI) or inline Lambda functions do not support Lambda Layers." -[Lambda Layer](https://docs.aws.amazon.com/lambda/latest/dg/configuration-layers.html){target="_blank" rel="nofollow"} is a .zip file archive that can contain additional code, pre-packaged dependencies, data, or configuration files. Layers promote code sharing and separation of responsibilities so that you can iterate faster on writing business logic. +[Lambda Layer](https://docs.aws.amazon.com/lambda/latest/dg/configuration-layers.html){target="_blank"} is a .zip file archive that can contain additional code, pre-packaged dependencies, data, or configuration files. Layers promote code sharing and separation of responsibilities so that you can iterate faster on writing business logic. -For our Layers, we compile and optimize [all dependencies](https://github.com/aws-powertools/powertools-lambda-python/blob/develop/pyproject.toml#L98){target="_blank" rel="nofollow"}, and [remove duplicate dependencies already available in the Lambda runtime](https://github.com/awslabs/cdk-aws-lambda-powertools-layer/blob/main/layer/Python/Dockerfile#L36){target="_blank" rel="nofollow"} to achieve the most optimal size. +For our Layers, we compile and optimize [all dependencies](https://github.com/aws-powertools/powertools-lambda-python/blob/develop/pyproject.toml#L98){target="_blank"}, and [remove duplicate dependencies already available in the Lambda runtime](https://github.com/awslabs/cdk-aws-lambda-powertools-layer/blob/main/layer/Python/Dockerfile#L36){target="_blank"} to achieve the most optimal size. -You can include Powertools for AWS Lambda (Python) Lambda Layer using [AWS Lambda Console](https://docs.aws.amazon.com/lambda/latest/dg/invocation-layers.html#invocation-layers-using){target="_blank" rel="nofollow"}, or your preferred deployment framework. +You can include Powertools for AWS Lambda (Python) Lambda Layer using [AWS Lambda Console](https://docs.aws.amazon.com/lambda/latest/dg/invocation-layers.html#invocation-layers-using){target="_blank"}, or your preferred deployment framework. ??? note "Note: Click to expand and copy any regional Lambda Layer ARN" @@ -464,8 +464,8 @@ Compared with the [public Layer ARN](#lambda-layer) option, SAR allows you to ch | App | ARN | Description | | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------- | -| [aws-lambda-powertools-python-layer](https://serverlessrepo.aws.amazon.com/applications/eu-west-1/057560766410/aws-lambda-powertools-python-layer){target="_blank" rel="nofollow"} | [arn:aws:serverlessrepo:eu-west-1:057560766410:applications/aws-lambda-powertools-python-layer](#){: .copyMe}:clipboard: | Contains all extra dependencies (e.g: pydantic). | -| [aws-lambda-powertools-python-layer-arm64](https://serverlessrepo.aws.amazon.com/applications/eu-west-1/057560766410/aws-lambda-powertools-python-layer-arm64){target="_blank" rel="nofollow"} | [arn:aws:serverlessrepo:eu-west-1:057560766410:applications/aws-lambda-powertools-python-layer-arm64](#){: .copyMe}:clipboard: | Contains all extra dependencies (e.g: pydantic). For arm64 functions. | +| [aws-lambda-powertools-python-layer](https://serverlessrepo.aws.amazon.com/applications/eu-west-1/057560766410/aws-lambda-powertools-python-layer){target="_blank"} | [arn:aws:serverlessrepo:eu-west-1:057560766410:applications/aws-lambda-powertools-python-layer](#){: .copyMe}:clipboard: | Contains all extra dependencies (e.g: pydantic). | +| [aws-lambda-powertools-python-layer-arm64](https://serverlessrepo.aws.amazon.com/applications/eu-west-1/057560766410/aws-lambda-powertools-python-layer-arm64){target="_blank"} | [arn:aws:serverlessrepo:eu-west-1:057560766410:applications/aws-lambda-powertools-python-layer-arm64](#){: .copyMe}:clipboard: | Contains all extra dependencies (e.g: pydantic). For arm64 functions. | ??? note "Click to expand and copy SAR code snippets for popular frameworks" @@ -681,21 +681,21 @@ Core utilities such as Tracing, Logging, Metrics, and Event Handler will be avai | Utility | Description | | --------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | -| [**Tracing**](./core/tracer.md){target="_blank" rel="nofollow"} | Decorators and utilities to trace Lambda function handlers, and both synchronous and asynchronous functions | -| [**Logger**](./core/logger.md){target="_blank" rel="nofollow"} | Structured logging made easier, and decorator to enrich structured logging with key Lambda context details | -| [**Metrics**](./core/metrics.md){target="_blank" rel="nofollow"} | Custom Metrics created asynchronously via CloudWatch Embedded Metric Format (EMF) | -| [**Event handler: AppSync**](./core/event_handler/appsync.md){target="_blank" rel="nofollow"} | AppSync event handler for Lambda Direct Resolver and Amplify GraphQL Transformer function | +| [**Tracing**](./core/tracer.md){target="_blank"} | Decorators and utilities to trace Lambda function handlers, and both synchronous and asynchronous functions | +| [**Logger**](./core/logger.md){target="_blank"} | Structured logging made easier, and decorator to enrich structured logging with key Lambda context details | +| [**Metrics**](./core/metrics.md){target="_blank"} | Custom Metrics created asynchronously via CloudWatch Embedded Metric Format (EMF) | +| [**Event handler: AppSync**](./core/event_handler/appsync.md){target="_blank"} | AppSync event handler for Lambda Direct Resolver and Amplify GraphQL Transformer function | | [**Event handler: API Gateway, ALB and Lambda Function URL**](https://docs.powertools.aws.dev/lambda/python/latest/core/event_handler/api_gateway/) | Amazon API Gateway REST/HTTP API and ALB event handler for Lambda functions invoked using Proxy integration, and Lambda Function URL | -| [**Middleware factory**](./utilities/middleware_factory.md){target="_blank" rel="nofollow"} | Decorator factory to create your own middleware to run logic before, and after each Lambda invocation | -| [**Parameters**](./utilities/parameters.md){target="_blank" rel="nofollow"} | Retrieve parameter values from AWS Systems Manager Parameter Store, AWS Secrets Manager, or Amazon DynamoDB, and cache them for a specific amount of time | -| [**Batch processing**](./utilities/batch.md){target="_blank" rel="nofollow"} | Handle partial failures for AWS SQS batch processing | -| [**Typing**](./utilities/typing.md){target="_blank" rel="nofollow"} | Static typing classes to speedup development in your IDE | -| [**Validation**](./utilities/validation.md){target="_blank" rel="nofollow"} | JSON Schema validator for inbound events and responses | -| [**Event source data classes**](./utilities/data_classes.md){target="_blank" rel="nofollow"} | Data classes describing the schema of common Lambda event triggers | -| [**Parser**](./utilities/parser.md){target="_blank" rel="nofollow"} | Data parsing and deep validation using Pydantic | -| [**Idempotency**](./utilities/idempotency.md){target="_blank" rel="nofollow"} | Idempotent Lambda handler | -| [**Feature Flags**](./utilities/feature_flags.md){target="_blank" rel="nofollow"} | A simple rule engine to evaluate when one or multiple features should be enabled depending on the input | -| [**Streaming**](./utilities/streaming.md){target="_blank" rel="nofollow"} | Streams datasets larger than the available memory as streaming data. | +| [**Middleware factory**](./utilities/middleware_factory.md){target="_blank"} | Decorator factory to create your own middleware to run logic before, and after each Lambda invocation | +| [**Parameters**](./utilities/parameters.md){target="_blank"} | Retrieve parameter values from AWS Systems Manager Parameter Store, AWS Secrets Manager, or Amazon DynamoDB, and cache them for a specific amount of time | +| [**Batch processing**](./utilities/batch.md){target="_blank"} | Handle partial failures for AWS SQS batch processing | +| [**Typing**](./utilities/typing.md){target="_blank"} | Static typing classes to speedup development in your IDE | +| [**Validation**](./utilities/validation.md){target="_blank"} | JSON Schema validator for inbound events and responses | +| [**Event source data classes**](./utilities/data_classes.md){target="_blank"} | Data classes describing the schema of common Lambda event triggers | +| [**Parser**](./utilities/parser.md){target="_blank"} | Data parsing and deep validation using Pydantic | +| [**Idempotency**](./utilities/idempotency.md){target="_blank"} | Idempotent Lambda handler | +| [**Feature Flags**](./utilities/feature_flags.md){target="_blank"} | A simple rule engine to evaluate when one or multiple features should be enabled depending on the input | +| [**Streaming**](./utilities/streaming.md){target="_blank"} | Streams datasets larger than the available memory as streaming data. | ## Environment variables @@ -705,18 +705,18 @@ Core utilities such as Tracing, Logging, Metrics, and Event Handler will be avai | Environment variable | Description | Utility | Default | | ----------------------------------------- | -------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------- | --------------------- | | **POWERTOOLS_SERVICE_NAME** | Sets service name used for tracing namespace, metrics dimension and structured logging | All | `"service_undefined"` | -| **POWERTOOLS_METRICS_NAMESPACE** | Sets namespace used for metrics | [Metrics](./core/metrics){target="_blank" rel="nofollow"} | `None` | -| **POWERTOOLS_TRACE_DISABLED** | Explicitly disables tracing | [Tracing](./core/tracer){target="_blank" rel="nofollow"} | `false` | -| **POWERTOOLS_TRACER_CAPTURE_RESPONSE** | Captures Lambda or method return as metadata. | [Tracing](./core/tracer){target="_blank" rel="nofollow"} | `true` | -| **POWERTOOLS_TRACER_CAPTURE_ERROR** | Captures Lambda or method exception as metadata. | [Tracing](./core/tracer){target="_blank" rel="nofollow"} | `true` | -| **POWERTOOLS_TRACE_MIDDLEWARES** | Creates sub-segment for each custom middleware | [Middleware factory](./utilities/middleware_factory){target="_blank" rel="nofollow"} | `false` | -| **POWERTOOLS_LOGGER_LOG_EVENT** | Logs incoming event | [Logging](./core/logger){target="_blank" rel="nofollow"} | `false` | -| **POWERTOOLS_LOGGER_SAMPLE_RATE** | Debug log sampling | [Logging](./core/logger){target="_blank" rel="nofollow"} | `0` | -| **POWERTOOLS_LOG_DEDUPLICATION_DISABLED** | Disables log deduplication filter protection to use Pytest Live Log feature | [Logging](./core/logger){target="_blank" rel="nofollow"} | `false` | -| **POWERTOOLS_PARAMETERS_MAX_AGE** | Adjust how long values are kept in cache (in seconds) | [Parameters](./utilities/parameters/#adjusting-cache-ttl){target="_blank" rel="nofollow"} | `5` | -| **POWERTOOLS_PARAMETERS_SSM_DECRYPT** | Sets whether to decrypt or not values retrieved from AWS SSM Parameters Store | [Parameters](./utilities/parameters/#ssmprovider){target="_blank" rel="nofollow"} | `false` | +| **POWERTOOLS_METRICS_NAMESPACE** | Sets namespace used for metrics | [Metrics](./core/metrics){target="_blank"} | `None` | +| **POWERTOOLS_TRACE_DISABLED** | Explicitly disables tracing | [Tracing](./core/tracer){target="_blank"} | `false` | +| **POWERTOOLS_TRACER_CAPTURE_RESPONSE** | Captures Lambda or method return as metadata. | [Tracing](./core/tracer){target="_blank"} | `true` | +| **POWERTOOLS_TRACER_CAPTURE_ERROR** | Captures Lambda or method exception as metadata. | [Tracing](./core/tracer){target="_blank"} | `true` | +| **POWERTOOLS_TRACE_MIDDLEWARES** | Creates sub-segment for each custom middleware | [Middleware factory](./utilities/middleware_factory){target="_blank"} | `false` | +| **POWERTOOLS_LOGGER_LOG_EVENT** | Logs incoming event | [Logging](./core/logger){target="_blank"} | `false` | +| **POWERTOOLS_LOGGER_SAMPLE_RATE** | Debug log sampling | [Logging](./core/logger){target="_blank"} | `0` | +| **POWERTOOLS_LOG_DEDUPLICATION_DISABLED** | Disables log deduplication filter protection to use Pytest Live Log feature | [Logging](./core/logger){target="_blank"} | `false` | +| **POWERTOOLS_PARAMETERS_MAX_AGE** | Adjust how long values are kept in cache (in seconds) | [Parameters](./utilities/parameters/#adjusting-cache-ttl){target="_blank"} | `5` | +| **POWERTOOLS_PARAMETERS_SSM_DECRYPT** | Sets whether to decrypt or not values retrieved from AWS SSM Parameters Store | [Parameters](./utilities/parameters/#ssmprovider){target="_blank"} | `false` | | **POWERTOOLS_DEV** | Increases verbosity across utilities | Multiple; see [POWERTOOLS_DEV effect below](#optimizing-for-non-production-environments) | `false` | -| **LOG_LEVEL** | Sets logging level | [Logging](./core/logger){target="_blank" rel="nofollow"} | `INFO` | +| **LOG_LEVEL** | Sets logging level | [Logging](./core/logger){target="_blank"} | `INFO` | ### Optimizing for non-production environments @@ -743,7 +743,7 @@ When necessary, you can use `POWERTOOLS_DEBUG` environment variable to enable de ### Becoming a reference customer -Knowing which companies are using this library is important to help prioritize the project internally. If your company is using Powertools for AWS Lambda (Python), you can request to have your name and logo added to the README file by raising a [Support Powertools for AWS Lambda (Python) (become a reference)](https://github.com/aws-powertools/powertools-lambda-python/issues/new?assignees=&labels=customer-reference&template=support_powertools.yml&title=%5BSupport+Lambda+Powertools%5D%3A+%3Cyour+organization+name%3E){target="_blank" rel="nofollow"} issue. +Knowing which companies are using this library is important to help prioritize the project internally. If your company is using Powertools for AWS Lambda (Python), you can request to have your name and logo added to the README file by raising a [Support Powertools for AWS Lambda (Python) (become a reference)](https://github.com/aws-powertools/powertools-lambda-python/issues/new?assignees=&labels=customer-reference&template=support_powertools.yml&title=%5BSupport+Lambda+Powertools%5D%3A+%3Cyour+organization+name%3E){target="_blank"} issue. The following companies, among others, use Powertools: @@ -760,7 +760,7 @@ The following companies, among others, use Powertools: ### Sharing your work -Share what you did with Powertools for AWS Lambda (Python) 💞💞. Blog post, workshops, presentation, sample apps and others. Check out what the community has already shared about Powertools for AWS Lambda (Python) [here](https://docs.powertools.aws.dev/lambda/python/latest/we_made_this/){target="_blank" rel="nofollow"}. +Share what you did with Powertools for AWS Lambda (Python) 💞💞. Blog post, workshops, presentation, sample apps and others. Check out what the community has already shared about Powertools for AWS Lambda (Python) [here](https://docs.powertools.aws.dev/lambda/python/latest/we_made_this/){target="_blank"}. ### Using Lambda Layer or SAR diff --git a/docs/maintainers.md b/docs/maintainers.md index 061acbb4c8f..455d33f6d8a 100644 --- a/docs/maintainers.md +++ b/docs/maintainers.md @@ -9,7 +9,7 @@ description: Playbook for active maintainers in Powertools for AWS Lambda (Pytho !!! note "Please treat this content as a living document." -This is document explains who the maintainers are, their responsibilities, and how they should be doing it. If you're interested in contributing, see [CONTRIBUTING](https://github.com/aws-powertools/powertools-lambda-python/blob/develop/CONTRIBUTING.md){target="_blank" rel="nofollow"}. +This is document explains who the maintainers are, their responsibilities, and how they should be doing it. If you're interested in contributing, see [CONTRIBUTING](https://github.com/aws-powertools/powertools-lambda-python/blob/develop/CONTRIBUTING.md){target="_blank"}. ## Current Maintainers @@ -81,7 +81,7 @@ Be aware of recurring ambiguous situations and [document them](#common-scenarios ### Uphold Code of Conduct -Model the behavior set forward by the [Code of Conduct](https://github.com/aws-powertools/powertools-lambda-python/blob/develop/CODE_OF_CONDUCT.md){target="_blank" rel="nofollow"} and raise any violations to other maintainers and admins. There could be unusual circumstances where inappropriate behavior does not immediately fall within the [Code of Conduct](https://github.com/aws-powertools/powertools-lambda-python/blob/develop/CODE_OF_CONDUCT.md){target="_blank" rel="nofollow"}. +Model the behavior set forward by the [Code of Conduct](https://github.com/aws-powertools/powertools-lambda-python/blob/develop/CODE_OF_CONDUCT.md){target="_blank"} and raise any violations to other maintainers and admins. There could be unusual circumstances where inappropriate behavior does not immediately fall within the [Code of Conduct](https://github.com/aws-powertools/powertools-lambda-python/blob/develop/CODE_OF_CONDUCT.md){target="_blank"}. These might be nuanced and should be handled with extra care - when in doubt, do not engage and reach out to other maintainers and admins. @@ -89,7 +89,7 @@ These might be nuanced and should be handled with extra care - when in doubt, do Security is your number one priority. Maintainer's Github keys must be password protected securely and any reported security vulnerabilities are addressed before features or bugs. -Note that this repository is monitored and supported 24/7 by Amazon Security, see [Reporting a Vulnerability](https://github.com/aws-powertools/powertools-lambda-python/blob/develop/SECURITY.md){target="_blank" rel="nofollow"} for details. +Note that this repository is monitored and supported 24/7 by Amazon Security, see [Reporting a Vulnerability](https://github.com/aws-powertools/powertools-lambda-python/blob/develop/SECURITY.md){target="_blank"} for details. ### Review Pull Requests @@ -97,7 +97,7 @@ Review pull requests regularly, comment, suggest, reject, merge and close. Accep PRs are [labeled](#labels) based on file changes and semantic title. Pay attention to whether labels reflect the current state of the PR and correct accordingly. -Use and enforce [semantic versioning](https://semver.org/) pull request titles, as these will be used for [CHANGELOG](https://github.com/aws-powertools/powertools-lambda-python/blob/develop/CHANGELOG.md){target="_blank" rel="nofollow"} and [Release notes](https://github.com/aws-powertools/powertools-lambda-python/releases) - make sure they communicate their intent at the human level. +Use and enforce [semantic versioning](https://semver.org/) pull request titles, as these will be used for [CHANGELOG](https://github.com/aws-powertools/powertools-lambda-python/blob/develop/CHANGELOG.md){target="_blank"} and [Release notes](https://github.com/aws-powertools/powertools-lambda-python/releases) - make sure they communicate their intent at the human level. > TODO: This is an area we want to automate using the new GitHub GraphQL API. @@ -289,7 +289,7 @@ Add integration checks that validate pull requests and pushes to ease the burden ### Negative Impact on the Project -Actions that negatively impact the project will be handled by the admins, in coordination with other maintainers, in balance with the urgency of the issue. Examples would be [Code of Conduct](https://github.com/aws-powertools/powertools-lambda-python/blob/develop/CODE_OF_CONDUCT.md){target="_blank" rel="nofollow"} violations, deliberate harmful or malicious actions, spam, monopolization, and security risks. +Actions that negatively impact the project will be handled by the admins, in coordination with other maintainers, in balance with the urgency of the issue. Examples would be [Code of Conduct](https://github.com/aws-powertools/powertools-lambda-python/blob/develop/CODE_OF_CONDUCT.md){target="_blank"} violations, deliberate harmful or malicious actions, spam, monopolization, and security risks. ### Becoming a maintainer @@ -315,7 +315,7 @@ When in doubt, use `need-more-information` or `need-customer-feedback` labels to ### Crediting contributions -We credit all contributions as part of each [release note](https://github.com/aws-powertools/powertools-lambda-python/releases){target="_blank" rel="nofollow"} as an automated process. If you find contributors are missing from the release note you're producing, please add them manually. +We credit all contributions as part of each [release note](https://github.com/aws-powertools/powertools-lambda-python/releases){target="_blank"} as an automated process. If you find contributors are missing from the release note you're producing, please add them manually. ### Is that a bug? @@ -388,7 +388,7 @@ Where: ### Mechanics -Under [`BaseInfrastructure`](https://github.com/aws-powertools/powertools-lambda-python/blob/develop/tests/e2e/utils/infrastructure.py){target="_blank" rel="nofollow"}, we hide the complexity of deployment and delete coordination under `deploy`, `delete`, and `create_lambda_functions` methods. +Under [`BaseInfrastructure`](https://github.com/aws-powertools/powertools-lambda-python/blob/develop/tests/e2e/utils/infrastructure.py){target="_blank"}, we hide the complexity of deployment and delete coordination under `deploy`, `delete`, and `create_lambda_functions` methods. This allows us to benefit from test and deployment parallelization, use IDE step-through debugging for a single test, run one, subset, or all tests and only deploy their related infrastructure, without any custom configuration. @@ -590,7 +590,7 @@ graph TD #### CDK CLI parallelization -For CDK CLI to work with [independent CDK Apps](https://docs.aws.amazon.com/cdk/v2/guide/apps.html){target="_blank" rel="nofollow"}, we specify an output directory when synthesizing our stack and deploy from said output directory. +For CDK CLI to work with [independent CDK Apps](https://docs.aws.amazon.com/cdk/v2/guide/apps.html){target="_blank"}, we specify an output directory when synthesizing our stack and deploy from said output directory. ```mermaid flowchart TD @@ -658,6 +658,6 @@ Where: - **`layer_build`**. Contains our Lambda Layer source code built once, used by all stacks independently - **`layer_build.diff`**. Contains a hash on whether our source code has changed to speed up further deployments and E2E tests -Together, all of this allows us to use Pytest like we would for any project, use CDK CLI and its [context methods](https://docs.aws.amazon.com/cdk/v2/guide/context.html#context_methods){target="_blank" rel="nofollow"} (`from_lookup`), and use step-through debugging for a single E2E test without any extra configuration. +Together, all of this allows us to use Pytest like we would for any project, use CDK CLI and its [context methods](https://docs.aws.amazon.com/cdk/v2/guide/context.html#context_methods){target="_blank"} (`from_lookup`), and use step-through debugging for a single E2E test without any extra configuration. > NOTE: VSCode doesn't support debugging processes spawning sub-processes (like CDK CLI does w/ shell and CDK App). Maybe [this works](https://stackoverflow.com/a/65339352){target="_blank" rel="nofollow"}. PyCharm works just fine. diff --git a/docs/roadmap.md b/docs/roadmap.md index f432c2c56f7..6b4807f6ddc 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -4,13 +4,13 @@ This is our public roadmap that outlines the high level direction we are working towards, namely [Themes](#themes). We update this document when our priorities change: security and stability is our top priority. -[See our latest list of activities »](https://github.com/orgs/awslabs/projects/51/views/1?query=is%3Aopen+sort%3Aupdated-desc){target="_blank" rel="nofollow"} +[See our latest list of activities »](https://github.com/orgs/aws-powertools/projects/3?query=is%3Aopen+sort%3Aupdated-desc){target="_blank"} ## Themes !!! info "Operational Excellence is priority number 1." -Themes are key activities maintainers are focusing on, besides bug reports. These are updated periodically and you can find the latest [under Epics in our public board](https://github.com/orgs/awslabs/projects/51/views/11?query=is%3Aopen+sort%3Aupdated-desc){target="_blank" rel="nofollow"}. +Themes are key activities maintainers are focusing on, besides bug reports. These are updated periodically and you can find the latest [under Epics in our public board](https://github.com/orgs/aws-powertools/projects/3/views/11?query=is%3Aopen+sort%3Aupdated-desc){target="_blank"}. ### Increased end-to-end coverage @@ -20,7 +20,7 @@ Some Lambda Event Sources require clusters (e.g., MSK) leading to additional del ### Observability providers -We want to extend Tracer, Metrics, and Logger to support any [observability provider](https://github.com/aws-powertools/powertools-lambda-python/issues/1433){target="_blank" rel="nofollow"}. We need a RFC to define a contract and to identify two most requested observability providers that we can work with as an initial step. +We want to extend Tracer, Metrics, and Logger to support any [observability provider](https://github.com/aws-powertools/powertools-lambda-python/issues/1433){target="_blank"}. We need a RFC to define a contract and to identify two most requested observability providers that we can work with as an initial step. ### Lambda Layer in release notes @@ -38,7 +38,7 @@ This also means bringing `typing-extensions` as a runtime dependency to ensure c ### New utilities -With V2 launched, we want to resume working on new utilities, specifically but not limited to the most commonly asked: **(1)** [Sensitive Data Masking](https://github.com/aws-powertools/powertools-lambda-python/issues/1173){target="_blank" rel="nofollow"}, **(2)** [Integration/End-to-end Testing](https://github.com/aws-powertools/powertools-lambda-python/issues/1169){target="_blank" rel="nofollow"}, and **(3)** [Event Bridge](https://github.com/aws-powertools/powertools-lambda-python/issues/1168){target="_blank" rel="nofollow"}. +With V2 launched, we want to resume working on new utilities, specifically but not limited to the most commonly asked: **(1)** [Sensitive Data Masking](https://github.com/aws-powertools/powertools-lambda-python/issues/1173){target="_blank"}, **(2)** [Integration/End-to-end Testing](https://github.com/aws-powertools/powertools-lambda-python/issues/1169){target="_blank"}, and **(3)** [Event Bridge](https://github.com/aws-powertools/powertools-lambda-python/issues/1168){target="_blank"}. ### Open iteration planning @@ -54,7 +54,7 @@ graph LR Visual representation -Within our [public board](https://github.com/orgs/awslabs/projects/51/views/1?query=is%3Aopen+sort%3Aupdated-desc){target="_blank" rel="nofollow"}, you'll see the following values in the `Status` column: +Within our [public board](https://github.com/orgs/aws-powertools/projects/3/views/1?query=is%3Aopen+sort%3Aupdated-desc){target="_blank"}, you'll see the following values in the `Status` column: * **Ideas**. Incoming and existing feature requests that are not being actively considered yet. These will be reviewed when bandwidth permits. * **Backlog**. Accepted feature requests or enhancements that we want to work on. @@ -82,14 +82,14 @@ graph LR Our end-to-end mechanism follows four major steps: -* **Feature Request**. Ideas start with a [feature request](https://github.com/aws-powertools/powertools-lambda-python/issues/new?assignees=&labels=feature-request%2Ctriage&template=feature_request.yml&title=Feature+request%3A+TITLE){target="_blank" rel="nofollow"} to outline their use case at a high level. For complex use cases, maintainers might ask for/write a RFC. - * Maintainers review requests based on [project tenets](index.md#tenets){target="_blank" rel="nofollow"}, customers reaction (👍), and use cases. -* **Request-for-comments (RFC)**. Design proposals use our [RFC issue template](https://github.com/aws-powertools/powertools-lambda-python/issues/new?assignees=&labels=RFC%2Ctriage&template=rfc.yml&title=RFC%3A+TITLE){target="_blank" rel="nofollow"} to describe its implementation, challenges, developer experience, dependencies, and alternative solutions. +* **Feature Request**. Ideas start with a [feature request](https://github.com/aws-powertools/powertools-lambda-python/issues/new?assignees=&labels=feature-request%2Ctriage&template=feature_request.yml&title=Feature+request%3A+TITLE){target="_blank"} to outline their use case at a high level. For complex use cases, maintainers might ask for/write a RFC. + * Maintainers review requests based on [project tenets](index.md#tenets){target="_blank"}, customers reaction (👍), and use cases. +* **Request-for-comments (RFC)**. Design proposals use our [RFC issue template](https://github.com/aws-powertools/powertools-lambda-python/issues/new?assignees=&labels=RFC%2Ctriage&template=rfc.yml&title=RFC%3A+TITLE){target="_blank"} to describe its implementation, challenges, developer experience, dependencies, and alternative solutions. * This helps refine the initial idea with community feedback before a decision is made. * **Decision**. After carefully reviewing and discussing them, maintainers make a final decision on whether to start implementation, defer or reject it, and update everyone with the next steps. * **Implementation**. For approved features, maintainers give priority to the original authors for implementation unless it is a sensitive task that is best handled by maintainers. -???+ info "See [Maintainers](https://github.com/aws-powertools/powertools-lambda-python/blob/develop/MAINTAINERS.md){target="_blank" rel="nofollow"} document to understand how we triage issues and pull requests, labels and governance." +???+ info "See [Maintainers](https://github.com/aws-powertools/powertools-lambda-python/blob/develop/MAINTAINERS.md){target="_blank"} document to understand how we triage issues and pull requests, labels and governance." ## Disclaimer diff --git a/docs/security.md b/docs/security.md index 4572f921573..e8900366f3b 100644 --- a/docs/security.md +++ b/docs/security.md @@ -79,7 +79,7 @@ You can do this manually or automated via a shell script. We maintain the latter 5. **Runs SLSA Verifier against attestation**, GitHub Source, and release binary 6. **Cleanup** by removing downloaded files to keep your current directory tidy - ??? info "Expand or [click here](https://github.com/heitorlessa/aws-lambda-powertools-python/blob/refactor/ci-seal/.github/actions/verify-provenance/verify_provenance.sh#L95){target="_blank" rel="nofollow"} to see the script source code" + ??? info "Expand or [click here](https://github.com/aws-powertools/powertools-lambda-python/blob/develop/.github/actions/verify-provenance/verify_provenance.sh){target="_blank"} to see the script source code" ```bash title=".github/actions/verify-provenance/verify_provenance.sh" ---8<-- ".github/actions/verify-provenance/verify_provenance.sh" diff --git a/docs/tutorial/index.md b/docs/tutorial/index.md index 28021ea191a..ae4fd4a7181 100644 --- a/docs/tutorial/index.md +++ b/docs/tutorial/index.md @@ -9,8 +9,8 @@ This tutorial progressively introduces Powertools for AWS Lambda (Python) core u ## Requirements -* [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html){target="_blank" rel="nofollow"} and [configured with your credentials](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-getting-started-set-up-credentials.html){target="_blank" rel="nofollow"}. -* [AWS SAM CLI](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-cli-install.html){target="_blank" rel="nofollow"} installed. +* [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html){target="_blank"} and [configured with your credentials](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-getting-started-set-up-credentials.html){target="_blank"}. +* [AWS SAM CLI](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-cli-install.html){target="_blank"} installed. ## Getting started @@ -87,7 +87,7 @@ Our Lambda code consists of an entry point function named `lambda_handler`, and When API Gateway receives a HTTP GET request on `/hello` route, Lambda will call our `lambda_handler` function, subsequently calling the `hello` function. API Gateway will use this response to return the correct HTTP Status Code and payload back to the caller. ???+ warning - For simplicity, we do not set up authentication and authorization! You can find more information on how to implement it on [AWS SAM documentation](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-controlling-access-to-apis.html){target="_blank" rel="nofollow"}. + For simplicity, we do not set up authentication and authorization! You can find more information on how to implement it on [AWS SAM documentation](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-controlling-access-to-apis.html){target="_blank"}. ### Run your code @@ -111,7 +111,7 @@ As a result, a local API endpoint will be exposed and you can invoke it using yo ``` ???+ info - To learn more about local testing, please visit the [AWS SAM CLI local testing](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/sam-cli-command-reference-sam-local-start-api.html){target="_blank" rel="nofollow"} documentation. + To learn more about local testing, please visit the [AWS SAM CLI local testing](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/sam-cli-command-reference-sam-local-start-api.html){target="_blank"} documentation. #### Live test @@ -147,7 +147,7 @@ At the end of the deployment, you will find the API endpoint URL within `Outputs ``` ???+ Info - For more details on AWS SAM deployment mechanism, see [SAM Deploy reference docs](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/sam-cli-command-reference-sam-deploy.html){target="_blank" rel="nofollow"}. + For more details on AWS SAM deployment mechanism, see [SAM Deploy reference docs](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/sam-cli-command-reference-sam-deploy.html){target="_blank"}. ## Routing @@ -226,7 +226,7 @@ For this to work, we could create a new Lambda function to handle incoming reque We could group similar routes and intents, separate read and write operations resulting in fewer functions. It doesn't address the boilerplate routing code, but maybe it will be easier to add additional URLs. ???+ info "Info: You might be already asking yourself about mono vs micro-functions" - If you want a more detailed explanation of these two approaches, head over to the [trade-offs on each approach](../core/event_handler/api_gateway/#considerations){target="_blank" rel="nofollow"} later. + If you want a more detailed explanation of these two approaches, head over to the [trade-offs on each approach](../core/event_handler/api_gateway/#considerations){target="_blank"} later. A first attempt at the routing logic might look similar to the following code snippet. @@ -318,10 +318,10 @@ However, it forces us to understand the internal structure of the API Gateway re ### Simplifying with Event Handler -We can massively simplify cross-cutting concerns while keeping it lightweight by using [Event Handler](../core/event_handler/api_gateway.md){target="_blank" rel="nofollow"}. +We can massively simplify cross-cutting concerns while keeping it lightweight by using [Event Handler](../core/event_handler/api_gateway.md){target="_blank"}. ???+ tip - This is available for both [REST API (API Gateway, ALB)](../core/event_handler/api_gateway.md){target="_blank" rel="nofollow"} and [GraphQL API (AppSync)](../core/event_handler/appsync.md){target="_blank" rel="nofollow"}. + This is available for both [REST API (API Gateway, ALB)](../core/event_handler/api_gateway.md){target="_blank"} and [GraphQL API (AppSync)](../core/event_handler/appsync.md){target="_blank"}. Let's include Powertools for AWS Lambda (Python) as a dependency in `requirement.txt`, and use Event Handler to refactor our previous example. @@ -355,13 +355,13 @@ Let's include Powertools for AWS Lambda (Python) as a dependency in `requirement Use `sam build && sam local start-api` and try run it locally again. ???+ note - If you're coming from [Flask](https://flask.palletsprojects.com/en/2.0.x/){target="_blank" rel="nofollow"}, you will be familiar with this experience already. [Event Handler for API Gateway](../core/event_handler/api_gateway.md){target="_blank" rel="nofollow"} uses `APIGatewayRestResolver` to give a Flask-like experience while staying true to our tenet `Keep it lean`. + If you're coming from [Flask](https://flask.palletsprojects.com/en/2.0.x/){target="_blank" rel="nofollow"}, you will be familiar with this experience already. [Event Handler for API Gateway](../core/event_handler/api_gateway.md){target="_blank"} uses `APIGatewayRestResolver` to give a Flask-like experience while staying true to our tenet `Keep it lean`. We have added the route annotation as the decorator for our methods. It enables us to use the parameters passed in the request directly, and our responses are simply dictionaries. Lastly, we used `return app.resolve(event, context)` so Event Handler can resolve routes, inject the current request, handle serialization, route validation, etc. -From here, we could handle [404 routes](../core/event_handler/api_gateway.md#handling-not-found-routes){target="_blank" rel="nofollow"}, [error handling](../core/event_handler/api_gateway.md#exception-handling){target="_blank" rel="nofollow"}, [access query strings, payload](../core/event_handler/api_gateway.md#accessing-request-details){target="_blank" rel="nofollow"}, etc. +From here, we could handle [404 routes](../core/event_handler/api_gateway.md#handling-not-found-routes){target="_blank"}, [error handling](../core/event_handler/api_gateway.md#exception-handling){target="_blank"}, [access query strings, payload](../core/event_handler/api_gateway.md#accessing-request-details){target="_blank"}, etc. ???+ tip If you'd like to learn how python decorators work under the hood, you can follow [Real Python](https://realpython.com/primer-on-python-decorators/){target="_blank" rel="nofollow"}'s article. @@ -370,7 +370,7 @@ From here, we could handle [404 routes](../core/event_handler/api_gateway.md#han Over time, you realize that searching logs as text results in poor observability, it's hard to create metrics from, enumerate common exceptions, etc. -Then, you decided to propose production quality logging capabilities to your Lambda code. You found out that by having logs as `JSON` you can [structure them](https://docs.aws.amazon.com/lambda/latest/operatorguide/parse-logs.html){target="_blank" rel="nofollow"}, so that you can use any Log Analytics tool out there to quickly analyze them. +Then, you decided to propose production quality logging capabilities to your Lambda code. You found out that by having logs as `JSON` you can [structure them](https://docs.aws.amazon.com/lambda/latest/operatorguide/parse-logs.html){target="_blank"}, so that you can use any Log Analytics tool out there to quickly analyze them. This helps not only in searching, but produces consistent logs containing enough context and data to ask arbitrary questions on the status of your system. We can take advantage of CloudWatch Logs and Cloudwatch Insight for this purpose. @@ -454,7 +454,7 @@ We could start by creating a dictionary with Lambda context information or somet ???+ question "Surely this could be easier, right?" Yes! Powertools for AWS Lambda (Python) Logger to the rescue :-) -As we already have Powertools for AWS Lambda (Python) as a dependency, we can simply import [Logger](../core/logger.md){target="_blank" rel="nofollow"}. +As we already have Powertools for AWS Lambda (Python) as a dependency, we can simply import [Logger](../core/logger.md){target="_blank"}. ```python title="Refactoring with Powertools for AWS Lambda (Python) Logger" hl_lines="1 3 5 12 18 22" from aws_lambda_powertools import Logger @@ -487,8 +487,8 @@ Let's break this down: * **L5**: We add Powertools for AWS Lambda (Python) Logger; the boilerplate is now done for you. By default, we set `INFO` as the logging level if `LOG_LEVEL` env var isn't set. * **L22**: We use `logger.inject_lambda_context` decorator to inject key information from Lambda context into every log. -* **L22**: We also instruct Logger to use the incoming API Gateway Request ID as a [correlation id](../core/logger.md##set_correlation_id-method){target="_blank" rel="nofollow"} automatically. -* **L22**: Since we're in dev, we also use `log_event=True` to automatically log each incoming request for debugging. This can be also set via [environment variables](./index.md#environment-variables){target="_blank" rel="nofollow"}. +* **L22**: We also instruct Logger to use the incoming API Gateway Request ID as a [correlation id](../core/logger.md##set_correlation_id-method){target="_blank"} automatically. +* **L22**: Since we're in dev, we also use `log_event=True` to automatically log each incoming request for debugging. This can be also set via [environment variables](./index.md#environment-variables){target="_blank"}. This is how the logs would look like now: @@ -510,9 +510,9 @@ This is how the logs would look like now: We can now search our logs by the request ID to find a specific operation. Additionally, we can also search our logs for function name, Lambda request ID, Lambda function ARN, find out whether an operation was a cold start, etc. -From here, we could [set specific keys](../core/logger.md#append_keys-method){target="_blank" rel="nofollow"} to add additional contextual information about a given operation, [log exceptions](../core/logger.md#logging-exceptions){target="_blank" rel="nofollow"} to easily enumerate them later, [sample debug logs](../core/logger.md#sampling-debug-logs){target="_blank" rel="nofollow"}, etc. +From here, we could [set specific keys](../core/logger.md#append_keys-method){target="_blank"} to add additional contextual information about a given operation, [log exceptions](../core/logger.md#logging-exceptions){target="_blank"} to easily enumerate them later, [sample debug logs](../core/logger.md#sampling-debug-logs){target="_blank"}, etc. -By having structured logs like this, we can easily search and analyse them in [CloudWatch Logs Insight](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/AnalyzingLogData.html){target="_blank" rel="nofollow"}. +By having structured logs like this, we can easily search and analyse them in [CloudWatch Logs Insight](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/AnalyzingLogData.html){target="_blank"}. === "CloudWatch Logs Insight Example" ![CloudWatch Logs Insight Example](../media/cloudwatch_logs_insight_example.png) @@ -528,14 +528,14 @@ Combined with structured logs, it is an important step to be able to observe how ### Generating traces -[AWS X-Ray](https://aws.amazon.com/xray/){target="_blank" rel="nofollow"} is the distributed tracing service we're going to use. But how do we generate application traces in the first place? +[AWS X-Ray](https://aws.amazon.com/xray/){target="_blank"} is the distributed tracing service we're going to use. But how do we generate application traces in the first place? -It's a [two-step process](https://docs.aws.amazon.com/lambda/latest/dg/services-xray.html){target="_blank" rel="nofollow"}: +It's a [two-step process](https://docs.aws.amazon.com/lambda/latest/dg/services-xray.html){target="_blank"}: 1. Enable tracing in your Lambda function. 2. Instrument your application code. -Let's explore how we can instrument our code with [AWS X-Ray SDK](https://docs.aws.amazon.com/xray-sdk-for-python/latest/reference/index.html){target="_blank" rel="nofollow"}, and then simplify it with [Powertools for AWS Lambda (Python) Tracer](../core/tracer.md){target="_blank" rel="nofollow"} feature. +Let's explore how we can instrument our code with [AWS X-Ray SDK](https://docs.aws.amazon.com/xray-sdk-for-python/latest/reference/index.html){target="_blank"}, and then simplify it with [Powertools for AWS Lambda (Python) Tracer](../core/tracer.md){target="_blank"} feature. === "app.py" @@ -616,7 +616,7 @@ Let's explore how we can instrument our code with [AWS X-Ray SDK](https://docs.a Let's break it down: -* **L1**: First, we import AWS X-Ray SDK. `xray_recorder` records blocks of code being traced ([subsegment](https://docs.aws.amazon.com/xray/latest/devguide/xray-concepts.html#xray-concepts-subsegments){target="_blank" rel="nofollow"}). It also sends generated traces to the AWS X-Ray daemon running in the Lambda service who subsequently forwards them to AWS X-Ray service. +* **L1**: First, we import AWS X-Ray SDK. `xray_recorder` records blocks of code being traced ([subsegment](https://docs.aws.amazon.com/xray/latest/devguide/xray-concepts.html#xray-concepts-subsegments){target="_blank"}). It also sends generated traces to the AWS X-Ray daemon running in the Lambda service who subsequently forwards them to AWS X-Ray service. * **L13,20,27**: We decorate our function so the SDK traces the end-to-end execution, and the argument names the generated block being traced. ???+ question @@ -625,9 +625,9 @@ Let's break it down: We've made the following changes in `template.yaml` for this to work seamless: * **L7-8**: Enables tracing for Amazon API Gateway. -* **L16**: Enables tracing for our Serverless Function. This will also add a managed IAM Policy named [AWSXRayDaemonWriteAccess](https://console.aws.amazon.com/iam/home#/policies/arn:aws:iam::aws:policy/AWSXRayDaemonWriteAccess){target="_blank" rel="nofollow"} to allow Lambda to send traces to AWS X-Ray. +* **L16**: Enables tracing for our Serverless Function. This will also add a managed IAM Policy named [AWSXRayDaemonWriteAccess](https://console.aws.amazon.com/iam/home#/policies/arn:aws:iam::aws:policy/AWSXRayDaemonWriteAccess){target="_blank"} to allow Lambda to send traces to AWS X-Ray. -You can now build and deploy our updates with `sam build && sam deploy`. Once deployed, try invoking the application via the API endpoint, and visit [AWS X-Ray Console](https://console.aws.amazon.com/xray/home#/traces/){target="_blank" rel="nofollow"} to see how much progress we've made so far!! +You can now build and deploy our updates with `sam build && sam deploy`. Once deployed, try invoking the application via the API endpoint, and visit [AWS X-Ray Console](https://console.aws.amazon.com/xray/home#/traces/){target="_blank"} to see how much progress we've made so far!! ![AWS X-Ray Console trace view](../media/tracer_xray_sdk_showcase.png) @@ -644,7 +644,7 @@ What we've done helps bring an initial visibility, but we can do so much more. Within AWS X-Ray, we can answer these questions by using two features: tracing **Annotations** and **Metadata**. -**Annotations** are simple key-value pairs that are indexed for use with [filter expressions](https://docs.aws.amazon.com/xray/latest/devguide/xray-console-filters.html){target="_blank" rel="nofollow"}. +**Annotations** are simple key-value pairs that are indexed for use with [filter expressions](https://docs.aws.amazon.com/xray/latest/devguide/xray-console-filters.html){target="_blank"}. **Metadata** are key-value pairs with values of any type, including objects and lists, but that are not indexed. Let's put them into action. @@ -708,11 +708,11 @@ Let's break it down: * **L44**: We include the final response under `response` key as part of the `handler` subsegment. ???+ info - If you want to understand how the Lambda execution environment (sandbox) works and why cold starts can occur, see this [blog series on Lambda performance](https://aws.amazon.com/blogs/compute/operating-lambda-performance-optimization-part-1/){target="_blank" rel="nofollow"}. + If you want to understand how the Lambda execution environment (sandbox) works and why cold starts can occur, see this [blog series on Lambda performance](https://aws.amazon.com/blogs/compute/operating-lambda-performance-optimization-part-1/){target="_blank"}. Repeat the process of building, deploying, and invoking your application via the API endpoint. -Within the [AWS X-Ray Console](https://console.aws.amazon.com/xray/home#/traces/){target="_blank" rel="nofollow"}, you should now be able to group traces by the `User` and `ColdStart` annotation. +Within the [AWS X-Ray Console](https://console.aws.amazon.com/xray/home#/traces/){target="_blank"}, you should now be able to group traces by the `User` and `ColdStart` annotation. ![Filtering traces by annotations](../media/tracer_xray_sdk_enriched.png) @@ -724,7 +724,7 @@ If you choose any of the traces available, try opening the `handler` subsegment Cross-cutting concerns like filtering traces by Cold Start, including response as well as exceptions as tracing metadata can take a considerable amount of boilerplate. -We can simplify our previous patterns by using [Powertools for AWS Lambda (Python) Tracer](../core/tracer.md){target="_blank" rel="nofollow"}; a thin wrapper on top of X-Ray SDK. +We can simplify our previous patterns by using [Powertools for AWS Lambda (Python) Tracer](../core/tracer.md){target="_blank"}; a thin wrapper on top of X-Ray SDK. ???+ note You can now safely remove `aws-xray-sdk` from `requirements.txt`; keep `aws-lambda-powertools` only. @@ -773,21 +773,21 @@ Another subtle difference is that you can now run your Lambda functions and unit Powertools for AWS Lambda (Python) optimizes for Lambda compute environment. As such, we add these and other common approaches to accelerate your development, so you don't worry about implementing every cross-cutting concern. ???+ tip - You can [opt-out some of these behaviours](../core/tracer/#advanced){target="_blank" rel="nofollow"} like disabling response capturing, explicitly patching only X modules, etc. + You can [opt-out some of these behaviours](../core/tracer/#advanced){target="_blank"} like disabling response capturing, explicitly patching only X modules, etc. -Repeat the process of building, deploying, and invoking your application via the API endpoint. Within the [AWS X-Ray Console](https://console.aws.amazon.com/xray/home#/traces/){target="_blank" rel="nofollow"}, you should see a similar view: +Repeat the process of building, deploying, and invoking your application via the API endpoint. Within the [AWS X-Ray Console](https://console.aws.amazon.com/xray/home#/traces/){target="_blank"}, you should see a similar view: ![AWS X-Ray Console trace view using Powertools for AWS Lambda (Python) Tracer](../media/tracer_utility_showcase_2.png) ???+ tip - Consider using [Amazon CloudWatch ServiceLens view](https://console.aws.amazon.com/cloudwatch/home#servicelens:service-map/map){target="_blank" rel="nofollow"} as it aggregates AWS X-Ray traces and CloudWatch metrics and logs in one view. + Consider using [Amazon CloudWatch ServiceLens view](https://console.aws.amazon.com/cloudwatch/home#servicelens:service-map/map){target="_blank"} as it aggregates AWS X-Ray traces and CloudWatch metrics and logs in one view. From here, you can browse to specific logs in CloudWatch Logs Insight, Metrics Dashboard or AWS X-Ray traces. ![CloudWatch ServiceLens View](../media/tracer_utility_showcase_3.png) ???+ info - For more information on Amazon CloudWatch ServiceLens, please visit [link](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/ServiceLens.html){target="_blank" rel="nofollow"}. + For more information on Amazon CloudWatch ServiceLens, please visit [link](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/ServiceLens.html){target="_blank"}. ## Custom Metrics @@ -795,10 +795,10 @@ From here, you can browse to specific logs in CloudWatch Logs Insight, Metrics D Let's add custom metrics to better understand our application and business behavior (e.g. number of reservations, etc.). -By default, AWS Lambda adds [invocation and performance metrics](https://docs.aws.amazon.com/lambda/latest/dg/monitoring-metrics.html#monitoring-metrics-types){target="_blank" rel="nofollow"}, and Amazon API Gateway adds [latency and some HTTP metrics](https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-metrics-and-dimensions.html#api-gateway-metrics){target="_blank" rel="nofollow"}. +By default, AWS Lambda adds [invocation and performance metrics](https://docs.aws.amazon.com/lambda/latest/dg/monitoring-metrics.html#monitoring-metrics-types){target="_blank"}, and Amazon API Gateway adds [latency and some HTTP metrics](https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-metrics-and-dimensions.html#api-gateway-metrics){target="_blank"}. ???+ tip - You can [optionally enable detailed metrics](https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-metrics-and-dimensions.html#api-gateway-metricdimensions){target="_blank" rel="nofollow"} per each API route, stage, and method in API Gateway. + You can [optionally enable detailed metrics](https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-metrics-and-dimensions.html#api-gateway-metricdimensions){target="_blank"} per each API route, stage, and method in API Gateway. Let's expand our application with custom metrics using AWS SDK to see how it works, then let's upgrade it with Powertools for AWS Lambda (Python) :-) @@ -917,16 +917,16 @@ Let's expand our application with custom metrics using AWS SDK to see how it wor There's a lot going on, let's break this down: -* **L10**: We define a container where all of our application metrics will live `MyApp`, a.k.a [Metrics Namespace](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/cloudwatch_concepts.html){target="_blank" rel="nofollow"}. +* **L10**: We define a container where all of our application metrics will live `MyApp`, a.k.a [Metrics Namespace](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/cloudwatch_concepts.html){target="_blank"}. * **L14**: We initialize a CloudWatch client to send metrics later. -* **L19-47**: We create a custom function to prepare and send `ColdStart` and `SuccessfulGreetings` metrics using CloudWatch expected data structure. We also set [dimensions](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/cloudwatch_concepts.html#Dimension){target="_blank" rel="nofollow"} of these metrics. +* **L19-47**: We create a custom function to prepare and send `ColdStart` and `SuccessfulGreetings` metrics using CloudWatch expected data structure. We also set [dimensions](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/cloudwatch_concepts.html#Dimension){target="_blank"} of these metrics. * Think of them as metadata to define to slice and dice them later; an unique metric is a combination of metric name + metric dimension(s). * **L55,64**: We call our custom function to create metrics for every greeting received. ???+ question But what permissions do I need to send metrics to CloudWatch? -Within `template.yaml`, we add [CloudWatchPutMetricPolicy](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-policy-template-list.html#cloudwatch-put-metric-policy){target="_blank" rel="nofollow"} policy in SAM. +Within `template.yaml`, we add [CloudWatchPutMetricPolicy](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-policy-template-list.html#cloudwatch-put-metric-policy){target="_blank"} policy in SAM. ???+ note "Adding metrics via AWS SDK gives a lot of flexibility at a cost" `put_metric_data` is a synchronous call to CloudWatch Metrics API. This means establishing a connection to CloudWatch endpoint, sending metrics payload, and waiting from a response. @@ -935,12 +935,12 @@ Within `template.yaml`, we add [CloudWatchPutMetricPolicy](https://docs.aws.amaz ### Simplifying with Metrics -[Powertools for AWS Lambda (Python) Metrics](../core/metrics.md){target="_blank} uses [Amazon CloudWatch Embedded Metric Format (EMF)](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch_Embedded_Metric_Format.html){target="_blank" rel="nofollow"} to create custom metrics **asynchronously** via a native integration with Lambda. +[Powertools for AWS Lambda (Python) Metrics](../core/metrics.md){target="_blank} uses [Amazon CloudWatch Embedded Metric Format (EMF)](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch_Embedded_Metric_Format.html){target="_blank"} to create custom metrics **asynchronously** via a native integration with Lambda. In general terms, EMF is a specification that expects metrics in a JSON payload within CloudWatch Logs. Lambda ingests all logs emitted by a given function into CloudWatch Logs. CloudWatch automatically looks up for log entries that follow the EMF format and transforms them into a CloudWatch metric. ???+ info - If you are interested in the details of the EMF mechanism, follow [blog post](https://aws.amazon.com/blogs/mt/enhancing-workload-observability-using-amazon-cloudwatch-embedded-metric-format/){target="_blank" rel="nofollow"}. + If you are interested in the details of the EMF mechanism, follow [blog post](https://aws.amazon.com/blogs/mt/enhancing-workload-observability-using-amazon-cloudwatch-embedded-metric-format/){target="_blank"}. Let's implement that using [Metrics](../core/metrics.md){target="_blank}: @@ -991,7 +991,7 @@ That's a lot less boilerplate code! Let's break this down: * **L9**: We initialize `Metrics` with our service name (`APP`) and metrics namespace (`MyApp`), reducing the need to add the `service` dimension for every metric and setting the namespace later * **L18, 27**: We use `add_metric` similarly to our custom function, except we now have an enum `MetricCount` to help us understand which Metric Units we have at our disposal * **L33**: We use `@metrics.log_metrics` decorator to ensure that our metrics are aligned with the EMF output and validated before-hand, like in case we forget to set namespace, or accidentally use a metric unit as a string that doesn't exist in CloudWatch. -* **L33**: We also use `capture_cold_start_metric=True` so we don't have to handle that logic either. Note that [Metrics](../core/metrics.md){target="_blank" rel="nofollow"} does not publish a warm invocation metric (ColdStart=0) for cost reasons. As such, treat the absence (sparse metric) as a non-cold start invocation. +* **L33**: We also use `capture_cold_start_metric=True` so we don't have to handle that logic either. Note that [Metrics](../core/metrics.md){target="_blank"} does not publish a warm invocation metric (ColdStart=0) for cost reasons. As such, treat the absence (sparse metric) as a non-cold start invocation. Repeat the process of building, deploying, and invoking your application via the API endpoint a few times to generate metrics - [Artillery](https://www.artillery.io/){target="_blank" rel="nofollow"} and [K6.io](https://k6.io/open-source){target="_blank" rel="nofollow"} are quick ways to generate some load. @@ -999,7 +999,7 @@ Within [CloudWatch Metrics view](https://console.aws.amazon.com/cloudwatch/home# ![Custom Metrics Example](../media/metrics_utility_showcase.png) -If you're curious about how the EMF portion of your function logs look like, you can quickly go to [CloudWatch ServiceLens view](https://console.aws.amazon.com/cloudwatch/home#servicelens:service-map/map){target="_blank" rel="nofollow"}, choose your function and open logs. You will see a similar entry that looks like this: +If you're curious about how the EMF portion of your function logs look like, you can quickly go to [CloudWatch ServiceLens view](https://console.aws.amazon.com/cloudwatch/home#servicelens:service-map/map){target="_blank"}, choose your function and open logs. You will see a similar entry that looks like this: ```json { @@ -1035,14 +1035,14 @@ If you're curious about how the EMF portion of your function logs look like, you We covered a lot of ground here and we only scratched the surface of the feature set available within Powertools for AWS Lambda (Python). -When it comes to the observability features ([Tracer](../core/tracer.md){target="_blank" rel="nofollow"}, [Metrics](../core/metrics.md){target="_blank" rel="nofollow"}, [Logging](../core/logger.md){target="_blank" rel="nofollow"}), don't stop there! The goal here is to ensure you can ask arbitrary questions to assess your system's health; these features are only part of the wider story! +When it comes to the observability features ([Tracer](../core/tracer.md){target="_blank"}, [Metrics](../core/metrics.md){target="_blank"}, [Logging](../core/logger.md){target="_blank"}), don't stop there! The goal here is to ensure you can ask arbitrary questions to assess your system's health; these features are only part of the wider story! This requires a change in mindset to ensure operational excellence is part of the software development lifecycle. ???+ tip - You can find more details on other leading practices described in the [Well-Architected Serverless Lens](https://aws.amazon.com/blogs/aws/new-serverless-lens-in-aws-well-architected-tool/){target="_blank" rel="nofollow"}. + You can find more details on other leading practices described in the [Well-Architected Serverless Lens](https://aws.amazon.com/blogs/aws/new-serverless-lens-in-aws-well-architected-tool/){target="_blank"}. Powertools for AWS Lambda (Python) is largely designed to make some of these practices easier to adopt from day 1. ???+ question "Have ideas for other tutorials?" - You can open up a [documentation issue](https://github.com/aws-powertools/powertools-lambda-python/issues/new?assignees=&labels=documentation&template=documentation-improvements.md&title=Tutorial%20Suggestion){target="_blank" rel="nofollow"}, or via e-mail [aws-lambda-powertools-feedback@amazon.com](mailto:aws-lambda-powertools-feedback@amazon.com). + You can open up a [documentation issue](https://github.com/aws-powertools/powertools-lambda-python/issues/new?assignees=&labels=documentation&template=documentation-improvements.md&title=Tutorial%20Suggestion){target="_blank"}, or via e-mail [aws-lambda-powertools-feedback@amazon.com](mailto:aws-lambda-powertools-feedback@amazon.com). diff --git a/docs/upgrade.md b/docs/upgrade.md index 509bffda15c..064c6a9657a 100644 --- a/docs/upgrade.md +++ b/docs/upgrade.md @@ -9,7 +9,7 @@ description: Guide to update between major Powertools for AWS Lambda (Python) ve !!! warning "On March 31st, 2023, Powertools for AWS Lambda (Python) v1 reached end of support and will no longer receive updates or releases. If you are still using v1, we strongly recommend you to read our upgrade guide and update to the latest version." -Given our commitment to all of our customers using Powertools for AWS Lambda (Python), we will keep [Pypi](https://pypi.org/project/aws-lambda-powertools/){target="_blank" rel="nofollow"} v1 releases and documentation 1.x versions to prevent any disruption. +Given our commitment to all of our customers using Powertools for AWS Lambda (Python), we will keep [Pypi](https://pypi.org/project/aws-lambda-powertools/){target="_blank"} v1 releases and documentation 1.x versions to prevent any disruption. ## Migrate to v2 from v1 @@ -20,20 +20,20 @@ We've made minimal breaking changes to make your transition to v2 as smooth as p | Area | Change | Code change required | IAM Permissions change required | | ---------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------- | ------------------------------- | | **Batch** | Removed legacy [SQS batch processor](#legacy-sqs-batch-processor) in favour of **`BatchProcessor`**. | Yes | - | -| **Environment variables** | Removed legacy **`POWERTOOLS_EVENT_HANDLER_DEBUG`** in favour of [`POWERTOOLS_DEV`](index.md#optimizing-for-non-production-environments){target="_blank" rel="nofollow"}. | - | - | -| **Event Handler** | Updated [headers response format](#event-handler-headers-response-format) due to [multi-value headers and cookie support](./core/event_handler/api_gateway.md#fine-grained-responses){target="_blank" rel="nofollow"}. | Tests only | - | +| **Environment variables** | Removed legacy **`POWERTOOLS_EVENT_HANDLER_DEBUG`** in favour of [`POWERTOOLS_DEV`](index.md#optimizing-for-non-production-environments){target="_blank"}. | - | - | +| **Event Handler** | Updated [headers response format](#event-handler-headers-response-format) due to [multi-value headers and cookie support](./core/event_handler/api_gateway.md#fine-grained-responses){target="_blank"}. | Tests only | - | | **Event Source Data Classes** | Replaced [DynamoDBStreamEvent](#dynamodbstreamevent-in-event-source-data-classes) `AttributeValue` with native Python types. | Yes | - | | **Feature Flags** / **Parameters** | Updated [AppConfig API calls](#feature-flags-and-appconfig-parameter-utility) due to **`GetConfiguration`** API deprecation. | - | Yes | | **Idempotency** | Updated [partition key](#idempotency-partition-key-format) to include fully qualified function/method names. | - | - | ### First Steps -!!! note "All dependencies are optional now. [Tracer](core/tracer.md#install){target="_blank" rel="nofollow"}, [Validation](./utilities/validation.md#install){target="_blank" rel="nofollow"}, and [Parser](./utilities/parser.md){target="_blank" rel="nofollow"} now require additional dependencies." +!!! note "All dependencies are optional now. [Tracer](core/tracer.md#install){target="_blank"}, [Validation](./utilities/validation.md#install){target="_blank"}, and [Parser](./utilities/parser.md){target="_blank"} now require additional dependencies." Before you start, we suggest making a copy of your current working project or create a new branch with git. 1. **Upgrade** Python to at least v3.7 -2. **Ensure** you have the latest version via [Lambda Layer or PyPi](index.md#install){target="_blank" rel="nofollow"}. +2. **Ensure** you have the latest version via [Lambda Layer or PyPi](index.md#install){target="_blank"}. 3. **Review** the following sections to confirm whether they affect your code ## Legacy SQS Batch Processor @@ -44,7 +44,7 @@ You can migrate to `BatchProcessor` with the following changes: 1. If you use **`sqs_batch_decorator`**, change to **`batch_processor`** decorator 2. If you use **`PartialSQSProcessor`**, change to **`BatchProcessor`** -3. [Enable **`ReportBatchItemFailures`** in your Lambda Event Source](../utilities/batch#required-resources){target="_blank" rel="nofollow"} +3. [Enable **`ReportBatchItemFailures`** in your Lambda Event Source](../utilities/batch#required-resources){target="_blank"} 4. Change your Lambda Handler to return the new response format === "[Before] Decorator" @@ -139,7 +139,7 @@ Previously, custom headers were available under `headers` key in the Event Handl } ``` -In V2, we add all headers under `multiValueHeaders` key. This enables seamless support for multi-value headers and cookies in [fine grained responses](./core/event_handler/api_gateway.md#fine-grained-responses){target="_blank" rel="nofollow"}. +In V2, we add all headers under `multiValueHeaders` key. This enables seamless support for multi-value headers and cookies in [fine grained responses](./core/event_handler/api_gateway.md#fine-grained-responses){target="_blank"}. ```python title="V2 response headers" hl_lines="2" { @@ -151,7 +151,7 @@ In V2, we add all headers under `multiValueHeaders` key. This enables seamless s ## DynamoDBStreamEvent in Event Source Data Classes -!!! info "This also applies if you're using [**DynamoDB BatchProcessor**](https://docs.powertools.aws.dev/lambda/python/latest/utilities/batch/#processing-messages-from-dynamodb){target="_blank" rel="nofollow"}." +!!! info "This also applies if you're using [**DynamoDB BatchProcessor**](https://docs.powertools.aws.dev/lambda/python/latest/utilities/batch/#processing-messages-from-dynamodb){target="_blank"}." You will now receive native Python types when accessing DynamoDB records via `keys`, `new_image`, and `old_image` attributes in `DynamoDBStreamEvent`. @@ -196,7 +196,7 @@ def lambda_handler(event: DynamoDBStreamEvent, context): !!! note "No code changes required" -We replaced `GetConfiguration` API ([now deprecated](https://github.com/aws-powertools/powertools-lambda-python/issues/1506#issuecomment-1266645884){target="_blank" rel="nofollow"}) with `GetLatestConfiguration` and `StartConfigurationSession`. +We replaced `GetConfiguration` API ([now deprecated](https://github.com/aws-powertools/powertools-lambda-python/issues/1506#issuecomment-1266645884){target="_blank"}) with `GetLatestConfiguration` and `StartConfigurationSession`. As such, you must update your IAM Role permissions to allow the following IAM actions: @@ -217,7 +217,7 @@ Previously, we used the function/method name to generate the partition key value In V2, we now distinguish between distinct classes or modules that may have the same function/method name. -[For example](https://github.com/aws-powertools/powertools-lambda-python/issues/1330){target="_blank" rel="nofollow"}, an ABC or Protocol class may have multiple implementations of `process_payment` method and may have different results. +[For example](https://github.com/aws-powertools/powertools-lambda-python/issues/1330){target="_blank"}, an ABC or Protocol class may have multiple implementations of `process_payment` method and may have different results. diff --git a/docs/utilities/batch.md b/docs/utilities/batch.md index 57e0a6020b1..0ef0a1366cb 100644 --- a/docs/utilities/batch.md +++ b/docs/utilities/batch.md @@ -59,9 +59,9 @@ This behavior changes when you enable Report Batch Item Failures feature in your ???+ warning "Warning: This utility lowers the chance of processing records more than once; it does not guarantee it" - We recommend implementing processing logic in an [idempotent manner](idempotency.md){target="_blank" rel="nofollow"} wherever possible. + We recommend implementing processing logic in an [idempotent manner](idempotency.md){target="_blank"} wherever possible. - You can find more details on how Lambda works with either [SQS](https://docs.aws.amazon.com/lambda/latest/dg/with-sqs.html){target="_blank" rel="nofollow"}, [Kinesis](https://docs.aws.amazon.com/lambda/latest/dg/with-kinesis.html){target="_blank" rel="nofollow"}, or [DynamoDB](https://docs.aws.amazon.com/lambda/latest/dg/with-ddb.html){target="_blank" rel="nofollow"} in the AWS Documentation. + You can find more details on how Lambda works with either [SQS](https://docs.aws.amazon.com/lambda/latest/dg/with-sqs.html){target="_blank"}, [Kinesis](https://docs.aws.amazon.com/lambda/latest/dg/with-kinesis.html){target="_blank"}, or [DynamoDB](https://docs.aws.amazon.com/lambda/latest/dg/with-ddb.html){target="_blank"} in the AWS Documentation. ## Getting started @@ -98,7 +98,7 @@ The remaining sections of the documentation will rely on these samples. For comp Processing batches from SQS works in three stages: 1. Instantiate **`BatchProcessor`** and choose **`EventType.SQS`** for the event type -2. Define your function to handle each batch record, and use [`SQSRecord`](data_classes.md#sqs){target="_blank" rel="nofollow"} type annotation for autocompletion +2. Define your function to handle each batch record, and use [`SQSRecord`](data_classes.md#sqs){target="_blank"} type annotation for autocompletion 3. Use **`process_partial_response`** to kick off processing !!! info "This code example uses Tracer and Logger for completion." @@ -169,7 +169,7 @@ This helps preserve the ordering of messages in your queue. Processing batches from Kinesis works in three stages: 1. Instantiate **`BatchProcessor`** and choose **`EventType.KinesisDataStreams`** for the event type -2. Define your function to handle each batch record, and use [`KinesisStreamRecord`](data_classes.md#kinesis-streams){target="_blank" rel="nofollow"} type annotation for autocompletion +2. Define your function to handle each batch record, and use [`KinesisStreamRecord`](data_classes.md#kinesis-streams){target="_blank"} type annotation for autocompletion 3. Use **`process_partial_response`** to kick off processing !!! info "This code example uses Tracer and Logger for completion." @@ -213,7 +213,7 @@ Processing batches from Kinesis works in three stages: Processing batches from DynamoDB Streams works in three stages: 1. Instantiate **`BatchProcessor`** and choose **`EventType.DynamoDBStreams`** for the event type -2. Define your function to handle each batch record, and use [`DynamoDBRecord`](data_classes.md#dynamodb-streams){target="_blank" rel="nofollow"} type annotation for autocompletion +2. Define your function to handle each batch record, and use [`DynamoDBRecord`](data_classes.md#dynamodb-streams){target="_blank"} type annotation for autocompletion 3. Use **`process_partial_response`** to kick off processing !!! info "This code example uses Tracer and Logger for completion." @@ -578,7 +578,7 @@ You can then use this class as a context manager, or pass it to `batch_processor When using Tracer to capture responses for each batch record processing, you might exceed 64K of tracing data depending on what you return from your `record_handler` function, or how big is your batch size. -If that's the case, you can configure [Tracer to disable response auto-capturing](../core/tracer.md#disabling-response-auto-capture){target="_blank" rel="nofollow"}. +If that's the case, you can configure [Tracer to disable response auto-capturing](../core/tracer.md#disabling-response-auto-capture){target="_blank"}. ```python hl_lines="17" title="Disabling Tracer response auto-capturing" --8<-- "examples/batch_processing/src/disable_tracing.py" @@ -626,7 +626,7 @@ As 2.12.0, `process_partial_response` and `async_process_partial_response` are t When using Sentry.io for error monitoring, you can override `failure_handler` to capture each processing exception with Sentry SDK: -> Credits to [Charles-Axel Dein](https://github.com/aws-powertools/powertools-lambda-python/issues/293#issuecomment-781961732){target="_blank" rel="nofollow"} +> Credits to [Charles-Axel Dein](https://github.com/aws-powertools/powertools-lambda-python/issues/293#issuecomment-781961732){target="_blank"} ```python hl_lines="1 7-8" title="Integrating error tracking with Sentry.io" --8<-- "examples/batch_processing/src/sentry_error_tracking.py" diff --git a/docs/utilities/data_classes.md b/docs/utilities/data_classes.md index ca41dbe01ea..7b3aa74e275 100644 --- a/docs/utilities/data_classes.md +++ b/docs/utilities/data_classes.md @@ -111,8 +111,8 @@ Log Data Event for Troubleshooting ### Active MQ -It is used for [Active MQ payloads](https://docs.aws.amazon.com/lambda/latest/dg/with-mq.html){target="_blank" rel="nofollow"}, also see -the [AWS blog post](https://aws.amazon.com/blogs/compute/using-amazon-mq-as-an-event-source-for-aws-lambda/){target="_blank" rel="nofollow"} +It is used for [Active MQ payloads](https://docs.aws.amazon.com/lambda/latest/dg/with-mq.html){target="_blank"}, also see +the [AWS blog post](https://aws.amazon.com/blogs/compute/using-amazon-mq-as-an-event-source-for-aws-lambda/){target="_blank"} for more details. === "app.py" @@ -138,7 +138,7 @@ for more details. > New in 1.20.0 -It is used for [API Gateway Rest API Lambda Authorizer payload](https://docs.aws.amazon.com/apigateway/latest/developerguide/apigateway-use-lambda-authorizer.html){target="_blank" rel="nofollow"}. +It is used for [API Gateway Rest API Lambda Authorizer payload](https://docs.aws.amazon.com/apigateway/latest/developerguide/apigateway-use-lambda-authorizer.html){target="_blank"}. Use **`APIGatewayAuthorizerRequestEvent`** for type `REQUEST` and **`APIGatewayAuthorizerTokenEvent`** for type `TOKEN`. @@ -234,8 +234,8 @@ Use **`APIGatewayAuthorizerRequestEvent`** for type `REQUEST` and **`APIGatewayA > New in 1.20.0 -It is used for [API Gateway HTTP API Lambda Authorizer payload version 2](https://docs.aws.amazon.com/apigateway/latest/developerguide/http-api-lambda-authorizer.html){target="_blank" rel="nofollow"}. -See also [this blog post](https://aws.amazon.com/blogs/compute/introducing-iam-and-lambda-authorizers-for-amazon-api-gateway-http-apis/){target="_blank" rel="nofollow"} for more details. +It is used for [API Gateway HTTP API Lambda Authorizer payload version 2](https://docs.aws.amazon.com/apigateway/latest/developerguide/http-api-lambda-authorizer.html){target="_blank"}. +See also [this blog post](https://aws.amazon.com/blogs/compute/introducing-iam-and-lambda-authorizers-for-amazon-api-gateway-http-apis/){target="_blank"} for more details. === "app.py" @@ -320,9 +320,9 @@ Is it used for Application load balancer event. > New in 1.20.0 -Used when building an [AWS_LAMBDA Authorization](https://docs.aws.amazon.com/appsync/latest/devguide/security-authz.html#aws-lambda-authorization){target="_blank" rel="nofollow"} with AppSync. -See blog post [Introducing Lambda authorization for AWS AppSync GraphQL APIs](https://aws.amazon.com/blogs/mobile/appsync-lambda-auth/){target="_blank" rel="nofollow"} -or read the Amplify documentation on using [AWS Lambda for authorization](https://docs.amplify.aws/lib/graphqlapi/authz/q/platform/js#aws-lambda){target="_blank" rel="nofollow"} with AppSync. +Used when building an [AWS_LAMBDA Authorization](https://docs.aws.amazon.com/appsync/latest/devguide/security-authz.html#aws-lambda-authorization){target="_blank"} with AppSync. +See blog post [Introducing Lambda authorization for AWS AppSync GraphQL APIs](https://aws.amazon.com/blogs/mobile/appsync-lambda-auth/){target="_blank"} +or read the Amplify documentation on using [AWS Lambda for authorization](https://docs.amplify.aws/lib/graphqlapi/authz/q/platform/js#aws-lambda){target="_blank"} with AppSync. In this example extract the `requestId` as the `correlation_id` for logging, used `@event_source` decorator and builds the AppSync authorizer using the `AppSyncAuthorizerResponse` helper. @@ -368,8 +368,8 @@ In this example extract the `requestId` as the `correlation_id` for logging, use > New in 1.12.0 -Used when building Lambda GraphQL Resolvers with [Amplify GraphQL Transform Library](https://docs.amplify.aws/cli/graphql-transformer/function){target="_blank" rel="nofollow"} (`@function`), -and [AppSync Direct Lambda Resolvers](https://aws.amazon.com/blogs/mobile/appsync-direct-lambda/){target="_blank" rel="nofollow"}. +Used when building Lambda GraphQL Resolvers with [Amplify GraphQL Transform Library](https://docs.amplify.aws/cli/graphql-transformer/function){target="_blank"} (`@function`), +and [AppSync Direct Lambda Resolvers](https://aws.amazon.com/blogs/mobile/appsync-direct-lambda/){target="_blank"}. In this example, we also use the new Logger `correlation_id` and built-in `correlation_paths` to extract, if available, X-Ray Trace ID in AppSync request headers: @@ -538,7 +538,7 @@ decompress and parse json data from the event. #### Kinesis integration -[When streaming CloudWatch Logs to a Kinesis Data Stream](https://aws.amazon.com/premiumsupport/knowledge-center/streaming-cloudwatch-logs/){target="_blank" rel="nofollow"} (cross-account or not), you can use `extract_cloudwatch_logs_from_event` to decode, decompress and extract logs as `CloudWatchLogsDecodedData` to ease log processing. +[When streaming CloudWatch Logs to a Kinesis Data Stream](https://aws.amazon.com/premiumsupport/knowledge-center/streaming-cloudwatch-logs/){target="_blank"} (cross-account or not), you can use `extract_cloudwatch_logs_from_event` to decode, decompress and extract logs as `CloudWatchLogsDecodedData` to ease log processing. === "app.py" @@ -560,7 +560,7 @@ decompress and parse json data from the event. return "nothing to be processed" ``` -Alternatively, you can use `extract_cloudwatch_logs_from_record` to seamless integrate with the [Batch utility](./batch.md){target="_blank" rel="nofollow"} for more robust log processing. +Alternatively, you can use `extract_cloudwatch_logs_from_record` to seamless integrate with the [Batch utility](./batch.md){target="_blank"} for more robust log processing. === "app.py" @@ -636,7 +636,7 @@ Data classes and utility functions to help create continuous delivery pipelines ### Cognito User Pool -Cognito User Pools have several [different Lambda trigger sources](https://docs.aws.amazon.com/cognito/latest/developerguide/cognito-user-identity-pools-working-with-aws-lambda-triggers.html#cognito-user-identity-pools-working-with-aws-lambda-trigger-sources){target="_blank" rel="nofollow"}, all of which map to a different data class, which +Cognito User Pools have several [different Lambda trigger sources](https://docs.aws.amazon.com/cognito/latest/developerguide/cognito-user-identity-pools-working-with-aws-lambda-triggers.html#cognito-user-identity-pools-working-with-aws-lambda-trigger-sources){target="_blank"}, all of which map to a different data class, which can be imported from `aws_lambda_powertools.data_classes.cognito_user_pool_event`: | Trigger/Event Source | Data Class | @@ -671,7 +671,7 @@ can be imported from `aws_lambda_powertools.data_classes.cognito_user_pool_event ???+ note In this example we are modifying the wrapped dict response fields, so we need to return the json serializable wrapped event in `event.raw_event`. -This example is based on the AWS Cognito docs for [Define Auth Challenge Lambda Trigger](https://docs.aws.amazon.com/cognito/latest/developerguide/user-pool-lambda-define-auth-challenge.html){target="_blank" rel="nofollow"}. +This example is based on the AWS Cognito docs for [Define Auth Challenge Lambda Trigger](https://docs.aws.amazon.com/cognito/latest/developerguide/user-pool-lambda-define-auth-challenge.html){target="_blank"}. === "app.py" @@ -825,7 +825,7 @@ This example is based on the AWS Cognito docs for [Define Auth Challenge Lambda #### Create Auth Challenge Example -This example is based on the AWS Cognito docs for [Create Auth Challenge Lambda Trigger](https://docs.aws.amazon.com/cognito/latest/developerguide/user-pool-lambda-create-auth-challenge.html){target="_blank" rel="nofollow"}. +This example is based on the AWS Cognito docs for [Create Auth Challenge Lambda Trigger](https://docs.aws.amazon.com/cognito/latest/developerguide/user-pool-lambda-create-auth-challenge.html){target="_blank"}. === "app.py" @@ -844,7 +844,7 @@ This example is based on the AWS Cognito docs for [Create Auth Challenge Lambda #### Verify Auth Challenge Response Example -This example is based on the AWS Cognito docs for [Verify Auth Challenge Response Lambda Trigger](https://docs.aws.amazon.com/cognito/latest/developerguide/user-pool-lambda-verify-auth-challenge-response.html){target="_blank" rel="nofollow"}. +This example is based on the AWS Cognito docs for [Verify Auth Challenge Response Lambda Trigger](https://docs.aws.amazon.com/cognito/latest/developerguide/user-pool-lambda-verify-auth-challenge-response.html){target="_blank"}. === "app.py" @@ -936,7 +936,7 @@ The class automatically deserializes DynamoDB types into their equivalent Python ### Kafka -This example is based on the AWS docs for [Amazon MSK](https://docs.aws.amazon.com/lambda/latest/dg/with-msk.html){target="_blank" rel="nofollow"} and [self-managed Apache Kafka](https://docs.aws.amazon.com/lambda/latest/dg/with-kafka.html){target="_blank" rel="nofollow"}. +This example is based on the AWS docs for [Amazon MSK](https://docs.aws.amazon.com/lambda/latest/dg/with-msk.html){target="_blank"} and [self-managed Apache Kafka](https://docs.aws.amazon.com/lambda/latest/dg/with-kafka.html){target="_blank"}. === "app.py" @@ -1001,8 +1001,8 @@ function to access the data either as json or plain text, depending on the origi ### Rabbit MQ -It is used for [Rabbit MQ payloads](https://docs.aws.amazon.com/lambda/latest/dg/with-mq.html){target="_blank" rel="nofollow"}, also see -the [blog post](https://aws.amazon.com/blogs/compute/using-amazon-mq-for-rabbitmq-as-an-event-source-for-lambda/){target="_blank" rel="nofollow"} +It is used for [Rabbit MQ payloads](https://docs.aws.amazon.com/lambda/latest/dg/with-mq.html){target="_blank"}, also see +the [blog post](https://aws.amazon.com/blogs/compute/using-amazon-mq-for-rabbitmq-as-an-event-source-for-lambda/){target="_blank"} for more details. === "app.py" @@ -1047,7 +1047,7 @@ for more details. ### S3 Object Lambda -This example is based on the AWS Blog post [Introducing Amazon S3 Object Lambda – Use Your Code to Process Data as It Is Being Retrieved from S3](https://aws.amazon.com/blogs/aws/introducing-amazon-s3-object-lambda-use-your-code-to-process-data-as-it-is-being-retrieved-from-s3/){target="_blank" rel="nofollow"}. +This example is based on the AWS Blog post [Introducing Amazon S3 Object Lambda – Use Your Code to Process Data as It Is Being Retrieved from S3](https://aws.amazon.com/blogs/aws/introducing-amazon-s3-object-lambda-use-your-code-to-process-data-as-it-is-being-retrieved-from-s3/){target="_blank"}. === "app.py" @@ -1147,7 +1147,7 @@ This example is based on the AWS Blog post [Introducing Amazon S3 Object Lambda You can register your Lambda functions as targets within an Amazon VPC Lattice service network. By doing this, your Lambda function becomes a service within the network, and clients that have access to the VPC Lattice service network can call your service. -[Click here](https://docs.aws.amazon.com/lambda/latest/dg/services-vpc-lattice.html){target="_blank" rel="nofollow"} for more information about using AWS Lambda with Amazon VPC Lattice. +[Click here](https://docs.aws.amazon.com/lambda/latest/dg/services-vpc-lattice.html){target="_blank"} for more information about using AWS Lambda with Amazon VPC Lattice. === "app.py" diff --git a/docs/utilities/feature_flags.md b/docs/utilities/feature_flags.md index 1b00cb46835..b68fcc594fb 100644 --- a/docs/utilities/feature_flags.md +++ b/docs/utilities/feature_flags.md @@ -6,7 +6,7 @@ description: Utility The feature flags utility provides a simple rule engine to define when one or multiple features should be enabled depending on the input. ???+ info - When using `AppConfigStore`, we currently only support AppConfig using [freeform configuration profile](https://docs.aws.amazon.com/appconfig/latest/userguide/appconfig-creating-configuration-and-profile.html#appconfig-creating-configuration-and-profile-free-form-configurations){target="_blank" rel="nofollow"} . + When using `AppConfigStore`, we currently only support AppConfig using [freeform configuration profile](https://docs.aws.amazon.com/appconfig/latest/userguide/appconfig-creating-configuration-and-profile.html#appconfig-creating-configuration-and-profile-free-form-configurations){target="_blank"} . ## Key features @@ -25,7 +25,7 @@ Feature flags are used to modify behaviour without changing the application's co **Dynamic flags**. Indicates something can have varying states, for example enable a list of premium features for customer X not Y. ???+ tip - You can use [Parameters utility](parameters.md){target="_blank" rel="nofollow"} for static flags while this utility can do both static and dynamic feature flags. + You can use [Parameters utility](parameters.md){target="_blank"} for static flags while this utility can do both static and dynamic feature flags. ???+ warning Be mindful that feature flags can increase the complexity of your application over time; use them sparingly. @@ -37,7 +37,7 @@ If you want to learn more about feature flags, their variations and trade-offs, * [Feature Flags Getting Started - CloudBees](https://www.cloudbees.com/blog/ultimate-feature-flag-guide){target="_blank" rel="nofollow"} ???+ note - AWS AppConfig requires two API calls to fetch configuration for the first time. You can improve latency by consolidating your feature settings in a single [Configuration](https://docs.aws.amazon.com/appconfig/latest/userguide/appconfig-creating-configuration-and-profile.html){target="_blank" rel="nofollow"}. + AWS AppConfig requires two API calls to fetch configuration for the first time. You can improve latency by consolidating your feature settings in a single [Configuration](https://docs.aws.amazon.com/appconfig/latest/userguide/appconfig-creating-configuration-and-profile.html){target="_blank"}. ## Getting started @@ -47,7 +47,7 @@ When using the default store `AppConfigStore`, your Lambda function IAM Role mus ### Required resources -By default, this utility provides [AWS AppConfig](https://docs.aws.amazon.com/appconfig/latest/userguide/what-is-appconfig.html){target="_blank" rel="nofollow"} as a configuration store. +By default, this utility provides [AWS AppConfig](https://docs.aws.amazon.com/appconfig/latest/userguide/what-is-appconfig.html){target="_blank"} as a configuration store. The following sample infrastructure will be used throughout this documentation: @@ -500,7 +500,7 @@ These are the available options for further customization. | **name** | `""` | AWS AppConfig Configuration name, e.g `features` | | **envelope** | `None` | JMESPath expression to use to extract feature flags configuration from AWS AppConfig configuration | | **max_age** | `5` | Number of seconds to cache feature flags configuration fetched from AWS AppConfig | -| **sdk_config** | `None` | [Botocore Config object](https://botocore.amazonaws.com/v1/documentation/api/latest/reference/config.html){target="_blank" rel="nofollow"} | +| **sdk_config** | `None` | [Botocore Config object](https://botocore.amazonaws.com/v1/documentation/api/latest/reference/config.html){target="_blank"} | | **jmespath_options** | `None` | For advanced use cases when you want to bring your own [JMESPath functions](https://github.com/jmespath/jmespath.py#custom-functions){target="_blank" rel="nofollow"} | | **logger** | `logging.Logger` | Logger to use for debug. You can optionally supply an instance of Powertools for AWS Lambda (Python) Logger. | @@ -577,6 +577,6 @@ You can unit test your feature flags locally and independently without setting u | Method | When to use | Requires new deployment on changes | Supported services | | --------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------- | ---------------------------------- | ----------------------------------------------------- | -| **[Environment variables](https://docs.aws.amazon.com/lambda/latest/dg/configuration-envvars.html){target="_blank" rel="nofollow"}** | Simple configuration that will rarely if ever change, because changing it requires a Lambda function deployment. | Yes | Lambda | -| **[Parameters utility](parameters.md){target="_blank" rel="nofollow"}** | Access to secrets, or fetch parameters in different formats from AWS System Manager Parameter Store or Amazon DynamoDB. | No | Parameter Store, DynamoDB, Secrets Manager, AppConfig | +| **[Environment variables](https://docs.aws.amazon.com/lambda/latest/dg/configuration-envvars.html){target="_blank"}** | Simple configuration that will rarely if ever change, because changing it requires a Lambda function deployment. | Yes | Lambda | +| **[Parameters utility](parameters.md){target="_blank"}** | Access to secrets, or fetch parameters in different formats from AWS System Manager Parameter Store or Amazon DynamoDB. | No | Parameter Store, DynamoDB, Secrets Manager, AppConfig | | **Feature flags utility** | Rule engine to define when one or multiple features should be enabled depending on the input. | No | AppConfig | diff --git a/docs/utilities/idempotency.md b/docs/utilities/idempotency.md index 0d018efaed1..1b2128cd7f0 100644 --- a/docs/utilities/idempotency.md +++ b/docs/utilities/idempotency.md @@ -94,14 +94,14 @@ If you're not [changing the default configuration for the DynamoDB persistence l ``` ???+ warning "Warning: Large responses with DynamoDB persistence layer" - When using this utility with DynamoDB, your function's responses must be [smaller than 400KB](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Limits.html#limits-items){target="_blank" rel="nofollow"}. + When using this utility with DynamoDB, your function's responses must be [smaller than 400KB](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Limits.html#limits-items){target="_blank"}. Larger items cannot be written to DynamoDB and will cause exceptions. ???+ info "Info: DynamoDB" Each function invocation will generally make 2 requests to DynamoDB. If the result returned by your Lambda is less than 1kb, you can expect 2 WCUs per invocation. For retried invocations, you will - see 1WCU and 1RCU. Review the [DynamoDB pricing documentation](https://aws.amazon.com/dynamodb/pricing/){target="_blank" rel="nofollow"} to + see 1WCU and 1RCU. Review the [DynamoDB pricing documentation](https://aws.amazon.com/dynamodb/pricing/){target="_blank"} to estimate the cost. ### Idempotent decorator @@ -135,7 +135,7 @@ Similar to [idempotent decorator](#idempotent-decorator), you can use `idempoten When using `idempotent_function`, you must tell us which keyword parameter in your function signature has the data we should use via **`data_keyword_argument`**. -!!! tip "We support JSON serializable data, [Python Dataclasses](https://docs.python.org/3.7/library/dataclasses.html){target="_blank" rel="nofollow"}, [Parser/Pydantic Models](parser.md){target="_blank" rel="nofollow"}, and our [Event Source Data Classes](./data_classes.md){target="_blank" rel="nofollow"}." +!!! tip "We support JSON serializable data, [Python Dataclasses](https://docs.python.org/3.7/library/dataclasses.html){target="_blank" rel="nofollow"}, [Parser/Pydantic Models](parser.md){target="_blank"}, and our [Event Source Data Classes](./data_classes.md){target="_blank"}." ???+ warning "Limitation" Make sure to call your decorated function using keyword arguments. @@ -154,7 +154,7 @@ When using `idempotent_function`, you must tell us which keyword parameter in yo #### Batch integration -You can can easily integrate with [Batch utility](batch.md){target="_blank" rel="nofollow"} via context manager. This ensures that you process each record in an idempotent manner, and guard against a [Lambda timeout](#lambda-timeouts) idempotent situation. +You can can easily integrate with [Batch utility](batch.md){target="_blank"} via context manager. This ensures that you process each record in an idempotent manner, and guard against a [Lambda timeout](#lambda-timeouts) idempotent situation. ???+ "Choosing an unique batch record attribute" In this example, we choose `messageId` as our idempotency key since we know it'll be unique. @@ -193,7 +193,7 @@ If we were to treat the entire request as our idempotency key, a simple HTTP hea The payload extracted by the `event_key_jmespath` is treated as a string by default. This means there could be differences in whitespace even when the JSON payload itself is identical. - To alter this behaviour, we can use the [JMESPath built-in function](jmespath_functions.md#powertools_json-function){target="_blank" rel="nofollow"} `powertools_json()` to treat the payload as a JSON object (dict) rather than a string. + To alter this behaviour, we can use the [JMESPath built-in function](jmespath_functions.md#powertools_json-function){target="_blank"} `powertools_json()` to treat the payload as a JSON object (dict) rather than a string. === "Payment function" @@ -212,7 +212,7 @@ If we were to treat the entire request as our idempotency key, a simple HTTP hea ???+ note This is automatically done when you decorate your Lambda handler with [@idempotent decorator](#idempotent-decorator). -To prevent against extended failed retries when a [Lambda function times out](https://aws.amazon.com/premiumsupport/knowledge-center/lambda-verify-invocation-timeouts/){target="_blank" rel="nofollow"}, +To prevent against extended failed retries when a [Lambda function times out](https://aws.amazon.com/premiumsupport/knowledge-center/lambda-verify-invocation-timeouts/){target="_blank"}, Powertools for AWS Lambda (Python) calculates and includes the remaining invocation available time as part of the idempotency record. ???+ example @@ -498,7 +498,7 @@ Idempotent decorator can be further configured with **`IdempotencyConfig`** as s | Parameter | Default | Description | | ------------------------------- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | -| **event_key_jmespath** | `""` | JMESPath expression to extract the idempotency key from the event record using [built-in functions](/utilities/jmespath_functions){target="_blank" rel="nofollow"} | +| **event_key_jmespath** | `""` | JMESPath expression to extract the idempotency key from the event record using [built-in functions](/utilities/jmespath_functions){target="_blank"} | | **payload_validation_jmespath** | `""` | JMESPath expression to validate whether certain parameters have changed in the event while the event payload | | **raise_on_no_idempotency_key** | `False` | Raise exception if no idempotency key was found in the request | | **expires_after_seconds** | 3600 | The number of seconds to wait before a record is expired | @@ -561,7 +561,7 @@ You can change this window with the **`expires_after_seconds`** parameter: This will mark any records older than 5 minutes as expired, and [your function will be executed as normal if it is invoked with a matching payload](#expired-idempotency-records). ???+ important "Idempotency record expiration vs DynamoDB time-to-live (TTL)" - [DynamoDB TTL is a feature](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/howitworks-ttl.html){target="_blank" rel="nofollow"} to remove items after a certain period of time, it may occur within 48 hours of expiration. + [DynamoDB TTL is a feature](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/howitworks-ttl.html){target="_blank"} to remove items after a certain period of time, it may occur within 48 hours of expiration. We don't rely on DynamoDB or any persistence storage layer to determine whether a record is expired to avoid eventual inconsistency states. @@ -636,7 +636,7 @@ This means that we will raise **`IdempotencyKeyError`** if the evaluation of **` ### Customizing boto configuration -The **`boto_config`** and **`boto3_session`** parameters enable you to pass in a custom [botocore config object](https://botocore.amazonaws.com/v1/documentation/api/latest/reference/config.html){target="_blank" rel="nofollow"} or a custom [boto3 session](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/core/session.html){target="_blank" rel="nofollow"} when constructing the persistence store. +The **`boto_config`** and **`boto3_session`** parameters enable you to pass in a custom [botocore config object](https://botocore.amazonaws.com/v1/documentation/api/latest/reference/config.html){target="_blank"} or a custom [boto3 session](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/core/session.html){target="_blank"} when constructing the persistence store. === "Custom session" @@ -760,7 +760,7 @@ with a truthy value. If you prefer setting this for specific tests, and are usin ### Testing with DynamoDB Local -To test with [DynamoDB Local](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/DynamoDBLocal.DownloadingAndRunning.html){target="_blank" rel="nofollow"}, you can replace the `DynamoDB client` used by the persistence layer with one you create inside your tests. This allows you to set the endpoint_url. +To test with [DynamoDB Local](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/DynamoDBLocal.DownloadingAndRunning.html){target="_blank"}, you can replace the `DynamoDB client` used by the persistence layer with one you create inside your tests. This allows you to set the endpoint_url. === "test_with_dynamodb_local.py" @@ -776,7 +776,7 @@ To test with [DynamoDB Local](https://docs.aws.amazon.com/amazondynamodb/latest/ ### How do I mock all DynamoDB I/O operations -The idempotency utility lazily creates the dynamodb [Table](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/dynamodb.html#table){target="_blank" rel="nofollow"} which it uses to access DynamoDB. +The idempotency utility lazily creates the dynamodb [Table](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/dynamodb.html#table){target="_blank"} which it uses to access DynamoDB. This means it is possible to pass a mocked Table resource, or stub various methods. === "test_with_io_operations.py" @@ -794,4 +794,4 @@ This means it is possible to pass a mocked Table resource, or stub various metho ## Extra resources If you're interested in a deep dive on how Amazon uses idempotency when building our APIs, check out -[this article](https://aws.amazon.com/builders-library/making-retries-safe-with-idempotent-APIs/){target="_blank" rel="nofollow"}. +[this article](https://aws.amazon.com/builders-library/making-retries-safe-with-idempotent-APIs/){target="_blank"}. diff --git a/docs/utilities/jmespath_functions.md b/docs/utilities/jmespath_functions.md index ae1c40c35e1..881b4c926f7 100644 --- a/docs/utilities/jmespath_functions.md +++ b/docs/utilities/jmespath_functions.md @@ -19,11 +19,11 @@ Built-in [JMESPath](https://jmespath.org/){target="_blank" rel="nofollow"} Funct ## Getting started ???+ tip - All examples shared in this documentation are available within the [project repository](https://github.com/aws-powertools/powertools-lambda-python/tree/develop/examples){target="_blank" rel="nofollow"}. + All examples shared in this documentation are available within the [project repository](https://github.com/aws-powertools/powertools-lambda-python/tree/develop/examples){target="_blank"}. You might have events that contains encoded JSON payloads as string, base64, or even in compressed format. It is a common use case to decode and extract them partially or fully as part of your Lambda function invocation. -Powertools for AWS Lambda (Python) also have utilities like [validation](validation.md){target="_blank" rel="nofollow"}, [idempotency](idempotency.md){target="_blank" rel="nofollow"}, or [feature flags](feature_flags.md){target="_blank" rel="nofollow"} where you might need to extract a portion of your data before using them. +Powertools for AWS Lambda (Python) also have utilities like [validation](validation.md){target="_blank"}, [idempotency](idempotency.md){target="_blank"}, or [feature flags](feature_flags.md){target="_blank"} where you might need to extract a portion of your data before using them. ???+ info "Terminology" **Envelope** is the terminology we use for the **JMESPath expression** to extract your JSON object from your data input. We might use those two terms interchangeably. @@ -120,7 +120,7 @@ This sample will deserialize the JSON string within the `data` key before valida > **Idempotency scenario** -This sample will deserialize the JSON string within the `body` key before [Idempotency](./idempotency.md){target="_blank" rel="nofollow"} processes it. +This sample will deserialize the JSON string within the `body` key before [Idempotency](./idempotency.md){target="_blank"} processes it. === "powertools_json_idempotency_jmespath.py" diff --git a/docs/utilities/middleware_factory.md b/docs/utilities/middleware_factory.md index 9ff3a6fb9c7..41670025b1e 100644 --- a/docs/utilities/middleware_factory.md +++ b/docs/utilities/middleware_factory.md @@ -15,7 +15,7 @@ Middleware factory provides a decorator factory to create your own middleware to ## Getting started ???+ tip - All examples shared in this documentation are available within the [project repository](https://github.com/aws-powertools/powertools-lambda-python/tree/develop/examples){target="_blank" rel="nofollow"}. + All examples shared in this documentation are available within the [project repository](https://github.com/aws-powertools/powertools-lambda-python/tree/develop/examples){target="_blank"}. You might need a custom middleware to abstract non-functional code. These are often custom authorization or any reusable logic you might need to run before/after a Lambda function invocation. @@ -70,7 +70,7 @@ You can also have your own keyword arguments after the mandatory arguments. ## Advanced -For advanced use cases, you can instantiate [Tracer](../core/tracer.md){target="_blank" rel="nofollow"} inside your middleware, and add annotations as well as metadata for additional operational insights. +For advanced use cases, you can instantiate [Tracer](../core/tracer.md){target="_blank"} inside your middleware, and add annotations as well as metadata for additional operational insights. === "advanced_middleware_tracer_function.py" ```python hl_lines="7 9 12 16 17 19 25 42" @@ -87,12 +87,12 @@ For advanced use cases, you can instantiate [Tracer](../core/tracer.md){target=" ### Tracing middleware **execution** -If you are making use of [Tracer](../core/tracer.md){target="_blank" rel="nofollow"}, you can trace the execution of your middleware to ease operations. +If you are making use of [Tracer](../core/tracer.md){target="_blank"}, you can trace the execution of your middleware to ease operations. This makes use of an existing Tracer instance that you may have initialized anywhere in your code. ???+ warning - You must [enable Active Tracing](../core/tracer/#permissions){target="_blank" rel="nofollow"} in your Lambda function when using this feature, otherwise Lambda cannot send traces to XRay. + You must [enable Active Tracing](../core/tracer/#permissions){target="_blank"} in your Lambda function when using this feature, otherwise Lambda cannot send traces to XRay. === "getting_started_middleware_tracer_function.py" ```python hl_lines="8 14 15 36" @@ -105,14 +105,14 @@ This makes use of an existing Tracer instance that you may have initialized anyw --8<-- "examples/middleware_factory/src/getting_started_middleware_tracer_payload.json" ``` -When executed, your middleware name will [appear in AWS X-Ray Trace details as](../core/tracer.md){target="_blank" rel="nofollow"} `## middleware_name`, in this example the middleware name is `## middleware_with_tracing`. +When executed, your middleware name will [appear in AWS X-Ray Trace details as](../core/tracer.md){target="_blank"} `## middleware_name`, in this example the middleware name is `## middleware_with_tracing`. ![Middleware simple Tracer](../media/middleware_factory_tracer_1.png) ### Combining Powertools for AWS Lambda (Python) utilities -You can create your own middleware and combine many features of Powertools for AWS Lambda (Python) such as [trace](../core/logger.md){target="_blank" rel="nofollow"}, [logs](../core/logger.md){target="_blank" rel="nofollow"}, [feature flags](feature_flags.md){target="_blank" rel="nofollow"}, [validation](validation.md){target="_blank" rel="nofollow"}, [jmespath_functions](jmespath_functions.md){target="_blank" rel="nofollow"} and others to abstract non-functional code. +You can create your own middleware and combine many features of Powertools for AWS Lambda (Python) such as [trace](../core/logger.md){target="_blank"}, [logs](../core/logger.md){target="_blank"}, [feature flags](feature_flags.md){target="_blank"}, [validation](validation.md){target="_blank"}, [jmespath_functions](jmespath_functions.md){target="_blank"} and others to abstract non-functional code. In the example below, we create a Middleware with the following features: diff --git a/docs/utilities/parameters.md b/docs/utilities/parameters.md index 867939da3f1..fa3241f43a8 100644 --- a/docs/utilities/parameters.md +++ b/docs/utilities/parameters.md @@ -4,7 +4,7 @@ description: Utility --- -The parameters utility provides high-level functions to retrieve one or multiple parameter values from [AWS Systems Manager Parameter Store](https://docs.aws.amazon.com/systems-manager/latest/userguide/systems-manager-parameter-store.html){target="_blank" rel="nofollow"}, [AWS Secrets Manager](https://aws.amazon.com/secrets-manager/){target="_blank" rel="nofollow"}, [AWS AppConfig](https://docs.aws.amazon.com/appconfig/latest/userguide/what-is-appconfig.html){target="_blank" rel="nofollow"}, [Amazon DynamoDB](https://aws.amazon.com/dynamodb/){target="_blank" rel="nofollow"}, or bring your own. +The parameters utility provides high-level functions to retrieve one or multiple parameter values from [AWS Systems Manager Parameter Store](https://docs.aws.amazon.com/systems-manager/latest/userguide/systems-manager-parameter-store.html){target="_blank"}, [AWS Secrets Manager](https://aws.amazon.com/secrets-manager/){target="_blank"}, [AWS AppConfig](https://docs.aws.amazon.com/appconfig/latest/userguide/what-is-appconfig.html){target="_blank"}, [Amazon DynamoDB](https://aws.amazon.com/dynamodb/){target="_blank"}, or bring your own. ## Key features @@ -16,7 +16,7 @@ The parameters utility provides high-level functions to retrieve one or multiple ## Getting started ???+ tip - All examples shared in this documentation are available within the [project repository](https://github.com/aws-powertools/powertools-lambda-python/tree/develop/examples){target="_blank" rel="nofollow"}. + All examples shared in this documentation are available within the [project repository](https://github.com/aws-powertools/powertools-lambda-python/tree/develop/examples){target="_blank"}. By default, we fetch parameters from System Manager Parameter Store, secrets from Secrets Manager, and application configuration from AppConfig. @@ -304,7 +304,7 @@ You can create your own custom parameter store provider by inheriting the `BaseP All transformation and caching logic is handled by the `get()` and `get_multiple()` methods from the base provider class. -Here are two examples of implementing a custom parameter store. One using an external service like [Hashicorp Vault](https://www.vaultproject.io/){target="_blank" rel="nofollow"}, a widely popular key-value and secret storage and the other one using [Amazon S3](https://aws.amazon.com/s3/?nc1=h_ls){target="_blank" rel="nofollow"}, a popular object storage. +Here are two examples of implementing a custom parameter store. One using an external service like [Hashicorp Vault](https://www.vaultproject.io/){target="_blank" rel="nofollow"}, a widely popular key-value and secret storage and the other one using [Amazon S3](https://aws.amazon.com/s3/?nc1=h_ls){target="_blank"}, a popular object storage. === "working_with_own_provider_vault.py" ```python hl_lines="5 13 20 24" @@ -399,15 +399,15 @@ Here is the mapping between this utility's functions and methods and the underly | Provider | Function/Method | Client name | Function name | | ------------------- | ------------------------------- | ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| SSM Parameter Store | `get_parameter` | `ssm` | [get_parameter](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/ssm.html#SSM.Client.get_parameter){target="_blank" rel="nofollow"} | -| SSM Parameter Store | `get_parameters` | `ssm` | [get_parameters_by_path](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/ssm.html#SSM.Client.get_parameters_by_path){target="_blank" rel="nofollow"} | -| SSM Parameter Store | `SSMProvider.get` | `ssm` | [get_parameter](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/ssm.html#SSM.Client.get_parameter){target="_blank" rel="nofollow"} | -| SSM Parameter Store | `SSMProvider.get_multiple` | `ssm` | [get_parameters_by_path](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/ssm.html#SSM.Client.get_parameters_by_path){target="_blank" rel="nofollow"} | -| Secrets Manager | `get_secret` | `secretsmanager` | [get_secret_value](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/secretsmanager.html#SecretsManager.Client.get_secret_value){target="_blank" rel="nofollow"} | -| Secrets Manager | `SecretsProvider.get` | `secretsmanager` | [get_secret_value](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/secretsmanager.html#SecretsManager.Client.get_secret_value){target="_blank" rel="nofollow"} | -| DynamoDB | `DynamoDBProvider.get` | `dynamodb` | ([Table resource](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/dynamodb.html#table){target="_blank" rel="nofollow"}) | [get_item](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/dynamodb.html#DynamoDB.Table.get_item) | -| DynamoDB | `DynamoDBProvider.get_multiple` | `dynamodb` | ([Table resource](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/dynamodb.html#table){target="_blank" rel="nofollow"}) | [query](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/dynamodb.html#DynamoDB.Table.query) | -| App Config | `get_app_config` | `appconfigdata` | [start_configuration_session](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/appconfigdata.html#AppConfigData.Client.start_configuration_session){target="_blank" rel="nofollow"} and [get_latest_configuration](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/appconfigdata.html#AppConfigData.Client.get_latest_configuration){target="_blank" rel="nofollow"} | +| SSM Parameter Store | `get_parameter` | `ssm` | [get_parameter](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/ssm.html#SSM.Client.get_parameter){target="_blank"} | +| SSM Parameter Store | `get_parameters` | `ssm` | [get_parameters_by_path](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/ssm.html#SSM.Client.get_parameters_by_path){target="_blank"} | +| SSM Parameter Store | `SSMProvider.get` | `ssm` | [get_parameter](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/ssm.html#SSM.Client.get_parameter){target="_blank"} | +| SSM Parameter Store | `SSMProvider.get_multiple` | `ssm` | [get_parameters_by_path](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/ssm.html#SSM.Client.get_parameters_by_path){target="_blank"} | +| Secrets Manager | `get_secret` | `secretsmanager` | [get_secret_value](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/secretsmanager.html#SecretsManager.Client.get_secret_value){target="_blank"} | +| Secrets Manager | `SecretsProvider.get` | `secretsmanager` | [get_secret_value](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/secretsmanager.html#SecretsManager.Client.get_secret_value){target="_blank"} | +| DynamoDB | `DynamoDBProvider.get` | `dynamodb` | ([Table resource](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/dynamodb.html#table){target="_blank"}) | [get_item](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/dynamodb.html#DynamoDB.Table.get_item) | +| DynamoDB | `DynamoDBProvider.get_multiple` | `dynamodb` | ([Table resource](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/dynamodb.html#table){target="_blank"}) | [query](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/dynamodb.html#DynamoDB.Table.query) | +| App Config | `get_app_config` | `appconfigdata` | [start_configuration_session](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/appconfigdata.html#AppConfigData.Client.start_configuration_session){target="_blank"} and [get_latest_configuration](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/appconfigdata.html#AppConfigData.Client.get_latest_configuration){target="_blank"} | ### Bring your own boto client @@ -432,7 +432,7 @@ Bringing them together in a single code snippet would look like this: ### Customizing boto configuration -The **`config`** , **`boto3_session`**, and **`boto3_client`** parameters enable you to pass in a custom [botocore config object](https://botocore.amazonaws.com/v1/documentation/api/latest/reference/config.html){target="_blank" rel="nofollow"}, [boto3 session](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/core/session.html){target="_blank" rel="nofollow"}, or a [boto3 client](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/core/boto3.html){target="_blank" rel="nofollow"} when constructing any of the built-in provider classes. +The **`config`** , **`boto3_session`**, and **`boto3_client`** parameters enable you to pass in a custom [botocore config object](https://botocore.amazonaws.com/v1/documentation/api/latest/reference/config.html){target="_blank"}, [boto3 session](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/core/session.html){target="_blank"}, or a [boto3 client](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/core/boto3.html){target="_blank"} when constructing any of the built-in provider classes. ???+ tip You can use a custom session for retrieving parameters cross-account/region and for snapshot testing. diff --git a/docs/utilities/parser.md b/docs/utilities/parser.md index 6abef997a5e..846460e43d2 100644 --- a/docs/utilities/parser.md +++ b/docs/utilities/parser.md @@ -21,7 +21,7 @@ Powertools for AWS Lambda (Python) supports Pydantic v1 and v2. Each Pydantic ve #### Using Pydantic v1 -!!! info "This is not necessary if you're installing Powertools for AWS Lambda (Python) via [Lambda Layer/SAR](../index.md#lambda-layer){target="_blank" rel="nofollow"}" +!!! info "This is not necessary if you're installing Powertools for AWS Lambda (Python) via [Lambda Layer/SAR](../index.md#lambda-layer){target="_blank"}" Add `aws-lambda-powertools[parser]` as a dependency in your preferred tool: _e.g._, _requirements.txt_, _pyproject.toml_. @@ -35,7 +35,7 @@ Add `aws-lambda-powertools[parser]` as a dependency in your preferred tool: _e.g #### Using Pydantic v2 -You need to bring Pydantic v2.0.3 or later as an external dependency. Note that [we suppress Pydantic v2 deprecation warnings](https://github.com/aws-powertools/powertools-lambda-python/issues/2672){target="_blank" rel="nofollow"} to reduce noise and optimize log costs. +You need to bring Pydantic v2.0.3 or later as an external dependency. Note that [we suppress Pydantic v2 deprecation warnings](https://github.com/aws-powertools/powertools-lambda-python/issues/2672){target="_blank"} to reduce noise and optimize log costs. Add `aws-lambda-powertools` and `pydantic>=2.0.3` as a dependency in your preferred tool: _e.g._, _requirements.txt_, _pyproject.toml_. diff --git a/docs/utilities/streaming.md b/docs/utilities/streaming.md index 5c43bb8bc9b..49a114235af 100644 --- a/docs/utilities/streaming.md +++ b/docs/utilities/streaming.md @@ -185,6 +185,6 @@ Create an input payload using `io.BytesIO` and assert the response of the transf ### AWS X-Ray segment size limit -We make multiple API calls to S3 as you read chunks from your S3 object. If your function is decorated with [Tracer](./../core/tracer.md){target="_blank" rel="nofollow"}, you can easily hit [AWS X-Ray 64K segment size](https://docs.aws.amazon.com/general/latest/gr/xray.html#limits_xray){target="_blank" rel="nofollow"} when processing large files. +We make multiple API calls to S3 as you read chunks from your S3 object. If your function is decorated with [Tracer](./../core/tracer.md){target="_blank"}, you can easily hit [AWS X-Ray 64K segment size](https://docs.aws.amazon.com/general/latest/gr/xray.html#limits_xray){target="_blank"} when processing large files. !!! tip "Use tracer decorators in parts where you don't read your `S3Object` instead." diff --git a/docs/utilities/typing.md b/docs/utilities/typing.md index a76d53ab78a..9e2cc5024ab 100644 --- a/docs/utilities/typing.md +++ b/docs/utilities/typing.md @@ -18,9 +18,9 @@ This typing utility provides static typing classes that can be used to ease the ## Getting started ???+ tip - All examples shared in this documentation are available within the [project repository](https://github.com/aws-powertools/powertools-lambda-python/tree/develop/examples){target="_blank" rel="nofollow"}. + All examples shared in this documentation are available within the [project repository](https://github.com/aws-powertools/powertools-lambda-python/tree/develop/examples){target="_blank"}. -We provide static typing for any context methods or properties implemented by [Lambda context object](https://docs.aws.amazon.com/lambda/latest/dg/python-context.html){target="_blank" rel="nofollow"}. +We provide static typing for any context methods or properties implemented by [Lambda context object](https://docs.aws.amazon.com/lambda/latest/dg/python-context.html){target="_blank"}. ## LambdaContext diff --git a/docs/utilities/validation.md b/docs/utilities/validation.md index 7dc009db38d..7d822cf4b4c 100644 --- a/docs/utilities/validation.md +++ b/docs/utilities/validation.md @@ -16,7 +16,7 @@ This utility provides JSON Schema validation for events and responses, including ## Getting started ???+ tip - All examples shared in this documentation are available within the [project repository](https://github.com/aws-powertools/powertools-lambda-python/tree/develop/examples){target="_blank" rel="nofollow"}. + All examples shared in this documentation are available within the [project repository](https://github.com/aws-powertools/powertools-lambda-python/tree/develop/examples){target="_blank"}. You can validate inbound and outbound events using [`validator` decorator](#validator-decorator). @@ -32,7 +32,7 @@ You can also use the standalone `validate` function, if you want more control ov ### Install -!!! info "This is not necessary if you're installing Powertools for AWS Lambda (Python) via [Lambda Layer/SAR](../index.md#lambda-layer){target="_blank" rel="nofollow"}" +!!! info "This is not necessary if you're installing Powertools for AWS Lambda (Python) via [Lambda Layer/SAR](../index.md#lambda-layer){target="_blank"}" Add `aws-lambda-powertools[validation]` as a dependency in your preferred tool: _e.g._, _requirements.txt_, _pyproject.toml_. This will ensure you have the required dependencies before using Validation. @@ -194,7 +194,7 @@ For each format defined in a dictionary key, you must use a regex, or a function You might have events or responses that contain non-encoded JSON, where you need to decode before validating them. -You can use our built-in [JMESPath functions](./jmespath_functions.md){target="_blank" rel="nofollow"} within your expressions to do exactly that to [deserialize JSON Strings](./jmespath_functions.md#powertools_json-function){target="_blank" rel="nofollow"}, [decode base64](./jmespath_functions.md#powertools_base64-function){target="_blank" rel="nofollow"}, and [decompress gzip data](./jmespath_functions.md#powertools_base64_gzip-function){target="_blank" rel="nofollow"}. +You can use our built-in [JMESPath functions](./jmespath_functions.md){target="_blank"} within your expressions to do exactly that to [deserialize JSON Strings](./jmespath_functions.md#powertools_json-function){target="_blank"}, [decode base64](./jmespath_functions.md#powertools_base64-function){target="_blank"}, and [decompress gzip data](./jmespath_functions.md#powertools_base64_gzip-function){target="_blank"}. ???+ info We use these for [built-in envelopes](#built-in-envelopes) to easily to decode and unwrap events from sources like Kinesis, CloudWatch Logs, etc.