diff --git a/website/www/site/content/en/documentation/sdks/python-unrecoverable-errors.md b/website/www/site/content/en/documentation/sdks/python-unrecoverable-errors.md new file mode 100644 index 000000000000..4e5d94ce8a8d --- /dev/null +++ b/website/www/site/content/en/documentation/sdks/python-unrecoverable-errors.md @@ -0,0 +1,61 @@ +--- +type: languages +title: "Unrecoverable Errors in Beam Python" +--- + + +# Unrecoverable Errors in Beam Python + +## What is an Unrecoverable Error? + +An unrecoverable error is an issue at job start-up time that will +prevent a job from ever running successfully, usually due to some kind +of misconfiguration. Solving these issues when they occur is key to +successfully running a Beam Python pipeline. + +## Common Unrecoverable Errors + +### Job Submission/Runtime Python Version Mismatch + +If the Python version used for job submission does not match the +Python version used to build the worker container, the job will not +execute. Ensure that the Python version being used for job submission +and the container Python version match. + +### PIP Dependency Resolution Failures + +During worker start-up, dependencies are checked and installed in +the worker container before accepting work. If a pipeline requires +additional dependencies not already present in the runtime environment, +they are installed here. If there’s an issue during this process +(e.g. a dependency version cannot be found, or a worker cannot +connect to PyPI) the worker will fail and may try to restart +depending on the runner. Ensure that dependency versions provided in +your requirements.txt file exist and can be installed locally before +submitting jobs. + +### Dependency Version Mismatches + +When additional dependencies like `torch`, `transformers`, etc. are not +specified via a requirements_file or preinstalled in a custom container +then the worker might fail to deserialize (unpickle) the user code. +This can result in `ModuleNotFound` errors. If dependencies are installed +but their versions don't match the versions in submission environment, +pipeline might have `AttributeError` messages. + +Ensure that the required dependencies at runtime and in the submission +environment are the same along with their versions. For better visibility, +debug logs are added specifying the dependencies at both stages starting in +Beam 2.52.0. For more information, see: https://beam.apache.org/documentation/sdks/python-pipeline-dependencies/#control-dependencies \ No newline at end of file diff --git a/website/www/site/content/en/documentation/sdks/python.md b/website/www/site/content/en/documentation/sdks/python.md index 80c26c258d5b..dc9f6a54d893 100644 --- a/website/www/site/content/en/documentation/sdks/python.md +++ b/website/www/site/content/en/documentation/sdks/python.md @@ -59,3 +59,7 @@ see [Machine Learning](/documentation/sdks/python-machine-learning). ## Python multi-language pipelines quickstart Apache Beam lets you combine transforms written in any supported SDK language and use them in one multi-language pipeline. To learn how to create a multi-language pipeline using the Python SDK, see the [Python multi-language pipelines quickstart](/documentation/sdks/python-multi-language-pipelines). + +## Unrecoverable Errors in Beam Python + +Some common errors can occur during worker start-up and prevent jobs from starting. To learn about these errors and how to troubleshoot them in the Python SDK, see [Unrecoverable Errors in Beam Python](/documentation/sdks/python-unrecoverable-errors). \ No newline at end of file diff --git a/website/www/site/layouts/partials/section-menu/en/sdks.html b/website/www/site/layouts/partials/section-menu/en/sdks.html index 434bddb936b6..73bea15a28d1 100644 --- a/website/www/site/layouts/partials/section-menu/en/sdks.html +++ b/website/www/site/layouts/partials/section-menu/en/sdks.html @@ -43,6 +43,7 @@