From 7a53da05387285d44d6ab42989bde19a1716bd6e Mon Sep 17 00:00:00 2001 From: Jack McCluskey Date: Wed, 27 Sep 2023 13:44:35 -0400 Subject: [PATCH 01/10] Create documentation page for Python SDK unrecoverable errors --- .../sdks/python-unrecoverable-errors.md | 34 +++++++++++++++++++ 1 file changed, 34 insertions(+) create mode 100644 website/www/site/content/en/documentation/sdks/python-unrecoverable-errors.md diff --git a/website/www/site/content/en/documentation/sdks/python-unrecoverable-errors.md b/website/www/site/content/en/documentation/sdks/python-unrecoverable-errors.md new file mode 100644 index 000000000000..dc697844e468 --- /dev/null +++ b/website/www/site/content/en/documentation/sdks/python-unrecoverable-errors.md @@ -0,0 +1,34 @@ +--- +type: languages +title: "Unrecoverable Errors in Beam Python" +--- + + +# Unrecoverable Errors in Beam Python + +## What is an Unrecoverable Error? + +An unrecoverable error is an issue at job start-up time that will +prevent a job from ever running successfully, usually due to some kind +of misconfiguration. Solving these issues when they occur is key to +successfully running a Beam Python pipeline. + +## Common Unrecoverable Errors + +| Error | Explanation | Troubleshooting | +|---|---|---| +| Job Submission/Runtime Python Version Mismatch | If the Python version used for job submission does not match the Python version used to build the worker container, the job will not execute. | Ensure that the Python version being used for job submission and the container Python version match. | +| PIP Dependency Resolution Failures | During worker start-up, dependencies are checked and installed in the worker container before accepting work. If there’s an issue during this process (e.g. a dependency version cannot be found) the worker will restart and try again up to four times before outright failing. | Ensure that dependency versions provided in your requirements.txt file exist and can be installed locally before submitting jobs. | +| Dependency Version Mismatches | When additional dependencies like torch, transformers, etc are not specified via requirements_file or preinstalled with a custom container then the worker may go into a restart loop trying to install dependencies again up to 4 times and finally fail. There is a debug log specifying `ModuleNotFoundError`. Similar outcome is observed when there is a dependency mismatch that often has `AttributeError` logged in debug mode. | Ensure that the required dependencies at runtime and in the submission environment are the same along with their versions. For better visibility, debug logs are added specifying the dependencies at both stages from Beam 2.52.0 | \ No newline at end of file From c8da589be3635f464b4fb013901101585672b52f Mon Sep 17 00:00:00 2001 From: Jack McCluskey Date: Wed, 27 Sep 2023 15:38:17 -0400 Subject: [PATCH 02/10] Trailing whitespace --- .../en/documentation/sdks/python-unrecoverable-errors.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/website/www/site/content/en/documentation/sdks/python-unrecoverable-errors.md b/website/www/site/content/en/documentation/sdks/python-unrecoverable-errors.md index dc697844e468..979d469135f4 100644 --- a/website/www/site/content/en/documentation/sdks/python-unrecoverable-errors.md +++ b/website/www/site/content/en/documentation/sdks/python-unrecoverable-errors.md @@ -20,8 +20,8 @@ limitations under the License. ## What is an Unrecoverable Error? -An unrecoverable error is an issue at job start-up time that will -prevent a job from ever running successfully, usually due to some kind +An unrecoverable error is an issue at job start-up time that will +prevent a job from ever running successfully, usually due to some kind of misconfiguration. Solving these issues when they occur is key to successfully running a Beam Python pipeline. From ad145aa3ef76022e020cb8c5f3a0d6f2bcd29d40 Mon Sep 17 00:00:00 2001 From: Jack McCluskey Date: Wed, 27 Sep 2023 16:22:37 -0400 Subject: [PATCH 03/10] Add link to page on Python SDK page --- website/www/site/content/en/documentation/sdks/python.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/website/www/site/content/en/documentation/sdks/python.md b/website/www/site/content/en/documentation/sdks/python.md index 80c26c258d5b..dc9f6a54d893 100644 --- a/website/www/site/content/en/documentation/sdks/python.md +++ b/website/www/site/content/en/documentation/sdks/python.md @@ -59,3 +59,7 @@ see [Machine Learning](/documentation/sdks/python-machine-learning). ## Python multi-language pipelines quickstart Apache Beam lets you combine transforms written in any supported SDK language and use them in one multi-language pipeline. To learn how to create a multi-language pipeline using the Python SDK, see the [Python multi-language pipelines quickstart](/documentation/sdks/python-multi-language-pipelines). + +## Unrecoverable Errors in Beam Python + +Some common errors can occur during worker start-up and prevent jobs from starting. To learn about these errors and how to troubleshoot them in the Python SDK, see [Unrecoverable Errors in Beam Python](/documentation/sdks/python-unrecoverable-errors). \ No newline at end of file From d3fd9143c493ae6ab1a19747f0130ab15b373401 Mon Sep 17 00:00:00 2001 From: Jack McCluskey Date: Wed, 27 Sep 2023 16:25:30 -0400 Subject: [PATCH 04/10] Add to sidebar --- website/www/site/layouts/partials/section-menu/en/sdks.html | 1 + 1 file changed, 1 insertion(+) diff --git a/website/www/site/layouts/partials/section-menu/en/sdks.html b/website/www/site/layouts/partials/section-menu/en/sdks.html index 434bddb936b6..73bea15a28d1 100644 --- a/website/www/site/layouts/partials/section-menu/en/sdks.html +++ b/website/www/site/layouts/partials/section-menu/en/sdks.html @@ -43,6 +43,7 @@
  • Machine Learning
  • Managing pipeline dependencies
  • Python multi-language pipelines quickstart
    • +
  • Python Unrecoverable Errors
    • From 940c6bc6c24e0e830ccaab8d063bd1dac420232e Mon Sep 17 00:00:00 2001 From: Jack McCluskey Date: Thu, 28 Sep 2023 10:24:40 -0400 Subject: [PATCH 05/10] Ditch table for text --- .../sdks/python-unrecoverable-errors.md | 32 ++++++++++++++++--- 1 file changed, 27 insertions(+), 5 deletions(-) diff --git a/website/www/site/content/en/documentation/sdks/python-unrecoverable-errors.md b/website/www/site/content/en/documentation/sdks/python-unrecoverable-errors.md index 979d469135f4..a07e44f25844 100644 --- a/website/www/site/content/en/documentation/sdks/python-unrecoverable-errors.md +++ b/website/www/site/content/en/documentation/sdks/python-unrecoverable-errors.md @@ -27,8 +27,30 @@ successfully running a Beam Python pipeline. ## Common Unrecoverable Errors -| Error | Explanation | Troubleshooting | -|---|---|---| -| Job Submission/Runtime Python Version Mismatch | If the Python version used for job submission does not match the Python version used to build the worker container, the job will not execute. | Ensure that the Python version being used for job submission and the container Python version match. | -| PIP Dependency Resolution Failures | During worker start-up, dependencies are checked and installed in the worker container before accepting work. If there’s an issue during this process (e.g. a dependency version cannot be found) the worker will restart and try again up to four times before outright failing. | Ensure that dependency versions provided in your requirements.txt file exist and can be installed locally before submitting jobs. | -| Dependency Version Mismatches | When additional dependencies like torch, transformers, etc are not specified via requirements_file or preinstalled with a custom container then the worker may go into a restart loop trying to install dependencies again up to 4 times and finally fail. There is a debug log specifying `ModuleNotFoundError`. Similar outcome is observed when there is a dependency mismatch that often has `AttributeError` logged in debug mode. | Ensure that the required dependencies at runtime and in the submission environment are the same along with their versions. For better visibility, debug logs are added specifying the dependencies at both stages from Beam 2.52.0 | \ No newline at end of file +### Job Submission/Runtime Python Version Mismatch + +If the Python version used for job submission does not match the +Python version used to build the worker container, the job will not +execute. Ensure that the Python version being used for job submission +and the container Python version match. + +### PIP Dependency Resolution Failures + +During worker start-up, dependencies are checked and installed in +the worker container before accepting work. If there’s an issue during +this process (e.g. a dependency version cannot be found) the worker +will restart and try again up to four times before outright failing. +Ensure that dependency versions provided in your requirements.txt file +exist and can be installed locally before submitting jobs. + +### Dependency Verision Mismatches + +When additional dependencies like torch, transformers, etc are not +specified via requirements_file or preinstalled with a custom container +then the worker may go into a restart loop trying to install dependencies +again up to 4 times and finally fail. There is a debug log specifying `ModuleNotFoundError`. +A similar outcome is observed when there is a dependency mismatch that +often has `AttributeError` logged in debug mode. Ensure that the required +dependencies at runtime and in the submission environment are the same +along with their versions. For better visibility, debug logs are added +specifying the dependencies at both stages starting in Beam 2.52.0. \ No newline at end of file From c4259bd6446a44129bbe7e0ef6ea60e4184f92c4 Mon Sep 17 00:00:00 2001 From: Jack McCluskey Date: Thu, 28 Sep 2023 13:38:58 -0400 Subject: [PATCH 06/10] Address comments --- .../sdks/python-unrecoverable-errors.md | 16 ++++++++++------ 1 file changed, 10 insertions(+), 6 deletions(-) diff --git a/website/www/site/content/en/documentation/sdks/python-unrecoverable-errors.md b/website/www/site/content/en/documentation/sdks/python-unrecoverable-errors.md index a07e44f25844..da1c089b6ed7 100644 --- a/website/www/site/content/en/documentation/sdks/python-unrecoverable-errors.md +++ b/website/www/site/content/en/documentation/sdks/python-unrecoverable-errors.md @@ -37,11 +37,14 @@ and the container Python version match. ### PIP Dependency Resolution Failures During worker start-up, dependencies are checked and installed in -the worker container before accepting work. If there’s an issue during -this process (e.g. a dependency version cannot be found) the worker -will restart and try again up to four times before outright failing. -Ensure that dependency versions provided in your requirements.txt file -exist and can be installed locally before submitting jobs. +the worker container before accepting work. If a pipeline requires +additional dependencies not already present in the runtime environment, +they are installed here. If there’s an issue during this process +(e.g. a dependency version cannot be found, or a worker cannot +connect to PyPI) the worker will fail and may try to restart +depending on the runner. Ensure that dependency versions provided in +your requirements.txt file exist and can be installed locally before +submitting jobs. ### Dependency Verision Mismatches @@ -53,4 +56,5 @@ A similar outcome is observed when there is a dependency mismatch that often has `AttributeError` logged in debug mode. Ensure that the required dependencies at runtime and in the submission environment are the same along with their versions. For better visibility, debug logs are added -specifying the dependencies at both stages starting in Beam 2.52.0. \ No newline at end of file +specifying the dependencies at both stages starting in Beam 2.52.0. +For more information, see: https://beam.apache.org/documentation/sdks/python-pipeline-dependencies/#control-dependencies \ No newline at end of file From cb3b309dbe0083aebf5807b21579a46451ca2733 Mon Sep 17 00:00:00 2001 From: Jack McCluskey <34928439+jrmccluskey@users.noreply.github.com> Date: Thu, 28 Sep 2023 14:20:42 -0400 Subject: [PATCH 07/10] Update website/www/site/content/en/documentation/sdks/python-unrecoverable-errors.md Co-authored-by: tvalentyn --- .../en/documentation/sdks/python-unrecoverable-errors.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/www/site/content/en/documentation/sdks/python-unrecoverable-errors.md b/website/www/site/content/en/documentation/sdks/python-unrecoverable-errors.md index da1c089b6ed7..f9f609e972c8 100644 --- a/website/www/site/content/en/documentation/sdks/python-unrecoverable-errors.md +++ b/website/www/site/content/en/documentation/sdks/python-unrecoverable-errors.md @@ -46,7 +46,7 @@ depending on the runner. Ensure that dependency versions provided in your requirements.txt file exist and can be installed locally before submitting jobs. -### Dependency Verision Mismatches +### Dependency Version Mismatches When additional dependencies like torch, transformers, etc are not specified via requirements_file or preinstalled with a custom container From 2d8ca07963f7a946b4cbbb8b134093314064b994 Mon Sep 17 00:00:00 2001 From: Jack McCluskey Date: Thu, 28 Sep 2023 14:29:39 -0400 Subject: [PATCH 08/10] anchor links + edit --- .../sdks/python-unrecoverable-errors.md | 29 ++++++++++--------- 1 file changed, 15 insertions(+), 14 deletions(-) diff --git a/website/www/site/content/en/documentation/sdks/python-unrecoverable-errors.md b/website/www/site/content/en/documentation/sdks/python-unrecoverable-errors.md index f9f609e972c8..38bd22f99179 100644 --- a/website/www/site/content/en/documentation/sdks/python-unrecoverable-errors.md +++ b/website/www/site/content/en/documentation/sdks/python-unrecoverable-errors.md @@ -18,23 +18,23 @@ limitations under the License. # Unrecoverable Errors in Beam Python -## What is an Unrecoverable Error? +## What is an Unrecoverable Error? An unrecoverable error is an issue at job start-up time that will prevent a job from ever running successfully, usually due to some kind of misconfiguration. Solving these issues when they occur is key to successfully running a Beam Python pipeline. -## Common Unrecoverable Errors +## Common Unrecoverable Errors -### Job Submission/Runtime Python Version Mismatch +### Job Submission/Runtime Python Version Mismatch If the Python version used for job submission does not match the Python version used to build the worker container, the job will not execute. Ensure that the Python version being used for job submission and the container Python version match. -### PIP Dependency Resolution Failures +### PIP Dependency Resolution Failures During worker start-up, dependencies are checked and installed in the worker container before accepting work. If a pipeline requires @@ -48,13 +48,14 @@ submitting jobs. ### Dependency Version Mismatches -When additional dependencies like torch, transformers, etc are not -specified via requirements_file or preinstalled with a custom container -then the worker may go into a restart loop trying to install dependencies -again up to 4 times and finally fail. There is a debug log specifying `ModuleNotFoundError`. -A similar outcome is observed when there is a dependency mismatch that -often has `AttributeError` logged in debug mode. Ensure that the required -dependencies at runtime and in the submission environment are the same -along with their versions. For better visibility, debug logs are added -specifying the dependencies at both stages starting in Beam 2.52.0. -For more information, see: https://beam.apache.org/documentation/sdks/python-pipeline-dependencies/#control-dependencies \ No newline at end of file +When additional dependencies like `torch`, `transformers`, etc. are not +specified via a requirements_file or preinstalled in a custom container +then the worker might fail to deserialize (unpickle) the user code. +This can result in `ModuleNotFound` errors. If dependencies are installed +but their versions don't match the versions in submission environment, +pipeline might have `AttributeError` messages. + +Ensure that the required dependencies at runtime and in the submission +environment are the same along with their versions. For better visibility, +debug logs are added specifying the dependencies at both stages starting in +Beam 2.52.0. For more information, see: https://beam.apache.org/documentation/sdks/python-pipeline-dependencies/#control-dependencies \ No newline at end of file From 77d0f909b6d0d240fa9181dfe11cfda3755f1dc8 Mon Sep 17 00:00:00 2001 From: Jack McCluskey Date: Thu, 28 Sep 2023 14:30:12 -0400 Subject: [PATCH 09/10] last anchor --- .../en/documentation/sdks/python-unrecoverable-errors.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/www/site/content/en/documentation/sdks/python-unrecoverable-errors.md b/website/www/site/content/en/documentation/sdks/python-unrecoverable-errors.md index 38bd22f99179..0f5a838d7db9 100644 --- a/website/www/site/content/en/documentation/sdks/python-unrecoverable-errors.md +++ b/website/www/site/content/en/documentation/sdks/python-unrecoverable-errors.md @@ -16,7 +16,7 @@ See the License for the specific language governing permissions and limitations under the License. --> -# Unrecoverable Errors in Beam Python +# Unrecoverable Errors in Beam Python ## What is an Unrecoverable Error? From 5fee528b89fbe5a3b619caa3440fb3199e46d9a5 Mon Sep 17 00:00:00 2001 From: Jack McCluskey Date: Thu, 28 Sep 2023 17:04:12 -0400 Subject: [PATCH 10/10] remove anchor links (didn't render in the sidebar correctly) --- .../documentation/sdks/python-unrecoverable-errors.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/website/www/site/content/en/documentation/sdks/python-unrecoverable-errors.md b/website/www/site/content/en/documentation/sdks/python-unrecoverable-errors.md index 0f5a838d7db9..4e5d94ce8a8d 100644 --- a/website/www/site/content/en/documentation/sdks/python-unrecoverable-errors.md +++ b/website/www/site/content/en/documentation/sdks/python-unrecoverable-errors.md @@ -16,25 +16,25 @@ See the License for the specific language governing permissions and limitations under the License. --> -# Unrecoverable Errors in Beam Python +# Unrecoverable Errors in Beam Python -## What is an Unrecoverable Error? +## What is an Unrecoverable Error? An unrecoverable error is an issue at job start-up time that will prevent a job from ever running successfully, usually due to some kind of misconfiguration. Solving these issues when they occur is key to successfully running a Beam Python pipeline. -## Common Unrecoverable Errors +## Common Unrecoverable Errors -### Job Submission/Runtime Python Version Mismatch +### Job Submission/Runtime Python Version Mismatch If the Python version used for job submission does not match the Python version used to build the worker container, the job will not execute. Ensure that the Python version being used for job submission and the container Python version match. -### PIP Dependency Resolution Failures +### PIP Dependency Resolution Failures During worker start-up, dependencies are checked and installed in the worker container before accepting work. If a pipeline requires