Failing tests for python SDK on ppc64le.

[vinodk99]

What happened?

Facing test issues for Python SDK

tox -c tox.ini run -e py311

.pkg: _optional_hooks> python /usr/local/lib/python3.11/site-packages/pyproject_api/_backend.py True setuptools.build_meta legacy
.pkg: get_requires_for_build_sdist> python /usr/local/lib/python3.11/site-packages/pyproject_api/_backend.py True setuptools.build_meta legacy
.pkg: get_requires_for_build_wheel> python /usr/local/lib/python3.11/site-packages/pyproject_api/_backend.py True setuptools.build_meta legacy
.pkg: prepare_metadata_for_build_wheel> python /usr/local/lib/python3.11/site-packages/pyproject_api/_backend.py True setuptools.build_meta legacy
.pkg: build_sdist> python /usr/local/lib/python3.11/site-packages/pyproject_api/_backend.py True setuptools.build_meta legacy
py311: install_package_deps> target/.tox/py311/bin/python /script/beam/sdks/python/target/.tox/py311/bin/pip install --retries 10 --pre 'cloudpickle~=2.2.1' 'crcmod<2.0,>=1.7' 'cryptography>=41.0.2' 'dill<0.3.2,>=0.3.1.1' 'fastavro<2,>=0.23.6' 'fasteners<1.0,>=0.3' 'freezegun>=0.3.12' grpcio==1.51.3 'hdfs<3.0.0,>=2.1.0' 'httplib2<0.23.0,>=0.8' 'hypothesis<=7.0.0,>5.0.0' 'joblib>=1.0.1' 'js2py<1,>=0.74' 'mock<6.0.0,>=1.0.1' 'numpy<1.25.0,>=1.14.3' 'objsize<0.7.0,>=0.6.1' 'orjson<4,>=3.9.7' 'packaging>=22.0' 'pandas!=1.5.0,!=1.5.1,<1.6,>=1.4.3; python_version >= "3.8"' 'pandas<2.0.0' 'parameterized<0.10.0,>=0.7.1' 'proto-plus<2,>=1.7.1' 'protobuf!=4.0.,!=4.21.,!=4.22.0,!=4.23.*,!=4.24.0,!=4.24.1,!=4.24.2,<4.25.0,>=3.20.3' 'psycopg2-binary<3.0.0,>=2.8.5' pyarrow==12.0.0 'pydot<2,>=1.2.0' 'pyhamcrest!=1.10.0,<3.0.0,>=1.9' 'pymongo<5.0.0,>=3.8.0' 'pytest-timeout<3,>=2.1.0' 'pytest-xdist<4,>=2.5.0' 'pytest<8.0,>=7.1.2' 'python-dateutil<3,>=2.8.0' 'pytz>=2018.3' 'pyyaml<7.0.0,>=3.12' 'regex>=2020.6.8' 'requests-mock<2.0,>=1.7' 'requests<3.0.0,>=2.24.0' 'scikit-learn>=0.20.0' 'sqlalchemy<2.0,>=1.3' 'tenacity<9,>=8.0.0' 'testcontainers[mysql]<4.0.0,>=3.0.3' 'typing-extensions>=3.7.0' 'zstandard<1,>=0.18.0'
py311: install_package> target/.tox/py311/bin/python /script/beam/sdks/python/target/.tox/py311/bin/pip install --retries 10 --pre --force-reinstall --no-deps /script/beam/sdks/python/target/.tox/.tmp/package/5/apache-beam-2.51.0.tar.gz
py311: commands_pre[0]> python --version
Python 3.11.2
py311: commands_pre[1]> pip --version
pip 23.2.1 from /script/beam/sdks/python/target/.tox/py311/lib/python3.11/site-packages/pip (python 3.11)
py311: commands_pre[2]> pip check
No broken requirements found.

py311: commands_pre[3]> bash /script/beam/sdks/python/scripts/run_tox_cleanup.sh
py311: commands[0]> python apache_beam/examples/complete/autocomplete_test.py
Traceback (most recent call last):
File "/script/beam/sdks/python/apache_beam/examples/complete/autocomplete_test.py", line 26, in
import apache_beam as beam
File "/script/beam/sdks/python/target/.tox/py311/lib/python3.11/site-packages/apache_beam/init.py", line 88, in
from apache_beam import io
File "/script/beam/sdks/python/target/.tox/py311/lib/python3.11/site-packages/apache_beam/io/init.py", line 28, in
from apache_beam.io.parquetio import *
File "/script/beam/sdks/python/target/.tox/py311/lib/python3.11/site-packages/apache_beam/io/parquetio.py", line 679, in
class _ParquetSink(filebasedsink.FileBasedSink):
File "/script/beam/sdks/python/target/.tox/py311/lib/python3.11/site-packages/apache_beam/io/parquetio.py", line 733, in _ParquetSink
def write_record(self, writer, table: pa.Table):
^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'Table'
py311: exit 1 (1.31 seconds) /script/beam/sdks/python> python apache_beam/examples/complete/autocomplete_test.py pid=102249
py311: commands_post[0]> bash /script/beam/sdks/python/scripts/run_tox_cleanup.sh
.pkg: _exit> python /usr/local/lib/python3.11/site-packages/pyproject_api/_backend.py True setuptools.build_meta legacy
py311: FAIL code 1 (71.34=setup[68.06]+cmd[0.00,0.40,1.51,0.03,1.31,0.03] seconds)
evaluation failed :( (71.54 seconds)

Issue Failure

Failure: Test is continually failing

Issue Priority

Priority: 2 (backlog / disabled test but we think the product is healthy)

Issue Components

• Component: Python SDK
• Component: Java SDK
• Component: Go SDK
• Component: Typescript SDK
• Component: IO connector
• Component: Beam YAML
• Component: Beam examples
• Component: Beam playground
• Component: Beam katas
• Component: Website
• Component: Spark Runner
• Component: Flink Runner
• Component: Samza Runner
• Component: Twister2 Runner
• Component: Hazelcast Jet Runner
• Component: Google Cloud Dataflow Runner

[AnandInguva]

The recent run of Python tests which includes the one mentioned above is passing - https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/7713/consoleFull. I don't see a failure in the recent runs too - https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/

Could this be a flake on your end? can you rerun the test again?