[BEAM-10982] Add linux aarch64 wheel build support

[odidev]

Added linux aarch64 wheel build support.

---

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

• Choose reviewer(s) and mention them in a comment (R: @username).
• Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
• Update CHANGES.md with noteworthy changes.
• If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

ValidatesRunner compliance status (on master branch)

Lang	ULR	Dataflow	Flink	Samza	Spark	Twister2
Go	---			---		---
Java
Python	---			---		---
XLang				---		---

Examples testing status on various runners

Lang	ULR	Dataflow	Flink	Samza	Spark	Twister2
Go	---	---	---	---	---	---	---
Java	---		---	---	---	---	---
Python	---	---	---	---	---	---	---
XLang	---	---	---	---	---	---	---

Post-Commit SDK/Transform Integration Tests Status (on master branch)

Go	Java	Python

Pre-Commit Tests Status (on master branch)

---	Java	Python	Go	Website	Whitespace	Typescript
Non-portable
Portable	---			---	---	---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

[iemejia]

👏

[tvalentyn]

We don't have test suites for Py39 yet, however this may give impression that we already support it; it would add py39 wheels when we add Py3.9 test suites and officially support it.

[odidev]

Ok, I have removed py39.

[tvalentyn]

Thanks for this change, @odidev . Trying it out here: https://github.com/apache/beam/runs/2778415390?check_suite_focus=true

[tvalentyn]

Thanks!

[iemejia]

This is excellent news ! So it seems with the recent dependency ugprades we are now fully aarch64. I can validate that on AWS Graviton if somewone here explains me how to install the equivalent of 'SNAPSHOT' or 'dev' dependencies on beam python

[tvalentyn]

Try https://cwiki.apache.org/confluence/display/BEAM/Python+Tips#PythonTips-HowtoInstallanUnreleasedPythonSDKwithoutBuildingIt - and pip install the whl file.
Ideally, we should add github action test suite for this arch as well if we can, similar to current github action tests for mac, windows, etc.

[iemejia]

Thanks @tvalentyn I think there is a way to test this via emulation + docker images I saw this somewhere but I cannot remember now. I remember only the project they used https://github.com/dockcross/dockcross

[iemejia]

I saw some error messages because of wheels not available on other dependencies.

Processing ./apache_beam-2.32.0.dev0-cp38-cp38-manylinux2014_aarch64.whl
Requirement already satisfied: grpcio<2,>=1.29.0 in ./.virtualenvs/beamtest2/lib/python3.8/site-packages (from apache-beam==2.32.0.dev0) (1.38.0)
Requirement already satisfied: httplib2<0.20.0,>=0.8 in ./.virtualenvs/beamtest2/lib/python3.8/site-packages (from apache-beam==2.32.0.dev0) (0.19.1)
Collecting avro-python3!=1.9.2,<1.10.0,>=1.8.1
  Using cached avro-python3-1.9.2.1.tar.gz (37 kB)
Requirement already satisfied: numpy<1.21.0,>=1.14.3 in ./.virtualenvs/beamtest2/lib/python3.8/site-packages (from apache-beam==2.32.0.dev0) (1.20.3)
Collecting pyarrow<5.0.0,>=0.15.1
  Using cached pyarrow-4.0.1-cp38-cp38-manylinux2014_aarch64.whl (20.3 MB)
Requirement already satisfied: pydot<2,>=1.2.0 in ./.virtualenvs/beamtest2/lib/python3.8/site-packages (from apache-beam==2.32.0.dev0) (1.4.2)
Collecting protobuf<4,>=3.12.2
  Using cached protobuf-3.17.3-cp38-cp38-manylinux2014_aarch64.whl (923 kB)
Requirement already satisfied: python-dateutil<3,>=2.8.0 in ./.virtualenvs/beamtest2/lib/python3.8/site-packages (from apache-beam==2.32.0.dev0) (2.8.1)
Requirement already satisfied: pytz>=2018.3 in ./.virtualenvs/beamtest2/lib/python3.8/site-packages (from apache-beam==2.32.0.dev0) (2021.1)
Collecting hdfs<3.0.0,>=2.1.0
  Using cached hdfs-2.6.0-py3-none-any.whl (33 kB)
Requirement already satisfied: oauth2client<5,>=2.0.1 in ./.virtualenvs/beamtest2/lib/python3.8/site-packages (from apache-beam==2.32.0.dev0) (4.1.3)
Collecting crcmod<2.0,>=1.7
  Using cached crcmod-1.7.tar.gz (89 kB)
Requirement already satisfied: future<1.0.0,>=0.18.2 in ./.virtualenvs/beamtest2/lib/python3.8/site-packages (from apache-beam==2.32.0.dev0) (0.18.2)
Collecting requests<3.0.0,>=2.24.0
  Using cached requests-2.25.1-py2.py3-none-any.whl (61 kB)
Requirement already satisfied: fastavro<2,>=0.21.4 in ./.virtualenvs/beamtest2/lib/python3.8/site-packages (from apache-beam==2.32.0.dev0) (1.4.1)
Requirement already satisfied: pymongo<4.0.0,>=3.8.0 in ./.virtualenvs/beamtest2/lib/python3.8/site-packages (from apache-beam==2.32.0.dev0) (3.11.4)
Requirement already satisfied: dill<0.3.2,>=0.3.1.1 in ./.virtualenvs/beamtest2/lib/python3.8/site-packages (from apache-beam==2.32.0.dev0) (0.3.1.1)
Collecting typing-extensions<3.8.0,>=3.7.0
  Using cached typing_extensions-3.7.4.3-py3-none-any.whl (22 kB)
Requirement already satisfied: six>=1.5.2 in ./.virtualenvs/beamtest2/lib/python3.8/site-packages (from grpcio<2,>=1.29.0->apache-beam==2.32.0.dev0) (1.16.0)
Requirement already satisfied: pyparsing<3,>=2.4.2 in ./.virtualenvs/beamtest2/lib/python3.8/site-packages (from httplib2<0.20.0,>=0.8->apache-beam==2.32.0.dev0) (2.4.7)
Collecting docopt
  Using cached docopt-0.6.2.tar.gz (25 kB)
Requirement already satisfied: pyasn1>=0.1.7 in ./.virtualenvs/beamtest2/lib/python3.8/site-packages (from oauth2client<5,>=2.0.1->apache-beam==2.32.0.dev0) (0.4.8)
Requirement already satisfied: pyasn1-modules>=0.0.5 in ./.virtualenvs/beamtest2/lib/python3.8/site-packages (from oauth2client<5,>=2.0.1->apache-beam==2.32.0.dev0) (0.2.8)
Requirement already satisfied: rsa>=3.1.4 in ./.virtualenvs/beamtest2/lib/python3.8/site-packages (from oauth2client<5,>=2.0.1->apache-beam==2.32.0.dev0) (4.7.2)
Collecting chardet<5,>=3.0.2
  Using cached chardet-4.0.0-py2.py3-none-any.whl (178 kB)
Collecting urllib3<1.27,>=1.21.1
  Using cached urllib3-1.26.5-py2.py3-none-any.whl (138 kB)
Collecting idna<3,>=2.5
  Using cached idna-2.10-py2.py3-none-any.whl (58 kB)
Collecting certifi>=2017.4.17
  Using cached certifi-2021.5.30-py2.py3-none-any.whl (145 kB)
Building wheels for collected packages: avro-python3, crcmod, docopt
  Building wheel for avro-python3 (setup.py): started
  Building wheel for avro-python3 (setup.py): finished with status 'error'
  Running setup.py clean for avro-python3
  Building wheel for crcmod (setup.py): started
  Building wheel for crcmod (setup.py): finished with status 'error'
  Running setup.py clean for crcmod
  Building wheel for docopt (setup.py): started
  Building wheel for docopt (setup.py): finished with status 'error'
  Running setup.py clean for docopt
Failed to build avro-python3 crcmod docopt
Installing collected packages: avro-python3, pyarrow, protobuf, docopt, chardet, urllib3, idna, certifi, requests, hdfs, crcmod, typing-extensions, apache-beam
    Running setup.py install for avro-python3: started
    Running setup.py install for avro-python3: finished with status 'error'

Curious I thought Beam did not use Apache Avro after the move to Python 3 or is it there only for backwards compatibility for users?
Also in case that is the reason we should move the avro dependency to be the 'new' avro one, for ref https://pypi.org/project/avro-python3/ is deprecated now in favor of https://pypi.org/project/avro/

[tvalentyn]

We switched to fastavro as a default option, but there is a branch allowing to use avro-python3. We could delete that branch, I am not aware of any issues with fastavro on Python3.