Migrate build repo code from using Python 3.7 to 3.9 version

[peterzhuamazon]

Migrate build repo code from using Python 3.7 to 3.9 version.

Python3.7 support will ends on 2023/06/23.
https://endoflife.date/python

Note: 3.9 Ends in 2 years and 5 months (05 Oct 2025)

We need to migrate the code to Python 3.9 for the highest stability possible.

Acceptance Criteria:

1. All the build code should be using 3.9
2. Moving from 3.7 to 3.9 in the build repo, updating the pipfile and lock file.
3. Test all the existing workflows with the new python version.
4. Update all the CI images to use Python 3.9.
5. Testing the compilation of the cpython with centos docker image.

Thanks.

[peterzhuamazon]

The version 3.11 is currently feature adding version.
It is not stable enough for the production use.
@bbarani I would recommend using version 3.10 if we really want to go that high, but in my personal experience the most stable one is 3.9 version after 3.8.

Thanks.

[rishabh6788]

[TRIAGE] This is pre-req for #1563.
We also need to upgrade pip from v21 to v23 to accommodate python upgrade.
Tagging @peterzhuamazon.

[peterzhuamazon]

It might benefit to change the default emulation buildx to cross compilation:

• https://www.docker.com/blog/faster-multi-platform-builds-dockerfile-cross-compilation-guide/

PRs:

• [Python 3.9 Upgrade] Docker CI Images update 3.7 to 3.9 (Part 1) #3634
• [Python 3.9 Upgrade] Change docker build jenkins workflow to use the v2 docker-builder images #3641
• [Python 3.9 Upgrade] Upgrade docker buildx 0.9.1 and fix Python compilation issues #3643
• [Python 3.9 Upgrade] Replace Python 3.9.17 to 3.9.7 to resolve compilation inconsistency on CentOS7 #3650
• [Python 3.9 Upgrade] OpenSearch-Build Repo Python Migration to 3.9 Version #3658
• [Python 3.9 Upgrade] Add the test docker images upgrades to Python 3.9 (Part 2) #3680
• [Python 3.9 Upgrade] Update macos and windows packer agent with Python 3.9 opensearch-ci#311
• [Python 3.9 Upgrade] Resolve python 3.9 and cmake installation on Windows opensearch-ci#312
• [Python 3.9 Upgrade] Lock Windows Python to 3.9.7 and cmake to 3.21.3 (Part 2) opensearch-ci#313
• [Python 3.9 Upgrade] Restore python39 back to latest version due to old json not supported on scoop for 3.9.7 anymore opensearch-ci#314
• [Python 3.9 Upgrade] Update Runner Ami IDs for Python 3.9 Upgrade opensearch-ci#316
• [Python 3.9 Upgrade] [NodeJS Upgrade] Add AL2023 agent and switch perftest and benchmarktest from AL2 to AL2023 with Python 3.9 opensearch-ci#317
• [Python 3.9 Upgrade] [NodeJS Upgrade] Switch the internal ami name from al2 to al2023 opensearch-ci#319
• [Python 3.9 Upgrade] [NodeJS Upgrade] Replace ami id for al2023 with agents in #319 opensearch-ci#320
• [Python 3.9 Upgrade] [NodeJS Upgrade] Update perftest and benchmark test runner to AL2023 #3711
• [Python 3.9 Upgrade] [NodeJS Upgrade] Update developer guide for Python 3.9 Upgrade and Node Version Upgrade #3714
• [Python 3.9 Upgrade] Update macos agent to Python 3.9 export and add M58xlarge gradle check runner opensearch-ci#323
• Switch validation workflow for docker to run on AL2023 and Gradle Check to new M58xlarge runners #3810
• Switch single host from AL2 to AL2023 opensearch-ci#332
• Add AL2023 Docker Host Runner and Make M58xlarge Gradle Check Host opensearch-ci#330
• Update promote artifact jenkins workflow to use Python3.9 image #3804

[peterzhuamazon]

Able to get the pip packages freeze version and didnt change behaviors on all the isort/flake8/mypy/pytest results.

% pipenv run isort --check .
Skipped 24 files

% pipenv run flake8 .

% pipenv run mypy .
Success: no issues found in 375 source files

Able to build/assemble artifacts for OS and OSD.

% curl https://localhost:9200/ -ku admin:admin
{
  "name" : "dev-dsk-zhujiaxi-2a-c8d595b4.us-west-2.amazon.com",
  "cluster_name" : "opensearch",
  "cluster_uuid" : "gxjxiKG0Qbu3mdKhHAQqMw",
  "version" : {
    "distribution" : "opensearch",
    "number" : "2.8.0",
    "build_type" : "tar",
    "build_hash" : "db90a415ff2fd428b4f7b3f800a51dc229287cb4",
    "build_date" : "2023-06-22T17:07:14.480104Z",
    "build_snapshot" : false,
    "lucene_version" : "9.6.0",
    "minimum_wire_compatibility_version" : "7.10.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "The OpenSearch Project: https://opensearch.org/"
}

{"overall":{"since":"2023-06-22T18:46:13.818Z","state":"green","title":"Green","nickname":"Looking good","icon":"success","uiColor":"secondary"}

Able to run integTest on OS and OSD:

2023-06-22 19:37:52 INFO     | geospatial           | with-security        | PASS  |
2023-06-22 19:37:52 INFO     | geospatial           | without-security     | PASS  |

2023-06-22 20:29:31 INFO     | alertingDashboards   | with-security        | PASS  |
2023-06-22 20:29:31 INFO     | alertingDashboards   | without-security     | PASS  |

[peterzhuamazon]

% pipenv run python --version
Python 3.9.7

Installing dependencies from Pipfile.lock (9f5fbd)...
To activate this project's virtualenv, run pipenv shell.
Alternatively, run a command inside the virtualenv with pipenv run.

% pipenv run pip freeze
atomicwrites==1.4.1
attrs==23.1.0
Cerberus==1.3.4
certifi==2023.5.7
cfgv==3.3.1
chardet==5.1.0
charset-normalizer==2.1.1
click==8.1.3
coverage==4.5.4
DataProperty==0.55.1
decorator==5.1.1
distlib==0.3.6
distro==1.8.0
exceptiongroup==1.1.1
filelock==3.12.2
flake8==3.9.2
identify==2.5.24
idna==3.4
importlib-metadata==4.12.0
iniconfig==2.0.0
isort==5.12.0
jproperties==2.1.1
maison==1.4.0
mbstrdecoder==1.1.3
mccabe==0.6.1
mypy==0.971
mypy-extensions==1.0.0
nodeenv==1.8.0
packaging==23.1
pathspec==0.11.1
pathvalidate==2.5.2
platformdirs==3.7.0
pluggy==1.2.0
pre-commit==2.15.0
psutil==5.9.5
py==1.11.0
pycodestyle==2.7.0
pydantic==1.10.9
pyflakes==2.3.1
pytablewriter==0.64.2
pytest==7.1.3
pytest-cov==2.10.1
python-dateutil==2.8.2
pytz==2023.3
PyYAML==5.4.1
requests==2.28.1
retry==0.9.2
ruamel.yaml==0.17.32
ruamel.yaml.clib==0.2.7
ruyaml==0.91.0
six==1.16.0
sortedcontainers==2.4.0
tabledata==1.3.1
tcolorpy==0.1.3
toml==0.10.2
tomli==2.0.1
typed-ast==1.5.4
typepy==1.3.1
types-PyYAML==5.4.12
types-requests==2.31.0.1
types-urllib3==1.26.25.13
typing_extensions==4.6.3
urllib3==1.26.16
validators==0.20.0
virtualenv==20.23.1
yamlfix==1.0.1
yamllint==1.27.1
zipp==3.8.1

[peterzhuamazon]

Also just fyi once we go to 3.9 all client release that uses this image release-centos7-clients-v3 needs to be upgraded to release-centos7-clients-v4.

[peterzhuamazon]

We are starting the Python 3.9 Upgrade now, as Jenkins instance will be unstable from now on.
Start time: 2023/07/05 4:29 PM PST.
ETA: 3hrs

An email to the team has been sent out last week.
A msg to the public slack channel was also sent.

Thanks.

[peterzhuamazon]

Test deployment staging:

+ ./build.sh manifests/2.9.0/opensearch-2.9.0.yml -d tar --component OpenSearch -p linux -a x64

Installing dependencies in . ...

Warning: the environment variable LANG is not set!

We recommend setting this in ~/.profile (or equivalent) for proper expected behavior.

Creating a virtualenv for this project...

Pipfile: /var/jenkins/workspace/Python39-test-distribution-build-opensearch-zhujiaxi@2/Pipfile

Using /usr/local/bin/python3.9 (3.9.7) to create virtualenv...

created virtual environment CPython3.9.7.final.0-64 in 483ms

  creator CPython3Posix(dest=/usr/share/opensearch/.local/share/virtualenvs/Python39-test-distribution-build-opensearc-QLML_HWD, clear=False, no_vcs_ignore=False, global=False)

  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/usr/share/opensearch/.local/share/virtualenv)

    added seed packages: pip==23.1.2, setuptools==67.8.0, wheel==0.40.0

  activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator

[peterzhuamazon]

Verified all build and assemble on all platforms are running well.

[peterzhuamazon]

All docker actions are running ok with no issues.

[peterzhuamazon]

Will send to production and check gradle check.

[gaiksaya]

@peterzhuamazon We need to update developer guide too to use python 3.9 https://github.com/opensearch-project/opensearch-build/blob/main/DEVELOPER_GUIDE.md#python-37

[peterzhuamazon]

Jenkins deploy success on production host and the migration deployment is complete.

[gaiksaya]

Adding meta task to update all 1-click release process: #3712

[bbarani]

@peterzhuamazon Can we close this issue now?

[peterzhuamazon]

Yes @bbarani we have just completed all the changes now, thanks.