Upgrade the Agent build to embed Python 3.8.1 #4464

remeh · 2019-11-20T17:29:11Z

What does this PR do?

Upgrade the Agent build to embed Python 3.8.1

I had to upgrade:

pip to 19.2.3 due to this: Don't append "m" ABI flag in Python 3.8 pypa/wheel#303
wheel to 0.34.1 to properly supports ARM builds
pip-tools to support the new pip version
in integrations-core, pywin32 to 227 see Upgrade pywin32 to 227 integrations-core#5036
in integrations-core, cryptography to 2.8 see Upgrade cryptography to 2.8 integrations-core#5047

Motivation

Provide the Agent with the latest Python 3 available.

We've a bug while building python3 due to this: - pypa/wheel#303 We need pip3 >= `19.2.3`.

…nto remeh/python-3.8

…-tools. pip >= 19.3 support landed in pip-tools 4.2.0

truthbk

I believe python 3.8 introduced some changes to the C-API (should def be non-breaking), but let's make sure the RTLoader is behaving as expected! :)

Also, though the pip-tools and setuptools version bumps are welcome, can we do those in a separate PR? Or are those changes required by python3.8?

remeh · 2019-11-21T08:31:56Z

@truthbk Yep it's needed for the build with Python 3.8 to succeed, the reason is linked in the PR description, I'm testing the build this morning :)

truthbk

Let's roll this out for ~~6.16~~ 6.17 (my bad!)

ruthnaebeck

Docs review

releasenotes/notes/python-3-8-c66cde69f972eb1e.yaml

docs/dev/agent_dev_env.md

Co-Authored-By: ruthnaebeck <[email protected]>

omnibus/package-scripts/agent/prerm

olivielpeau

A few comments, mostly nits except the one on the package scripts.

Overall LGTM! Let's make sure we start testing this as soon as 7.17.0 is out

omnibus/package-scripts/agent/prerm

releasenotes/notes/python-3-8-c66cde69f972eb1e.yaml

.gitlab-ci.yml

tasks/agent.py

…bpf part

….id()

This reverts commit 427eba3.

codecov · 2020-01-31T11:04:05Z

Codecov Report

Merging #4464 into master will increase coverage by 0.17%.
The diff coverage is 65.93%.

@@            Coverage Diff             @@
##           master    #4464      +/-   ##
==========================================
+ Coverage   52.35%   52.52%   +0.17%     
==========================================
  Files         709      710       +1     
  Lines       51004    51247     +243     
==========================================
+ Hits        26702    26919     +217     
- Misses      22722    22752      +30     
+ Partials     1580     1576       -4

Flag	Coverage Δ
#linux	`53.61% <65.93%> (+0.03%)`	⬆️
#windows	`52.76% <ø> (-0.07%)`	⬇️

Impacted Files	Coverage Δ
pkg/process/util/netns.go	`0% <0%> (ø)`
pkg/util/kubernetes/apiserver/apiserver.go	`10.86% <0%> (-0.1%)`	⬇️
pkg/util/kubernetes/apiserver/controllers.go	`0% <0%> (ø)`	⬆️
pkg/util/kubernetes/apiserver/hpa_controller.go	`64.03% <100%> (ø)`	⬆️
pkg/util/kubernetes/apiserver/controller_util.go	`60.31% <100%> (+0.31%)`	⬆️
pkg/autodiscovery/listeners/common.go	`79.16% <57.14%> (-2.66%)`	⬇️
pkg/util/kubernetes/apiserver/wpa_controller.go	`72.45% <91.22%> (+8.1%)`	⬆️
pkg/collector/runner/runner.go	`64.8% <0%> (-3.84%)`	⬇️
pkg/ebpf/config.go	`40.38% <0%> (-1.29%)`	⬇️
pkg/logs/auditor/auditor.go	`71.12% <0%> (-0.71%)`	⬇️
... and 7 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 427eba3...4fdc101. Read the comment docs.

olivielpeau

Thanks! One last nit on the release note, feel free to merge once addressed

releasenotes/notes/python-3-8-c66cde69f972eb1e.yaml

tasks/agent.py

Co-Authored-By: Olivier Vielpeau <[email protected]>

olivielpeau · 2020-02-03T14:00:52Z

releasenotes/notes/python-3-8-c66cde69f972eb1e.yaml

+  - |
+    Upgrade embedded Python 3 to 3.8.1. Link to Python 3.8 changelog: https://docs.python.org/3/whatsnew/3.8.html
+
+    Note that the Python 2 version shipped in Agent v6 continues to be version 2.7.17 (unchanged). Agent v7 will ship Python 3.8.1


Let's be either super explicit (see below):

Suggested change

Note that the Python 2 version shipped in Agent v6 continues to be version 2.7.17 (unchanged). Agent v7 will ship Python 3.8.1

Note that the Python 2 version shipped in Agent v6 continues to be version 2.7.17 (unchanged). Non-containerized packages of Agent v6 also ship Python 3 at version 3.8.1, and Agent v7 only ships Python 3.8.1.

or we can simply remove the Agent v7 will ship Python 3.8.1 sentence, since I feel it can be interpreted as only Agent v7 ships Python 3.8.1. Up to you

Fair enough!

remeh and others added 9 commits November 13, 2019 14:44

Upgrade to Python 3.8

a518ae7

Update Python 3.8 hash (Linux).

e86a831

Update python3 shasum for windows.

8e6ff1a

Merge branch 'master' into remeh/python-3.8

c79cdab

Patch must be applied to Python 3.8 now.

2952536

Update sha256 to reflect new zips

b354bdf

omnibus: update pip3.

49406d1

We've a bug while building python3 due to this: - pypa/wheel#303 We need pip3 >= `19.2.3`.

Merge branch 'remeh/python-3.8' of github.com:DataDog/datadog-agent i…

a829a05

…nto remeh/python-3.8

omnibus: in order to use a more recent pip, we need a more recent pip…

aa0f231

…-tools. pip >= 19.3 support landed in pip-tools 4.2.0

remeh added [deprecated] team/agent-core Deprecated. Use metrics-logs / shared-components labels instead.. changelog/no-changelog labels Nov 20, 2019

remeh requested review from a team as code owners November 20, 2019 17:29

remeh requested a review from therve November 20, 2019 17:29

Release note entry.

fd75ce0

remeh removed the changelog/no-changelog label Nov 20, 2019

remeh added this to the 6.16.0 milestone Nov 20, 2019

truthbk reviewed Nov 20, 2019

View reviewed changes

remeh modified the milestones: 6.16.0, Triage Nov 21, 2019

truthbk previously approved these changes Dec 9, 2019

View reviewed changes

olivielpeau modified the milestones: Triage, 7.17.0 Dec 9, 2019

ruthnaebeck suggested changes Jan 2, 2020

View reviewed changes

Apply suggestions from code review

776b240

Co-Authored-By: ruthnaebeck <[email protected]>

remeh dismissed truthbk’s stale review via 776b240 January 8, 2020 09:20

Update releasenotes/notes/python-3-8-c66cde69f972eb1e.yaml

47a07c8

Co-Authored-By: ruthnaebeck <[email protected]>

gitlab: use new builders images.

b730297

remeh dismissed ofek’s stale review via b730297 January 30, 2020 10:53

gitlab: use builders images merged in master.

0d841b7

ofek previously approved these changes Jan 30, 2020

View reviewed changes

KSerrania previously approved these changes Jan 30, 2020

View reviewed changes

omnibus/package-scripts/agent/prerm Outdated Show resolved Hide resolved

olivielpeau reviewed Jan 30, 2020

View reviewed changes

omnibus/package-scripts/agent/prerm Outdated Show resolved Hide resolved

releasenotes/notes/python-3-8-c66cde69f972eb1e.yaml Outdated Show resolved Hide resolved

.gitlab-ci.yml Outdated Show resolved Hide resolved

tasks/agent.py Show resolved Hide resolved

remeh added 2 commits January 31, 2020 10:13

gitlab-ci: use the requirements.txt while installing distro for the e…

e3e0a80

…bpf part

tasks/build_tags: proper entry for redhat since we switched to distro…

0ac14bb

….id()

remeh dismissed stale reviews from KSerrania and ofek via 0ac14bb January 31, 2020 09:46

remeh added 3 commits January 31, 2020 10:53

packages-scripts: also remove python 3.8 in preinst

de4142d

Revert "omnibus: properly call remove_downlaoder_data for centos."

8dc2f7c

This reverts commit 427eba3.

releasesnotes: update python 3.8.1 release note entry.

4fdc101

olivielpeau previously approved these changes Jan 31, 2020

View reviewed changes

releasenotes/notes/python-3-8-c66cde69f972eb1e.yaml Outdated Show resolved Hide resolved

tasks/agent.py Show resolved Hide resolved

This was referenced Feb 1, 2020

Add meta command to upgrade the Python version of all test environments DataDog/integrations-core#5616

Merged

Python 2.7 Docker image SyntaxError: invalid syntax DataDog/integrations-extras#530

Closed

Update releasenotes/notes/python-3-8-c66cde69f972eb1e.yaml

71f51ad

Co-Authored-By: Olivier Vielpeau <[email protected]>

remeh dismissed olivielpeau’s stale review via 71f51ad February 3, 2020 09:13

releasenotes: adapt python 3.8.1 release note entry.

df362eb

olivielpeau previously approved these changes Feb 3, 2020

View reviewed changes

olivielpeau reviewed Feb 3, 2020

View reviewed changes

releasenotes: remove mention of a7 py version.

ca1e959

remeh dismissed olivielpeau’s stale review via ca1e959 February 3, 2020 15:51

remeh merged commit a3afc20 into master Feb 3, 2020

remeh deleted the remeh/python-3.8 branch February 3, 2020 16:43

ofek mentioned this pull request Feb 3, 2020

Upgrade Python to 3.8 in test environments DataDog/integrations-core#5626

Merged

nmuesch mentioned this pull request Feb 5, 2020

Bump the version of python in the setup scripts DataDog/datadog-agent-boshrelease#112

Merged

5 tasks

L3n41c mentioned this pull request Feb 11, 2020

Use python3 DataDog/datadog-agent-buildimages#27

Merged

mx-psi mentioned this pull request Aug 10, 2020

[Omnibus] Fix incorrect copies and moves #6183

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Upgrade the Agent build to embed Python 3.8.1 #4464

Upgrade the Agent build to embed Python 3.8.1 #4464

remeh commented Nov 20, 2019 •

edited

Loading

truthbk left a comment

remeh commented Nov 21, 2019 •

edited

Loading

truthbk left a comment •

edited

Loading

ruthnaebeck left a comment

olivielpeau left a comment

codecov bot commented Jan 31, 2020 •

edited

Loading

olivielpeau left a comment

olivielpeau Feb 3, 2020

remeh Feb 3, 2020

	Note that the Python 2 version shipped in Agent v6 continues to be version 2.7.17 (unchanged). Agent v7 will ship Python 3.8.1
	Note that the Python 2 version shipped in Agent v6 continues to be version 2.7.17 (unchanged). Non-containerized packages of Agent v6 also ship Python 3 at version 3.8.1, and Agent v7 only ships Python 3.8.1.

Upgrade the Agent build to embed Python 3.8.1 #4464

Upgrade the Agent build to embed Python 3.8.1 #4464

Conversation

remeh commented Nov 20, 2019 • edited Loading

What does this PR do?

Motivation

truthbk left a comment

Choose a reason for hiding this comment

remeh commented Nov 21, 2019 • edited Loading

truthbk left a comment • edited Loading

Choose a reason for hiding this comment

ruthnaebeck left a comment

Choose a reason for hiding this comment

olivielpeau left a comment

Choose a reason for hiding this comment

codecov bot commented Jan 31, 2020 • edited Loading

Codecov Report

olivielpeau left a comment

Choose a reason for hiding this comment

olivielpeau Feb 3, 2020

Choose a reason for hiding this comment

remeh Feb 3, 2020

Choose a reason for hiding this comment

remeh commented Nov 20, 2019 •

edited

Loading

remeh commented Nov 21, 2019 •

edited

Loading

truthbk left a comment •

edited

Loading

codecov bot commented Jan 31, 2020 •

edited

Loading