Remove different build matrix selection for develop branches #1138

absurdfarce · 2023-03-07T21:40:44Z

Goal here is to use the smoke tests for as many build cases as we reasonably can

…ke to try and use the smoke tests for as much as we reasonably can.

absurdfarce · 2023-03-07T21:44:19Z

Right now the Jenkinsfile for the Python driver uses a different build matrix for feature branches named to match a PYTHON JIRA ticket. The build matrix in question doesn't really make sense for our plans going forward (it still includes Python 2.7 builds) and we do have the SMOKE build matrix which more naturally corresponds to the versions we'd like to support (and is already used as the default build matrix for most Jenkins builds).

Goal here is basically to remove the special-case handling for branches matching this naming pattern. Going forward we may want to remove the DEVELOP matrix all together, but honestly I'd expect something like that to be handled in a larger-scale refactoring of the various matrices defined in the Jenkinsfile.

For now, I just want to remove this special behaviour... I really want Jenkins feature branch builds to use the same matrix that will be used when we merge that feature branch into master.

absurdfarce · 2023-03-08T06:17:42Z

Worth mentioning here: the build for "develop branches" actually fails now due to an issue with something on 2.7:


[2023-03-07T20:45:29.706Z] + pip install -r test-requirements.txt
[2023-03-07T20:45:30.008Z] DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support pip 21.0 will remove support for this functionality.
[2023-03-07T20:45:30.008Z] Ignoring twisted: markers 'python_version >= "3.5"' don't match your environment
[2023-03-07T20:45:30.008Z] Ignoring futurist: markers 'python_version >= "3.7"' don't match your environment
[2023-03-07T20:45:30.008Z] Ignoring asynctest: markers 'python_version >= "3.5"' don't match your environment
...
[2023-03-07T20:45:31.480Z] Collecting pure-sasl
[2023-03-07T20:45:31.480Z]   Downloading pure_sasl-0.6.2-py2-none-any.whl (11 kB)
[2023-03-07T20:45:31.480Z] Collecting twisted[tls]==19.2.1
[2023-03-07T20:45:31.480Z]   Downloading Twisted-19.2.1.tar.bz2 (3.1 MB)
[2023-03-07T20:45:33.601Z]     ERROR: Command errored out with exit status 1:
[2023-03-07T20:45:33.601Z]      command: /home/jenkins/.pyenv/versions/2.7.18/bin/python2.7 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-uGalC0/twisted/setup.py'"'"'; __file__='"'"'/tmp/pip-install-uGalC0/twisted/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-JUjBaC
[2023-03-07T20:45:33.601Z]          cwd: /tmp/pip-install-uGalC0/twisted/
[2023-03-07T20:45:33.601Z]     Complete output (25 lines):
[2023-03-07T20:45:33.601Z]     WARNING: The wheel package is not available.
[2023-03-07T20:45:33.601Z]     DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support pip 21.0 will remove support for this functionality.
[2023-03-07T20:45:33.601Z]     Traceback (most recent call last):
[2023-03-07T20:45:33.601Z]       File "<string>", line 1, in <module>
[2023-03-07T20:45:33.601Z]       File "/tmp/pip-install-uGalC0/twisted/setup.py", line 20, in <module>
[2023-03-07T20:45:33.601Z]         setuptools.setup(**_setup["getSetupArgs"]())
[2023-03-07T20:45:33.601Z]       File "/home/jenkins/.pyenv/versions/2.7.18/lib/python2.7/site-packages/setuptools/__init__.py", line 162, in setup
[2023-03-07T20:45:33.601Z]         return distutils.core.setup(**attrs)
[2023-03-07T20:45:33.601Z]       File "/home/jenkins/.pyenv/versions/2.7.18/lib/python2.7/distutils/core.py", line 111, in setup
[2023-03-07T20:45:33.601Z]         _setup_distribution = dist = klass(attrs)
[2023-03-07T20:45:33.601Z]       File "/home/jenkins/.pyenv/versions/2.7.18/lib/python2.7/site-packages/setuptools/dist.py", line 448, in __init__
[2023-03-07T20:45:33.601Z]         k: v for k, v in attrs.items()
[2023-03-07T20:45:33.601Z]       File "/home/jenkins/.pyenv/versions/2.7.18/lib/python2.7/distutils/dist.py", line 287, in __init__
[2023-03-07T20:45:33.601Z]         self.finalize_options()
[2023-03-07T20:45:33.601Z]       File "/home/jenkins/.pyenv/versions/2.7.18/lib/python2.7/site-packages/setuptools/dist.py", line 740, in finalize_options
[2023-03-07T20:45:33.601Z]         ep(self)
[2023-03-07T20:45:33.601Z]       File "/home/jenkins/.pyenv/versions/2.7.18/lib/python2.7/site-packages/setuptools/dist.py", line 747, in _finalize_setup_keywords
[2023-03-07T20:45:33.601Z]         ep.load()(self, ep.name, value)
[2023-03-07T20:45:33.601Z]       File "/home/jenkins/.pyenv/versions/2.7.18/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2443, in load
[2023-03-07T20:45:33.601Z]         return self.resolve()
[2023-03-07T20:45:33.601Z]       File "/home/jenkins/.pyenv/versions/2.7.18/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2449, in resolve
[2023-03-07T20:45:33.601Z]         module = __import__(self.module_name, fromlist=['__name__'], level=0)
[2023-03-07T20:45:33.601Z]       File "/tmp/pip-install-uGalC0/twisted/.eggs/incremental-22.10.0-py2.7.egg/incremental/__init__.py", line 14, in <module>
[2023-03-07T20:45:33.601Z]         from typing import TYPE_CHECKING, Any, TypeVar, Union, Optional, Dict
[2023-03-07T20:45:33.601Z]     ImportError: No module named typing
[2023-03-07T20:45:33.601Z]     ----------------------------------------
[2023-03-07T20:45:33.601Z] ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

Looks to be a transitive dependency issue with something not supporting 2.7.x anymore (typing is a Python3 module). I've dug into this enough to identify the above but that's about it; given that 2.7.x support will be going away there's no obvious reason to investigate this any further. And since that's the case there's no obvious reason to continue to execute anything on Jenkins against 2.7.x.

joao-r-reis · 2023-03-09T11:18:07Z

Jenkinsfile

-    if (env.BRANCH_NAME.contains("long")) {
-      profile = "FULL"
-    }
+  if (env.BRANCH_NAME.contains("long")) {


Should we remove this if as well?

That's an interesting question.

There's going to be some changes to all these build matrices to support our new version support policy for the Python driver; from the new release on a given driver release will only support Python versions that aren't out-of-support at the time of release. So 2.7, 3.5 and 3.6 will be removed from all five build matrices, for instance. There's also a legit question for me about whether we really need five distinct build matrices... that seems like a lot.

But as I thought about it you're actually asking a different question; if we're already moving away from branch name correlation to a build matrix why not sever that tie completely? I think this is prolly right; I don't really like the idea that a branch name magically correlates to a specific build matrix. Seems better to maybe run other matrices on regular schedules and support ad-hoc runs if somebody really wants to run something now.

Whaddya think?

if we're already moving away from branch name correlation to a build matrix why not sever that tie completely

Exactly, if a branch gets named "something-long" and the build starts failing we may take a lot of time to realize it's using a completely different build because of this if, I'd just remove it.

Sold. I was originally avoiding making this PR too big under the assumption that the overall build matrix would have to be revisited once we change supported versions of the Python runtime but as discussed above that concern isn't explicitly connected to what you're advocating here @joao-r-reis .

I'll make the change.

absurdfarce · 2023-03-09T22:00:33Z

Jenkinsfile

-    "SERVER": ['3.11', '4.0', 'dse-6.8.30'],
-    "RUNTIME": ['3.7.7', '3.8.3'],
+    "SERVER": DEFAULT_CASSANDRA.takeRight(2) + DEFAULT_DSE.takeRight(1),
+    "RUNTIME": DEFAULT_RUNTIME.takeRight(2),
    "CYTHON": ["False"]


Just reducing some redundancy while we're here + making things a bit more declarative

absurdfarce · 2023-03-22T19:32:28Z

Jenkinsfile

@@ -33,30 +33,34 @@ slack = new Slack()
 //
 // Smoke tests are CI-friendly test configuration.  Currently-supported Python version + modern C*/DSE instances.
 // We also avoid cython since it's tested as part of the nightlies.
+DEFAULT_CASSANDRA = ['2.1', '2.2', '3.0', '3.11', '4.0']
+DEFAULT_DSE = ['dse-5.0.15', 'dse-5.1.35', 'dse-6.0.18', 'dse-6.7.17', 'dse-6.8.30']
+DEFAULT_RUNTIME = ['2.7.18', '3.5.9', '3.6.10', '3.7.7', '3.8.3']


It's a little strange to keep 2.7.x + 3.5.x + 3.6.x around here. My rationale is that this PR is really about removing the special handling of "develop" branches + some normalization for how versions are specified. Actual changes in versions we test against will be done in a PR explicitly focused on that role (and should be accompanied by corresponding changes to the runners).

This is a great way to keep these versions in a single place, TIL!

absurdfarce · 2023-03-22T19:32:47Z

@joao-r-reis Quick ping; are you good with this change as it stands?

…sync_with_upstream_3.29.1 version 3.28.0 * tag '3.28.0' of https://github.com/datastax/python-driver: Release 3.28.0: changelog & version PYTHON-1352 Add vector type, codec + support for parsing CQL type (datastax#1161) Update docs.yaml to point to most recent 3.27.0 docs changes CONN-38 Notes for 3.27.0 on PYTHON-1350 (datastax#1166) PYTHON-1356 Create session-specific protocol handlers to contain session-specific CLE policies (datastax#1165) PYTHON-1350 Store IV along with encrypted text when using column-level encryption (datastax#1160) PYTHON-1351 Convert cryptography to an optional dependency (datastax#1164) Jenkinsfile cleanup (datastax#1163) PYTHON-1343 Use Cython for smoke builds (datastax#1162) Don't fail when inserting UDTs with prepared queries with some missing fields (datastax#1151) Revert "remove unnecessary import __future__ (datastax#1156)" docs: convert print statement to function in docs (datastax#1157) remove unnecessary import __future__ (datastax#1156) Update docs.yaml to include recent fixes to CLE docs Fix for rendering of code blocks in CLE documentation (datastax#1159) DOC-3278 Update comment for retry policy (datastax#1158) DOC-2813 (datastax#1145) Remove different build matrix selection for develop branches (datastax#1138)

…x#1138)

Remove different build matrix selection for develop branches; we'd li…

395af1e

…ke to try and use the smoke tests for as much as we reasonably can.

absurdfarce requested review from joao-r-reis and elliottrusso March 7, 2023 21:40

absurdfarce added 2 commits March 9, 2023 02:32

Avoid setting profile value based on the name of a feature branch

4a3118c

A few small Jenkinsfile cleanups

3e572da

joao-r-reis reviewed Mar 9, 2023

View reviewed changes

absurdfarce commented Mar 9, 2023

View reviewed changes

absurdfarce requested a review from joao-r-reis March 13, 2023 15:24

elliottrusso approved these changes Mar 14, 2023

View reviewed changes

absurdfarce commented Mar 22, 2023

View reviewed changes

joao-r-reis approved these changes May 3, 2023

View reviewed changes

Merge branch 'master' into one_matrix_to_rule_them_all

024d57c

absurdfarce merged commit 078e2f6 into master May 9, 2023

absurdfarce deleted the one_matrix_to_rule_them_all branch May 9, 2023 14:47

dkropachev pushed a commit to dkropachev/python-driver that referenced this pull request Aug 9, 2024

Remove different build matrix selection for develop branches (datasta…

355ec1d

…x#1138)

dkropachev pushed a commit to dkropachev/python-driver that referenced this pull request Aug 9, 2024

Remove different build matrix selection for develop branches (datasta…

70b1e8a

…x#1138)

dkropachev pushed a commit to dkropachev/python-driver that referenced this pull request Aug 9, 2024

Remove different build matrix selection for develop branches (datasta…

d862482

…x#1138)

dkropachev pushed a commit to dkropachev/python-driver that referenced this pull request Aug 9, 2024

Remove different build matrix selection for develop branches (datasta…

0f35c9a

…x#1138)

dkropachev pushed a commit to dkropachev/python-driver that referenced this pull request Aug 9, 2024

Remove different build matrix selection for develop branches (datasta…

8a4ce83

…x#1138)

dkropachev pushed a commit to dkropachev/python-driver that referenced this pull request Aug 15, 2024

Remove different build matrix selection for develop branches (datasta…

436e09d

…x#1138)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Remove different build matrix selection for develop branches #1138

Remove different build matrix selection for develop branches #1138

absurdfarce commented Mar 7, 2023

absurdfarce commented Mar 7, 2023

absurdfarce commented Mar 8, 2023

joao-r-reis Mar 9, 2023

absurdfarce Mar 9, 2023

joao-r-reis Mar 9, 2023

absurdfarce Mar 9, 2023

absurdfarce Mar 9, 2023

absurdfarce Mar 22, 2023

joao-r-reis May 3, 2023

absurdfarce commented Mar 22, 2023

Remove different build matrix selection for develop branches #1138

Remove different build matrix selection for develop branches #1138

Conversation

absurdfarce commented Mar 7, 2023

absurdfarce commented Mar 7, 2023

absurdfarce commented Mar 8, 2023

joao-r-reis Mar 9, 2023

Choose a reason for hiding this comment

absurdfarce Mar 9, 2023

Choose a reason for hiding this comment

joao-r-reis Mar 9, 2023

Choose a reason for hiding this comment

absurdfarce Mar 9, 2023

Choose a reason for hiding this comment

absurdfarce Mar 9, 2023

Choose a reason for hiding this comment

absurdfarce Mar 22, 2023

Choose a reason for hiding this comment

joao-r-reis May 3, 2023

Choose a reason for hiding this comment

absurdfarce commented Mar 22, 2023