Build and publish pydeephaven-ticking wheels (#5290)

Refactors the build process for py-client-ticking to individual tasks, and greatly reduces the amount of disk space needed for the build process. The big change is that pydeephaven (and its transitive dependencies) are not actually needed to build pydeephaven-ticking.
 
This also migrates to the PyPi release to Trusted Publishers, https://docs.pypi.org/trusted-publishers/.

Fixes #5288
Fixes #5296

.github/workflows/publish-ci.yml

@@ -13,6 +13,8 @@ concurrency:
 jobs:
   publish:
     runs-on: ubuntu-22.04
+    permissions:
+      id-token: write
     steps:
       - name: Checkout
         uses: actions/checkout@v4
@@ -52,7 +54,7 @@ jobs:
         uses: burrunan/gradle-cache-action@v1
         with:
           job-id: publish-local
-          arguments: server-netty-app:build server-jetty-app:build py-server:build py-embedded-server:build py-client:build web-client-api:types:build publishToMavenLocal
+          arguments: server-netty-app:build server-jetty-app:build py-server:build py-embedded-server:build py-client:build py-client-ticking:build web-client-api:types:build publishToMavenLocal
           gradle-version: wrapper
 
       - name: Build all artifacts, publish to Sonatype for staging to Maven Central
@@ -61,7 +63,7 @@ jobs:
         with:
           job-id: publish
           # We need to be explicit here about no parallelism to ensure we don't create disjointed staging repositories.
-          arguments: --no-parallel server-netty-app:build server-jetty-app:build py-server:build py-client:build py-embedded-server:build web-client-api:types:build publish
+          arguments: --no-parallel server-netty-app:build server-jetty-app:build py-server:build py-embedded-server:build py-client:build py-client-ticking:build web-client-api:types:build publish
           gradle-version: wrapper
         env:
           ORG_GRADLE_PROJECT_ossrhUsername: ${{ secrets.SONATYPE_USERNAME }}
@@ -80,35 +82,37 @@ jobs:
             py/server/build/wheel/
             py/embedded-server/build/wheel/
             py/client/build/wheel/
+            py/client-ticking/build/wheel/
             web/client-api/types/build/*.tgz
 
       - name: Publish deephaven-core to PyPi
         if: ${{ startsWith(github.ref, 'refs/heads/release/v') }}
         uses: pypa/gh-action-pypi-publish@release/v1
         with:
-          user: __token__
-          password: ${{ secrets.DEEPHAVENCORE_PYPI_TOKEN }}
           packages_dir: py/server/build/wheel/
         continue-on-error: true
 
       - name: Publish deephaven-server to PyPi
         if: ${{ startsWith(github.ref, 'refs/heads/release/v') }}
         uses: pypa/gh-action-pypi-publish@release/v1
         with:
-          user: __token__
-          password: ${{ secrets.DEEPHAVENSERVER_PYPI_TOKEN }}
           packages_dir: py/embedded-server/build/wheel/
         continue-on-error: true
 
       - name: Publish pydeephaven to PyPi
         if: ${{ startsWith(github.ref, 'refs/heads/release/v') }}
         uses: pypa/gh-action-pypi-publish@release/v1
         with:
-          user: __token__
-          password: ${{ secrets.PYDEEPHAVEN_PYPI_TOKEN }}
           packages_dir: py/client/build/wheel/
         continue-on-error: true
 
+      - name: Publish pydeephaven-ticking to PyPi
+        if: ${{ startsWith(github.ref, 'refs/heads/release/v') }}
+        uses: pypa/gh-action-pypi-publish@release/v1
+        with:
+          packages_dir: py/client-ticking/build/wheel/
+        continue-on-error: true
+
       - name: Publish @deephaven/jsapi-types to npmjs
         if: ${{ startsWith(github.ref, 'refs/heads/release/v') }}
         env:

RELEASE.md

@@ -49,7 +49,8 @@ It may also be useful for release management purposes to ensure that no unexpect
 It is released to [GitHub releases](https://github.com/deephaven/deephaven-core/releases) in the [syft](https://github.com/anchore/syft) json format.
 
 ### Deephaven python client
-The Deephaven python client is released as the `pydeephaven` wheel at [PyPi](https://pypi.org/project/pydeephaven/).
+The Deephaven python client wheels are released on PyPi as [`pydeephaven`](https://pypi.org/project/pydeephaven/) and
+[`pydeephaven-ticking`](https://pypi.org/project/pydeephaven-ticking/).
 
 ### Deephaven go client
 The Deephaven go client is released as a [Go package](https://pkg.go.dev/github.com/deephaven/deephaven-core/go).
@@ -161,6 +162,9 @@ If this step fails, the deephaven-server wheel from the "Upload Artifacts" step
 The "Publish pydeephaven to PyPi" uploads the pydeephaven wheel to [PyPi](https://pypi.org/project/pydeephaven/).
 If this step fails, the pydeephaven wheel from the "Upload Artifacts" step can be uploaded manually.
 
+The "Publish pydeephaven-ticking to PyPi" uploads the pydeephaven-ticking wheels to [PyPi](https://pypi.org/project/pydeephaven-ticking/).
+If this step fails, the pydeephaven-ticking wheel from the "Upload Artifacts" step can be uploaded manually.
+
 The "Publish @deephaven/jsapi-types to npmjs" uploads the TypeScript tarball to [NPM](https://www.npmjs.com/package/@deephaven/jsapi-types).
 If this step fails, the deephaven-jsapi-types tarball from the "Upload Artifacts" step can be uploaded manually.
 
@@ -228,7 +232,7 @@ Create a new [GitHub release](https://github.com/deephaven/deephaven-core/releas
 
 The convention is to have the Release title of the form `vX.Y.Z` and to autogenerate the release notes in comparison to the previous release tag. Question: should we always generate release notes based off of the previous minor release, instead of patch? Our git release workflow suggests we may want to do it always minor to minor.
 
-Upload the Deephaven server application, deephaven-core wheel, pydeephaven wheel, @deephaven/jsapi-types tarball, and SBOM artifacts. Also, upload the C++, Java, Python, R and TypeScript docs artifacts.
+Upload the Deephaven server application, deephaven-core wheel, pydeephaven wheel, pydeephaven-ticking wheels, @deephaven/jsapi-types tarball, and SBOM artifacts. Also, upload the C++, Java, Python, R and TypeScript docs artifacts.
 (These are the artifacts downloaded in Step #5)
 
 Hit the GitHub "Publish release" button.

py/client-ticking/build.gradle

@@ -28,7 +28,9 @@ deephavenDocker {
     networkName.set "pydeephaven-network-${randomSuffix}"
 }
 
-def buildPyClientTickingManyLinux = { wheelsSet, taskName -> Docker.registerDockerTask(project, taskName) {
+// This task is based off of manylinux wheel, and is technically independent of specific python implementation. We are
+// currently using this for CPython wheel builds, but could use it in the future for other python implementations (PyPy).
+def pyClientTickingWheel = { String pythonTag -> Docker.registerDockerTask(project, "pyClientTickingWheel-${pythonTag}") {
     // Only tested on x86-64, and we only build dependencies for x86-64
     platform = 'linux/amd64'
 
@@ -40,63 +42,50 @@ def buildPyClientTickingManyLinux = { wheelsSet, taskName -> Docker.registerDock
             include 'src/**/*.py'
             include 'src/**/*.pyx'
             include 'src/**/*.pxd'
-        }
-        from(configurations.pythonWheel) {
-            into 'wheels'
+            include 'pyClientTickingWheel/entrypoint.sh'
         }
     }
     copyOut {
-        into layout.buildDirectory.dir('wheel')
+        into layout.buildDirectory.dir("pyClientTickingWheel/${pythonTag}")
     }
     dockerfile {
         from('deephaven/cpp-client-py:local-build')
         runCommand("""mkdir -p \\
                         /out \\
                         '${prefix}/log' \\
                         '${prefix}/src/py-client-ticking/src' \\
-                        '${prefix}/src/py-client-ticking/in-wheels'
-
+                        '${prefix}/src/py-client-ticking/in-wheels'; \\
+                   /opt/python/${pythonTag}/bin/pip install cython
                    """)
         copyFile('setup.py', "${prefix}/src/py-client-ticking")
         copyFile('README.md', "${prefix}/src/py-client-ticking")
         copyFile('src/', "${prefix}/src/py-client-ticking/src/")
-        copyFile('wheels/', "${prefix}/src/py-client-ticking/in-wheels")
-        runCommand("PREFIX='${prefix}'; WHEELS_SET='${wheelsSet}'; DEEPHAVEN_VERSION='${project.version}';" +
-                   '''set -eux ; \
-                      cd "${PREFIX}/src/py-client-ticking"; \
-                      . "${PREFIX}/env.sh"; \
-                      ORIG_PATH="$PATH"; \
-                      for spec in ${WHEELS_SET}; do \
-                        tag=`echo "$spec" | cut -d: -f 2`; \
-                        rm -f *.cpp *.so; \
-                        PATH="/opt/python/${tag}/bin:$ORIG_PATH"; \
-                        pip3 install cython; \
-                        MAKEFLAGS="-j${NCPUS}" \
-                          CFLAGS="-I${DHCPP}/include" \
-                          LDFLAGS="-L${DHCPP}/lib" \
-                          DEEPHAVEN_VERSION="${DEEPHAVEN_VERSION}" \
-                          python3 setup.py build_ext -i; \
-                        DEEPHAVEN_VERSION="${DEEPHAVEN_VERSION}" python3 setup.py bdist_wheel; \
-                        auditwheel repair dist/pydeephaven_ticking*"${tag}"*.whl; \
-                        rm -f dist/pydeephaven_ticking*"${tag}"*.whl; \
-                        mv -f wheelhouse/*.whl dist; \
-                        pip3 install in-wheels/*.whl; \
-                        pip3 install --no-deps dist/pydeephaven_ticking*"${tag}"*.whl; \
-                      done; \
-                      ln dist/*.whl /out; \
-                      cd /; \
-                      rm -fr "${PREFIX}/src/py-client-ticking"
-                   ''')
-
-        environmentVariable 'LD_LIBRARY_PATH', ""  // avoid conflict with libarrow.13.0.0.so
+        copyFile('pyClientTickingWheel/entrypoint.sh', '/entrypoint.sh')
+        environmentVariable([
+                PREFIX: prefix,
+                PYTHON_TAG: pythonTag,
+                DEEPHAVEN_VERSION: project.version.toString()
+        ])
     }
     parentContainers = [ project.tasks.getByPath(':cpp-client:cppClientPy') ]
+    entrypoint = ['/entrypoint.sh']
 }}
 
-def checksWheelSet = '3.9:cp39-cp39'
-def buildPyClientTicking = buildPyClientTickingManyLinux(checksWheelSet, 'pyClientTicking')
+def getCPythonTag = { String pythonVersion ->
+    def noPeriodVersion = pythonVersion.replace('.', '')
+    return "cp${noPeriodVersion}-cp${noPeriodVersion}".toString()
+}
+
+def pythonVersions = [ '3.8', '3.9', '3.10', '3.11', '3.12' ]
 
-def testPyClientTickingManyLinux = { wheelsSet, taskName, parentContainer, image -> Docker.registerDockerTask(project, taskName ) {
+Map<String, TaskProvider<? extends Task>> cpythonBuilds = pythonVersions.collectEntries { pythonVersion ->
+    def cpythonTag = getCPythonTag(pythonVersion)
+    return [(cpythonTag): pyClientTickingWheel(cpythonTag)]
+}
+
+// Our testing images are currently CPython specific
+def testCPythonClientTicking = { String pythonVersion, String image -> Docker.registerDockerTask(project, "testCPythonClientTicking-${pythonVersion}-${image}") {
+    def cpythonWheelTask = cpythonBuilds.get(getCPythonTag(pythonVersion)).get()
     // Only tested on x86-64, and we only build dependencies for x86-64
     platform = 'linux/amd64'
     copyIn {
@@ -105,95 +94,80 @@ def testPyClientTickingManyLinux = { wheelsSet, taskName, parentContainer, image
         }
         from(layout.projectDirectory) {
             include 'tests/**'
+            include 'testPyClientTicking/entrypoint.sh'
         }
-        from(layout.buildDirectory.dir('wheel')) {
+        from(cpythonWheelTask.outputs) {
             into 'pyt-wheels'
         }
     }
     dockerfile {
         from("deephaven/${image}:local-build")
-        runCommand("WHEELS_SET='${wheelsSet}'; PREFIX='${prefix}'; " +
-                   '''set -eux ; \
-                      DNF=`type microdnf >/dev/null 2>&1 && echo 'microdnf --disableplugin=subscription-manager' || echo 'dnf -q'`; \
-                      $DNF -y update; \
-                      $DNF -y install python3; \
-                      $DNF -y install python3-pip; \
-                      $DNF -y install python3-virtualenv; \
-                      for spec in ${WHEELS_SET}; do \
-                        pyver=`echo "$spec" | cut -d: -f 1`; \
-                        $DNF -y install "python${pyver}"; \
-                        "python$pyver" -m venv "/project/$pyver"; \
-                        source "/project/$pyver/bin/activate"; \
-                        pip install --upgrade pip; \
-                        pip install unittest-xml-reporting; \
-                        deactivate; \
-                      done; \
-                      $DNF clean all; \
-                      rm -fr /out; \
-                      mkdir -p \
-                        /out/report \
-                        /project/tests                      
+        runCommand(
+                '''set -eux; \
+                   DNF=`type microdnf >/dev/null 2>&1 && echo 'microdnf --disableplugin=subscription-manager' || echo 'dnf -q'`; \
+                   $DNF -y update
                    ''')
-        copyFile('tests/', "/project/tests/")
-        copyFile('dep-wheels/', "/project/dep-wheels")
-        copyFile('pyt-wheels/', "/project/pyt-wheels")
-        workingDir('/project')
-        runCommand("WHEELS_SET='${wheelsSet}'; " +
-                   '''set -eux ; \
-                      for spec in ${WHEELS_SET}; do \
-                        pyver=`echo "$spec" | cut -d: -f 1`; \
-                        tag=`echo "$spec" | cut -d: -f 2`; \
-                        [ -f /project/pyt-wheels/pydeephaven_ticking*"${tag}"*.whl ]; \
-                        source "/project/$pyver/bin/activate"; \
-                        pip install unittest-xml-reporting; \
-                        pip install /project/dep-wheels/*.whl; \
-                        pip install /project/pyt-wheels/pydeephaven_ticking*"${tag}"*.whl; \
-                        deactivate; \
-                     done
+        runCommand("PYTHON_VERSION='${pythonVersion}'; " +
+                '''set -eux; \
+                   DNF=`type microdnf >/dev/null 2>&1 && echo 'microdnf --disableplugin=subscription-manager' || echo 'dnf -q'`; \
+                   $DNF -y install "python${PYTHON_VERSION}"; \
+                   "python${PYTHON_VERSION}" -m venv "/project/${PYTHON_VERSION}"; \
+                   source "/project/${PYTHON_VERSION}/bin/activate"; \
+                   pip install --upgrade pip; \
+                   pip install unittest-xml-reporting; \
+                   rm -fr /out; \
+                   mkdir -p \
+                       /out/report \
+                       /project/tests
                    ''')
-
+        copyFile('tests/', '/project/tests/')
+        copyFile('dep-wheels/', '/project/dep-wheels')
+        copyFile('pyt-wheels/', '/project/pyt-wheels')
+        copyFile('testPyClientTicking/entrypoint.sh', '/entrypoint.sh')
+        workingDir('/project')
         //
         // Setup for test run.
         //
-        environmentVariable 'DH_HOST', deephavenDocker.containerName.get()
-        environmentVariable 'DH_PORT', '10000'
+        environmentVariable([
+                DH_HOST: deephavenDocker.containerName.get(),
+                DH_PORT: '10000',
+                PYTHON_VERSION: pythonVersion
+        ])
     }
     containerDependencies.dependsOn = [deephavenDocker.healthyTask]
     containerDependencies.finalizedBy = deephavenDocker.endTask
     network = deephavenDocker.networkName.get()
-    parentContainers = [ project.tasks.getByName(parentContainer),
-                         Docker.registryTask(project, "${image}") ]
-    entrypoint = ['bash', '-c',
-                  "WHEELS_SET='${wheelsSet}'; " +
-                  '''
-                  for spec in ${WHEELS_SET}; do \
-                    pyver=`echo "$spec" | cut -d: -f 1`; \
-                    source "/project/$pyver/bin/activate"; \
-                    "python$pyver" -m xmlrunner discover tests -v -o "/out/report/$pyver"; \
-                    deactivate
-                  done
-                  ''']
+    parentContainers = [ cpythonWheelTask, Docker.registryTask(project, image.toString()) ]
     copyOut {
-        into layout.buildDirectory.dir('test-results')
+        into layout.buildDirectory.dir("testCPythonClientTicking/${image}/${pythonVersion}")
     }
+    entrypoint = ['/entrypoint.sh']
 }}
 
-def testPyClientTicking = testPyClientTickingManyLinux(
-    checksWheelSet,
-    'testPyClientTicking',
-    'pyClientTicking',
-    'ubi-minimal')
+Map<String, TaskProvider<? extends Task>> fedoraCPythonTests = pythonVersions.collectEntries { pythonVersion ->
+    return [(pythonVersion): testCPythonClientTicking(pythonVersion, 'fedora')]
+}
+
+// Image does not support 3.10 nor 3.12
+Map<String, TaskProvider<? extends Task>> ubiMinimalCPythonTests = (pythonVersions - ['3.10', '3.12']).collectEntries { pythonVersion ->
+    return [(pythonVersion): testCPythonClientTicking(pythonVersion, 'ubi-minimal')]
+}
+
+def testPyClientTicking = project.tasks.register('testPyClientTicking') {
+    dependsOn fedoraCPythonTests.values()
+    dependsOn ubiMinimalCPythonTests.values()
+    doLast {
+        for (def cpythonTest in fedoraCPythonTests.values()) {
+            cpythonTest.get().state.rethrowFailure()
+        }
+        for (def cpythonTest in ubiMinimalCPythonTests.values()) {
+            cpythonTest.get().state.rethrowFailure()
+        }
+    }
+}
 
-def wheelsSet = [ '3.8:cp38-cp38', '3.9:cp39-cp39', '3.10:cp310-cp310', '3.11:cp311-cp311', '3.12:cp312-cp312' ]
-def pyClientTickingAllWheels = buildPyClientTickingManyLinux(String.join(' ', wheelsSet), 'pyClientTickingAllWheels')
+assemble.dependsOn cpythonBuilds.values()
 
-def testPyClientTickingAllWheels = testPyClientTickingManyLinux(
-    String.join(' ', wheelsSet),
-    'testPyClientTickingAllWheels',
-    'pyClientTickingAllWheels',
-    'fedora')
+check.dependsOn testPyClientTicking
 
-tasks.getByName('check').dependsOn(testPyClientTicking)
-tasks.getByName('testPyClientTickingPrepareDocker').dependsOn(pyClientTicking)
-tasks.getByName('testPyClientTickingAllWheelsPrepareDocker').dependsOn(pyClientTickingAllWheels)
 deephavenDocker.shouldLogIfTaskFails testPyClientTicking

py/client-ticking/pyClientTickingWheel/entrypoint.sh

@@ -0,0 +1,24 @@
+#!/usr/bin/env bash
+
+set -o errexit
+set -o pipefail
+set -o nounset
+
+cd "${PREFIX}/src/py-client-ticking"
+source "${PREFIX}/env.sh"
+
+rm -f ./*.cpp ./*.so
+
+# Note: using PATH b/c these do not have venv activates
+PATH="/opt/python/${PYTHON_TAG}/bin:$PATH"
+
+MAKEFLAGS="-j${NCPUS}" \
+  CFLAGS="-I${DHCPP}/include" \
+  LDFLAGS="-L${DHCPP}/lib" \
+  DEEPHAVEN_VERSION="${DEEPHAVEN_VERSION}" \
+  python setup.py build_ext -i
+
+DEEPHAVEN_VERSION="${DEEPHAVEN_VERSION}" \
+  python setup.py bdist_wheel
+
+auditwheel repair --wheel-dir /out dist/pydeephaven_ticking*"${PYTHON_TAG}"*.whl