Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use conda forge to build package #223

Merged
merged 6 commits into from
Oct 14, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/api-docs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ jobs:
# because the `env` context is only accessible at the step level;
# hence, it is hard-coded
image: |-
ghcr.io/khiopsml/khiops-python/khiopspydev-ubuntu22.04:${{ inputs.image-tag || '10.2.3-b.4.with_native_10.2.3-b.4' }}
ghcr.io/khiopsml/khiops-python/khiopspydev-ubuntu22.04:${{ inputs.image-tag || '10.2.3-b.4.with_conda-forge_10.2.3-b.4' }}
# Use the 'runner' user (1001) from github so checkout actions work properly
# https://github.com/actions/runner/issues/2033#issuecomment-1598547465
options: --user 1001
Expand Down
28 changes: 10 additions & 18 deletions .github/workflows/conda.yml
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ on:
workflow_dispatch:
inputs:
khiops-core-version:
default: 10.2.2
default: 10.2.3b.4
description: khiops-core version for testing
khiops-samples-version:
default: 10.2.0
Expand Down Expand Up @@ -39,19 +39,18 @@ jobs:
# Checkout the full repository to have the tags so versioneer works properly
# See issue https://github.com/actions/checkout/issues/701
fetch-depth: 0
- name: Install Miniconda
- name: Install Miniforge
uses: conda-incubator/setup-miniconda@v3
with:
miniconda-version: latest
miniforge-version: latest
python-version: '3.12'
- name: Install Dependency Requirements for Building Conda Packages
run: conda install -y conda-build
- name: Build the Conda Package
# Note: The "khiops-dev" conda channel is needed to retrieve the "khiops-core" package.
# The "test" part of the conda recipe needs this package.
run: |
conda build --channel conda-forge --channel khiops-dev \
--output-folder ./khiops-conda ./packaging/conda
conda build --channel khiops-dev --output-folder ./khiops-conda ./packaging/conda
- name: Upload Conda Package Artifact
uses: actions/upload-artifact@v4
with:
Expand All @@ -78,10 +77,10 @@ jobs:
runs-on: ${{ matrix.env.os }}
container: ${{ fromJSON(matrix.env.json-image) }}
steps:
- name: Install Miniconda
- name: Install Miniforge
uses: conda-incubator/setup-miniconda@v3
with:
miniconda-version: latest # needed for macOS 13
miniforge-version: latest # needed for macOS 13
python-version: ${{ matrix.python-version }}
- name: Download Conda Package Artifact
uses: actions/download-artifact@v4
Expand All @@ -92,16 +91,9 @@ jobs:
run: |
KHIOPS_CORE_VERSION="${{ inputs.khiops-core-version || env.DEFAULT_KHIOPS_CORE_VERSION }}"
echo "KHIOPS_CORE_VERSION=$KHIOPS_CORE_VERSION" >> "$GITHUB_ENV"
- name: Install the Khiops Conda package (Windows)
if: runner.os == 'Windows'
- name: Install the Khiops Conda package
run: |
conda install --channel conda-forge --channel khiops-dev khiops-core=$KHIOPS_CORE_VERSION
conda install --override-channels --channel conda-forge --channel ./khiops-conda/ khiops
# In Linux/macOS we need the conda-forge channel to install their pinned versions
- name: Install the Khiops Conda package (Linux/macOS)
if: runner.os != 'Windows'
run: |
conda install --channel conda-forge --channel khiops-dev khiops-core=$KHIOPS_CORE_VERSION
conda install --channel khiops-dev khiops-core=$KHIOPS_CORE_VERSION
conda install --channel ./khiops-conda/ khiops
- name: Test Khiops Installation Status
run: kh-status
Expand Down Expand Up @@ -135,10 +127,10 @@ jobs:
# See the upload-artifact step in the build job for the explanation of this pattern
name: khiops-conda
path: ./khiops-conda
- name: Install Miniconda
- name: Install Miniforge
uses: conda-incubator/setup-miniconda@v3
with:
miniconda-version: latest
miniforge-version: latest
python-version: '3.12'
- name: Install Requirement Packages
run: conda install -y anaconda-client conda-index
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/dev-docker.yml
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
---
name: Dev Docker
env:
DEFAULT_KHIOPS_REVISION: 10.2.2
DEFAULT_KHIOPS_REVISION: 10.2.3-b.4
DEFAULT_IMAGE_INCREMENT: 0
DEFAULT_SERVER_REVISION: main
DEFAULT_PYTHON_VERSIONS: 3.8 3.9 3.10 3.11 3.12
Expand All @@ -12,7 +12,7 @@ on:
inputs:
khiops-revision:
type: string
default: main
default: 10.2.3-b.4
description: Khiops Revision
image-increment:
type: number
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/pip.yml
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,7 @@ jobs:
# because the `env` context is only accessible at the step level;
# hence, it is hard-coded
image: |-
ghcr.io/khiopsml/khiops-python/khiopspydev-${{ matrix.container }}:${{ inputs.image-tag || '10.2.3-b.4.with_native_10.2.3-b.4' }}
ghcr.io/khiopsml/khiops-python/khiopspydev-${{ matrix.container }}:${{ inputs.image-tag || '10.2.3-b.4.with_conda-forge_10.2.3-b.4' }}
steps:
- name: Set parameters as env
run: |
Expand Down
63 changes: 36 additions & 27 deletions .github/workflows/unit-tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
name: Unit Tests
env:
DEFAULT_SAMPLES_REVISION: main
DEFAULT_KHIOPS_DESKTOP_REVISION: 10.2.3-b.3
DEFAULT_KHIOPS_DESKTOP_REVISION: 10.2.3-b.4
on:
workflow_dispatch:
inputs:
Expand All @@ -13,7 +13,7 @@ on:
default: latest
description: Development Docker Image Tag
khiops-desktop-revision:
default: 10.2.3-b.3
default: 10.2.3-b.4
description: Khiops Windows Desktop Application Version
run-long-tests:
type: boolean
Expand All @@ -36,7 +36,7 @@ jobs:
# because the `env` context is only accessible at the step level;
# hence, it is hard-coded
image: |-
ghcr.io/khiopsml/khiops-python/khiopspydev-ubuntu22.04:${{ inputs.image-tag || '10.2.3-b.4.with_native_10.2.3-b.4' }}
ghcr.io/khiopsml/khiops-python/khiopspydev-ubuntu22.04:${{ inputs.image-tag || '10.2.3-b.4.with_conda-forge_10.2.3-b.4' }}
credentials:
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
Expand Down Expand Up @@ -70,29 +70,36 @@ jobs:
- name: Setup and Install Test Requirements
if: success() || failure()
run: |
mkdir -p -m u+rwx reports/py${{ matrix.python-version }}
mkdir -p -m u+rwx reports/py${{ matrix.python-version }}_conda
# install within the conda environments without activating them
# Native Khiops-based Conda environment
/root/miniconda3/bin/conda install -y -n py${{ matrix.python-version }} -c conda-forge unittest-xml-reporting
/root/miniconda3/bin/conda install -y -n py${{ matrix.python-version }} --file test-requirements.txt
CONDA="/root/miniforge3/bin/conda"

# Native Khiops-based Conda environment, and
# `khiops-core`-based Conda environment
/root/miniconda3/bin/conda install -y -n py${{ matrix.python-version }}_conda -c conda-forge unittest-xml-reporting
/root/miniconda3/bin/conda install -y -n py${{ matrix.python-version }}_conda --file test-requirements.txt
CONDA_ENVS="py${{ matrix.python-version }} py${{ matrix.python-version }}_conda"
for CONDA_ENV in $CONDA_ENVS
do
mkdir -p -m u+rwx reports/"$CONDA_ENV"

# install within the conda environments without activating them
$CONDA install -y -n "$CONDA_ENV" unittest-xml-reporting
$CONDA install -y -n "$CONDA_ENV" --file test-requirements.txt
done
- name: Install khiops-python dependencies
if: success() || failure()
run: |
# The following git command is required,
# as the Git repository is in a directory the current user does not own,
# Python versioneer fails to compute the current version correctly otherwise
git config --global --add safe.directory $(realpath .)
# Native Khiops-based Conda environment
/root/miniconda3/bin/conda run --no-capture-output -n py${{ matrix.python-version }} python setup.py egg_info
/root/miniconda3/bin/conda install -y -n py${{ matrix.python-version }} `grep -v "^\[" khiops.egg-info/requires.txt`
git config --global --add safe.directory $(realpath .)
CONDA="/root/miniforge3/bin/conda"
# Native Khiops-based Conda environment, and
# `khiops-core`-based Conda environment
/root/miniconda3/bin/conda run --no-capture-output -n py${{ matrix.python-version }}_conda python setup.py egg_info
/root/miniconda3/bin/conda install -y -n py${{ matrix.python-version }}_conda `grep -v "^\[" khiops.egg-info/requires.txt`
rm -rf khiops.egg-info
CONDA_ENVS="py${{ matrix.python-version }} py${{ matrix.python-version }}_conda"
for CONDA_ENV in $CONDA_ENVS
do
$CONDA run --no-capture-output -n "$CONDA_ENV" python setup.py egg_info
$CONDA install -y -n "$CONDA_ENV" `grep -v "^\[" khiops.egg-info/requires.txt`
rm -rf khiops.egg-info
done
- name: Prepare Unit Tests Environment
if: github.ref != 'dev' && github.ref != 'main' && ! inputs.run-long-tests
run: echo "UNITTEST_ONLY_SHORT_TESTS=true" >> "$GITHUB_ENV"
Expand All @@ -113,14 +120,16 @@ jobs:
# This is needed so that the Git tag is parsed and the khiops-python
# version is retrieved
git config --global --add safe.directory $(realpath .)
# Native Khiops-based Conda environments
/root/miniconda3/bin/conda run --no-capture-output -n py${{ matrix.python-version }} coverage run -m xmlrunner -o "reports/py${{ matrix.python-version }}" -v
/root/miniconda3/bin/conda run --no-capture-output -n py${{ matrix.python-version }} coverage report -m
/root/miniconda3/bin/conda run --no-capture-output -n py${{ matrix.python-version }} coverage xml -o "reports/py${{ matrix.python-version }}/py-coverage.xml"
# `khiops-core`-based Conda environments
/root/miniconda3/bin/conda run --no-capture-output -n py${{ matrix.python-version }}_conda coverage run -m xmlrunner -o "reports/py${{ matrix.python-version }}_conda" -v
/root/miniconda3/bin/conda run --no-capture-output -n py${{ matrix.python-version }}_conda coverage report -m
/root/miniconda3/bin/conda run --no-capture-output -n py${{ matrix.python-version }}_conda coverage xml -o "reports/py${{ matrix.python-version }}_conda/py-coverage.xml"
CONDA="/root/miniforge3/bin/conda"
# Native Khiops-based Conda environment, and
# `khiops-core`-based Conda environment
CONDA_ENVS="py${{ matrix.python-version }} py${{ matrix.python-version }}_conda"
for CONDA_ENV in $CONDA_ENVS
do
$CONDA run --no-capture-output -n "$CONDA_ENV" coverage run -m xmlrunner -o "reports/py${{ matrix.python-version }}" -v
$CONDA run --no-capture-output -n "$CONDA_ENV" coverage report -m
$CONDA run --no-capture-output -n "$CONDA_ENV" coverage xml -o "reports/$CONDA_ENV/py-coverage.xml"
done
- name: Display Unit Test Reports
uses: dorny/test-reporter@v1
with:
Expand Down Expand Up @@ -217,7 +226,7 @@ jobs:
# because the `env` context is only accessible at the step level;
# hence, it is hard-coded
image: |-
ghcr.io/khiopsml/khiops-python/khiopspydev-${{ matrix.container }}:${{ inputs.image-tag || '10.2.3-b.4.with_native_10.2.3-b.4' }}
ghcr.io/khiopsml/khiops-python/khiopspydev-${{ matrix.container }}:${{ inputs.image-tag || '10.2.3-b.4.with_conda-forge_10.2.3-b.4' }}
credentials:
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
Expand Down
20 changes: 10 additions & 10 deletions packaging/docker/khiopspydev/Dockerfile.rocky
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ LABEL description="Container for the development of khiops-python"
# Reuse KHIOPSDEV_OS from previous stage
ARG KHIOPSDEV_OS
ARG KHIOPS_REVISION
# - Install dev tools and miniconda3 (for the unit tests)
# - Install dev tools and miniforge (for the unit tests)
# - Build and install Khiops
# - Set mpich as the default MPI
RUN true \
Expand Down Expand Up @@ -55,12 +55,12 @@ RUN true \
alternatives --install /usr/bin/python python /usr/bin/python3 1 \
&& alternatives --install /usr/bin/pip pip /usr/bin/pip3 1 ; \
fi \
# Install miniconda3 to have multiple Python versions via Conda \
&& mkdir -p /root/miniconda3 && cd /root/miniconda3 \
&& wget https://repo.anaconda.com/miniconda/Miniconda3-py312_24.1.2-0-Linux-x86_64.sh -O ./Miniconda3-py312_24.1.2-0-Linux-x86_64.sh \
&& echo "b978856ec3c826eb495b60e3fffe621f670c101150ebcbdeede4f961f22dc438 Miniconda3-py312_24.1.2-0-Linux-x86_64.sh" | sha256sum --check \
&& bash ./Miniconda3-py312_24.1.2-0-Linux-x86_64.sh -b -u -p /root/miniconda3 \
&& rm -rf /root/miniconda3/Miniconda3-py312_24.1.2-0-Linux-x86_64.sh \
# Install miniforge to have multiple Python versions via Conda \
&& mkdir -p /root/miniforge3 && cd /root/miniforge3 \
&& wget https://github.com/conda-forge/miniforge/releases/download/24.1.2-0/Miniforge3-24.1.2-0-Linux-x86_64.sh -O ./Miniforge3_24.1.2-0-Linux-x86_64.sh \
&& echo "dbadb808edf4da00af35d888d3eeebbfdce71972b60bf4b16dbacaee2ab57f28 Miniforge3_24.1.2-0-Linux-x86_64.sh" | sha256sum --check \
&& bash ./Miniforge3_24.1.2-0-Linux-x86_64.sh -b -u -p /root/miniforge3 \
&& rm -rf /root/miniforge3/Miniforge3_24.1.2-0-Linux-x86_64.sh \
# Clean build files \
&& dnf clean all \
&& rm -rf ./khiops \
Expand All @@ -71,9 +71,9 @@ RUN true \
ARG PYTHON_VERSIONS
RUN /bin/bash -c 'for version in ${PYTHON_VERSIONS}; \
do \
/root/miniconda3/bin/conda create -y -n py${version} python=${version}; \
/root/miniconda3/bin/conda create -y -n py${version}_conda python=${version}; \
/root/miniconda3/bin/conda install -y -n py${version}_conda -c conda-forge -c khiops-dev khiops-core=$(echo ${KHIOPS_REVISION} | tr -d "-") ; \
/root/miniforge3/bin/conda create -y -n py${version} python=${version}; \
/root/miniforge3/bin/conda create -y -n py${version}_conda python=${version}; \
/root/miniforge3/bin/conda install -y -n py${version}_conda -c khiops-dev khiops-core=$(echo ${KHIOPS_REVISION} | tr -d "-") ; \
done'

RUN mkdir -p /scripts
Expand Down
20 changes: 10 additions & 10 deletions packaging/docker/khiopspydev/Dockerfile.ubuntu
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ FROM ghcr.io/khiopsml/khiops/khiopsdev-${KHIOPSDEV_OS}:latest AS khiopsdev
LABEL maintainer="[email protected]"
LABEL description="Container for the development of khiops-python"

# Install dev tools and miniconda3 (for the unit tests); build and install Khiops
# Install dev tools and miniforge (for the unit tests); build and install Khiops
ARG KHIOPS_REVISION
RUN true \
# Install git (for khiops-python version calculation) and pip \
Expand All @@ -22,12 +22,12 @@ RUN true \
&& rm -f KHIOPS_CORE.deb \
# Set python to python3 \
&& update-alternatives --install /usr/bin/python python /usr/bin/python3 1 \
# Install miniconda3 to have multiple Python versions via Conda \
&& mkdir -p /root/miniconda3 && cd /root/miniconda3 \
&& wget https://repo.anaconda.com/miniconda/Miniconda3-py312_24.1.2-0-Linux-x86_64.sh -O ./Miniconda3-py312_24.1.2-0-Linux-x86_64.sh \
&& echo "b978856ec3c826eb495b60e3fffe621f670c101150ebcbdeede4f961f22dc438 Miniconda3-py312_24.1.2-0-Linux-x86_64.sh" | sha256sum --check \
&& bash ./Miniconda3-py312_24.1.2-0-Linux-x86_64.sh -b -u -p /root/miniconda3 \
&& rm -rf /root/miniconda3/Miniconda3-py312_24.1.2-0-Linux-x86_64.sh \
# Install miniforge to have multiple Python versions via Conda \
&& mkdir -p /root/miniforge3 && cd /root/miniforge3 \
&& wget https://github.com/conda-forge/miniforge/releases/download/24.1.2-0/Miniforge3-24.1.2-0-Linux-x86_64.sh -O ./Miniforge3_24.1.2-0-Linux-x86_64.sh \
&& echo "dbadb808edf4da00af35d888d3eeebbfdce71972b60bf4b16dbacaee2ab57f28 Miniforge3_24.1.2-0-Linux-x86_64.sh" | sha256sum --check \
&& bash ./Miniforge3_24.1.2-0-Linux-x86_64.sh -b -u -p /root/miniforge3 \
&& rm -rf /root/miniforge3/Miniforge3_24.1.2-0-Linux-x86_64.sh \
# Make sure that MPI is openmpi \
&& update-alternatives --set mpirun /usr/bin/mpirun.openmpi \
# Clean build files \
Expand All @@ -41,9 +41,9 @@ RUN true \
ARG PYTHON_VERSIONS
RUN /bin/bash -c 'for version in ${PYTHON_VERSIONS}; \
do \
/root/miniconda3/bin/conda create -y -n py${version} python=${version}; \
/root/miniconda3/bin/conda create -y -n py${version}_conda python=${version}; \
/root/miniconda3/bin/conda install -y -n py${version}_conda -c conda-forge -c khiops-dev khiops-core=$(echo ${KHIOPS_REVISION} | tr -d "-") ; \
/root/miniforge3/bin/conda create -y -n py${version} python=${version}; \
/root/miniforge3/bin/conda create -y -n py${version}_conda python=${version}; \
/root/miniforge3/bin/conda install -y -n py${version}_conda -c khiops-dev khiops-core=$(echo ${KHIOPS_REVISION} | tr -d "-") ; \
done'

RUN mkdir -p /scripts
Expand Down
Loading