Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Build and publish #6514

Merged
merged 20 commits into from
Mar 18, 2024
Merged

Build and publish #6514

Show file tree
Hide file tree
Changes from 16 commits
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
6ce5dfd
Format file
hoxbro Mar 17, 2024
b197fc6
Implement a two step release, conda first
hoxbro Mar 17, 2024
16e7c3f
Add Bokeh channel to build and automatic run the PR
hoxbro Mar 17, 2024
54ede30
Add pip step
hoxbro Mar 17, 2024
30b96aa
Add pip check back
hoxbro Mar 17, 2024
175b525
Add NPM build and publish
hoxbro Mar 17, 2024
060ad1d
DEBUG: Publish to npm
hoxbro Mar 17, 2024
5b464ee
Uncomment release lines
hoxbro Mar 17, 2024
f885bc9
Update npm publish
hoxbro Mar 17, 2024
bed4f6a
Misc updates
hoxbro Mar 17, 2024
fe803cc
Init of CDN
hoxbro Mar 17, 2024
c0424a0
Update cdn_build
hoxbro Mar 17, 2024
7006f63
Clean up
hoxbro Mar 17, 2024
7142109
Only publish CDN on tag push
hoxbro Mar 17, 2024
17e1080
Remove build from artifact names
hoxbro Mar 17, 2024
d0f54c0
Update .github/workflows/build.yaml
hoxbro Mar 17, 2024
dccf2f7
Update .github/workflows/build.yaml
hoxbro Mar 17, 2024
4091040
Simplify
hoxbro Mar 17, 2024
0b4aed4
Add approval step and test it
hoxbro Mar 18, 2024
661920f
Clean up
hoxbro Mar 18, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
306 changes: 174 additions & 132 deletions .github/workflows/build.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,88 +2,164 @@ name: packages
on:
push:
tags:
- 'v[0-9]+.[0-9]+.[0-9]+'
- 'v[0-9]+.[0-9]+.[0-9]+a[0-9]+'
- 'v[0-9]+.[0-9]+.[0-9]+b[0-9]+'
- 'v[0-9]+.[0-9]+.[0-9]+rc[0-9]+'
- "v[0-9]+.[0-9]+.[0-9]+"
- "v[0-9]+.[0-9]+.[0-9]+a[0-9]+"
- "v[0-9]+.[0-9]+.[0-9]+b[0-9]+"
- "v[0-9]+.[0-9]+.[0-9]+rc[0-9]+"
# Dry-run only
workflow_dispatch:
schedule:
- cron: '0 19 * * SUN'
- cron: "0 19 * * SUN"

defaults:
run:
shell: bash -el {0}

env:
PACKAGE: "panel"
PYTHON_VERSION: "3.9"
hoxbro marked this conversation as resolved.
Show resolved Hide resolved
NODE_VERSION: "18"
NODE_VERSION: "20"
MPLBACKEND: "Agg"
SETUPTOOLS_ENABLE_FEATURES: "legacy-editable"

jobs:
waiting_room:
name: Waiting Room
runs-on: ubuntu-latest
needs: [conda_build, pip_install, npm_build, cdn_build]
if: github.event_name == 'push' && startsWith(github.ref, 'refs/tags/')
# environment:
# name: publish
steps:
- run: echo "All builds have finished, have been approved, and ready to publish"

conda_build:
name: Build Conda Package
runs-on: 'ubuntu-latest'
defaults:
run:
shell: bash -l {0}
name: Build Conda
runs-on: "ubuntu-latest"
steps:
- name: remove nodejs
run: |
sudo rm /usr/local/bin/node
sudo rm /usr/local/bin/npm
- uses: actions/checkout@v3
- uses: actions/checkout@v4
with:
fetch-depth: "100"
- uses: conda-incubator/setup-miniconda@v2
- name: Fetch unshallow
run: git fetch --prune --tags --unshallow -f
- uses: conda-incubator/setup-miniconda@v3
with:
miniconda-version: "latest"
python-version: ${{ env.PYTHON_VERSION }}
- uses: actions/setup-node@v4
with:
node-version: ${{ env.NODE_VERSION }}
- name: Fetch unshallow
run: git fetch --prune --tags --unshallow -f
- name: Set output
id: vars
run: echo "tag=${GITHUB_REF#refs/*/}" >> $GITHUB_OUTPUT
- name: conda setup
run: |
conda config --set always_yes True
conda config --append channels pyviz/label/dev
conda config --append channels bokeh/label/dev
conda install -y conda-build anaconda-client build
conda install -y conda-build build
- name: conda build
run: |
source ./scripts/build_conda.sh
echo "CONDA_FILE="$CONDA_PREFIX/conda-bld/noarch/panel-$VERSION-py_0.tar.bz2"" >> $GITHUB_ENV
- uses: actions/upload-artifact@v3
echo "CONDA_FILE="$CONDA_PREFIX/conda-bld/noarch/$PACKAGE-$VERSION-py_0.tar.bz2"" >> $GITHUB_ENV
- uses: actions/upload-artifact@v4
if: always()
with:
name: conda_build
if-no-files-found: error
name: conda
path: ${{ env.CONDA_FILE }}
- name: conda dev deploy
if: (github.event_name == 'push' && (contains(steps.vars.outputs.tag, 'a') || contains(steps.vars.outputs.tag, 'b') || contains(steps.vars.outputs.tag, 'rc')))
if-no-files-found: error

conda_publish:
name: Publish Conda
runs-on: ubuntu-latest
needs: [conda_build, waiting_room]
if: github.event_name == 'push' && startsWith(github.ref, 'refs/tags/')
steps:
- uses: actions/download-artifact@v4
with:
name: conda
path: dist/
- name: Set environment variables
run: |
echo "TAG=${GITHUB_REF#refs/*/}" >> $GITHUB_ENV
echo "CONDA_FILE=$(ls dist/*.tar.bz2)" >> $GITHUB_ENV
- uses: conda-incubator/setup-miniconda@v2
with:
miniconda-version: "latest"
- name: conda setup
run: |
conda install -y anaconda-client
- name: conda dev upload
if: contains(env.TAG, 'a') || contains(env.TAG, 'b') || contains(env.TAG, 'rc')
run: |
anaconda --token ${{ secrets.CONDA_UPLOAD_TOKEN }} upload --user pyviz --label=dev $CONDA_FILE
- name: conda main deploy
if: (github.event_name == 'push' && !(contains(steps.vars.outputs.tag, 'a') || contains(steps.vars.outputs.tag, 'b') || contains(steps.vars.outputs.tag, 'rc')))
- name: conda main upload
if: (!(contains(env.TAG, 'a') || contains(env.TAG, 'b') || contains(env.TAG, 'rc')))
run: |
anaconda --token ${{ secrets.CONDA_UPLOAD_TOKEN }} upload --user pyviz --label=dev --label=main $CONDA_FILE

npm_build:
name: Build NPM Package
runs-on: 'ubuntu-latest'
needs: [conda_build, pip_build]
defaults:
run:
shell: bash -l {0}
pip_build:
name: Build PyPI
runs-on: "ubuntu-latest"
steps:
- name: remove nodejs
- uses: actions/checkout@v4
with:
fetch-depth: "100"
- name: Fetch unshallow
run: git fetch --prune --tags --unshallow -f
- uses: actions/setup-python@v5
with:
python-version: ${{ env.PYTHON_VERSION }}
- uses: actions/setup-node@v4
with:
node-version: ${{ env.NODE_VERSION }}
- name: Install build
run: |
sudo rm /usr/local/bin/node
sudo rm /usr/local/bin/npm
- uses: actions/checkout@v3
python -m pip install build
- name: Build package
run: python -m build .
- uses: actions/upload-artifact@v4
if: always()
with:
name: pip
path: dist/
if-no-files-found: error

pip_install:
name: Install PyPI
runs-on: "ubuntu-latest"
needs: [pip_build]
steps:
- uses: actions/setup-python@v5
with:
python-version: ${{ env.PYTHON_VERSION }}
- uses: actions/download-artifact@v4
with:
name: pip
path: dist/
- name: Install package
run: python -m pip install dist/*.whl
- name: Test package
run: python -c "import $PACKAGE; print($PACKAGE.__version__)"

pip_publish:
name: Publish PyPI
runs-on: ubuntu-latest
needs: [pip_build, waiting_room]
if: github.event_name == 'push' && startsWith(github.ref, 'refs/tags/')
steps:
- uses: actions/download-artifact@v4
with:
name: pip
path: dist/
- name: Publish to PyPI
uses: pypa/gh-action-pypi-publish@release/v1
with:
user: ${{ secrets.PPU }}
password: ${{ secrets.PPP }}
repository-url: "https://upload.pypi.org/legacy/"

npm_build:
name: Build NPM
runs-on: "ubuntu-latest"
steps:
- uses: actions/checkout@v4
with:
fetch-depth: "100"
- uses: actions/setup-python@v4
- uses: actions/setup-python@v5
with:
python-version: ${{ env.PYTHON_VERSION }}
- uses: actions/setup-node@v4
Expand All @@ -99,49 +175,56 @@ jobs:
echo "//registry.npmjs.org/:_authToken=${{ secrets.NPM_API_TOKEN }}" > $HOME/.npmrc
npm whoami
npm -v
- name: package install
run: |
python -m pip install -ve .
- name: npm build
run: |
SETUPTOOLS_ENABLE_FEATURES="legacy-editable" python -m pip install -ve .
cd ./panel
TARBALL=$(npm pack .)
echo "TARBALL=$TARBALL" >> $GITHUB_ENV
npm publish --dry-run $TARBALL
cd ..
- uses: actions/upload-artifact@v3
- uses: actions/upload-artifact@v4
if: always()
with:
name: npm_build
name: npm
if-no-files-found: error
path: ./panel/${{ env.TARBALL }}

npm_publish:
name: Publish NPM
runs-on: "ubuntu-latest"
needs: [npm_build, waiting_room]
if: github.event_name == 'push' && startsWith(github.ref, 'refs/tags/')
steps:
- uses: actions/download-artifact@v4
with:
name: npm
- uses: actions/setup-node@v4
with:
node-version: ${{ env.NODE_VERSION }}
- name: Set environment variables
run: |
echo "TAG=${GITHUB_REF#refs/*/}" >> $GITHUB_ENV
echo "TARBALL=$(ls *.tgz)" >> $GITHUB_ENV
- name: npm dev deploy
if: (github.event_name == 'push' && (contains(steps.vars.outputs.tag, 'a') || contains(steps.vars.outputs.tag, 'b') || contains(steps.vars.outputs.tag, 'rc')))
if: contains(env.TAG, 'a') || contains(env.TAG, 'b') || contains(env.TAG, 'rc')
run: |
cd ./panel
npm publish --tag dev $TARBALL
cd ..
- name: npm main deploy
if: (github.event_name == 'push' && !(contains(steps.vars.outputs.tag, 'a') || contains(steps.vars.outputs.tag, 'b') || contains(steps.vars.outputs.tag, 'rc')))
if: (!(contains(env.TAG, 'a') || contains(env.TAG, 'b') || contains(env.TAG, 'rc')))
run: |
cd ./panel
npm publish --tag latest $TARBALL
cd ..

cdn:
cdn_build:
name: Build CDN
runs-on: 'ubuntu-latest'
needs: [conda_build, pip_build]
defaults:
run:
shell: bash -l {0}
runs-on: "ubuntu-latest"
steps:
- name: remove nodejs
run: |
sudo rm /usr/local/bin/node
sudo rm /usr/local/bin/npm
- uses: actions/checkout@v3
- uses: actions/checkout@v4
with:
fetch-depth: "100"
- uses: actions/setup-python@v4
- uses: actions/setup-python@v5
with:
python-version: ${{ env.PYTHON_VERSION }}
- uses: actions/setup-node@v4
Expand All @@ -152,79 +235,38 @@ jobs:
- name: Set output
id: vars
run: echo "tag=${GITHUB_REF#refs/*/}" >> $GITHUB_OUTPUT
- name: build pyodide wheels for CDN
- name: build pyodide wheels
run: |
python -m pip install packaging tomli
python ./scripts/build_pyodide_wheels.py --verify-clean
SETUPTOOLS_ENABLE_FEATURES="legacy-editable" python -m pip install -ve .
- uses: actions/upload-artifact@v3
- name: build CDN
run: |
python -m pip install -ve .
- uses: actions/upload-artifact@v4
if: always()
with:
name: cdn_wheels
name: cdn
if-no-files-found: error
path: |
./panel/dist/wheels/panel*.whl
./panel/dist/wheels/bokeh*.whl
./panel/dist/
./panel/package.json

cdn_publish:
name: Publish CDN
runs-on: "ubuntu-latest"
needs: [cdn_build, waiting_room]
if: github.event_name == 'push' && startsWith(github.ref, 'refs/tags/')
steps:
- uses: actions/checkout@v4
with:
fetch-depth: "1"
- uses: actions/download-artifact@v4
with:
name: cdn
path: panel/
- name: Deploy to cdn.holoviz.org
env:
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
AWS_DEFAULT_REGION: 'us-east-1'
AWS_DEFAULT_REGION: "us-east-1"
run: python scripts/cdn_upload.py

pip_build:
name: Build PyPI Package
runs-on: 'ubuntu-latest'
defaults:
run:
shell: bash -l {0}
env:
CHANS_DEV: "-c pyviz/label/dev -c bokeh/label/dev -c conda-forge"
PKG_TEST_PYTHON: "--test-python=py39"
CHANS: "-c pyviz"
PYPI: "https://upload.pypi.org/legacy/"
steps:
- name: remove nodejs
run: |
sudo rm /usr/local/bin/node
sudo rm /usr/local/bin/npm
- uses: actions/checkout@v3
with:
fetch-depth: "100"
- uses: conda-incubator/setup-miniconda@v2
with:
miniconda-version: "latest"
- uses: actions/setup-node@v4
with:
node-version: ${{ env.NODE_VERSION }}
- name: Fetch unshallow
run: git fetch --prune --tags --unshallow -f
- name: conda setup
run: |
conda config --set always_yes True
conda install -c pyviz "pyctdev>=0.5"
doit ecosystem_setup
doit env_create $CHANS_DEV --python=$PYTHON_VERSION
- name: env setup
run: |
eval "$(conda shell.bash hook)"
conda activate test-environment
conda install ${{ env.CHANS_DEV }} "pip<21.2.1"
doit develop_install $CHANS_DEV -o build
pip uninstall -y panel
doit pip_on_conda
- name: pip build
run: |
eval "$(conda shell.bash hook)"
conda activate test-environment
doit ecosystem=pip package_build $PKG_TEST_PYTHON --test-group=flakes
- name: git status
run: |
git status
git diff
- name: pip upload
if: github.event_name == 'push'
run: |
eval "$(conda shell.bash hook)"
conda activate test-environment
doit ecosystem=pip package_upload -u ${{ secrets.PPU }} -p ${{ secrets.PPP }} -r $PYPI
Loading
Loading