Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Replace package manager poetry with uv #209

Merged
merged 4 commits into from
Dec 17, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
649 changes: 293 additions & 356 deletions .github/workflows/ci.yml

Large diffs are not rendered by default.

173 changes: 89 additions & 84 deletions .github/workflows/docker-ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -4,20 +4,20 @@ name: Build and push Docker image
on:
push:
paths:
- dockerfiles/**
- poetry.lock
- pyproject.toml
- .github/workflows/docker-ci.yml
- dockerfiles/**
- uv.lock
- pyproject.toml
- .github/workflows/docker-ci.yml
pull_request:
branches:
- master
- develop
- master
- develop

env:
VERSION_NUMBER: '1.0.4'
LATEST_PYTHON_VERSION: '3.12'
LATEST_SC2_VERSION: '4.10'
EXPERIMENTAL_PYTHON_VERSION: '3.13'
VERSION_NUMBER: "1.0.4"
LATEST_PYTHON_VERSION: "3.12"
LATEST_SC2_VERSION: "4.10"
EXPERIMENTAL_PYTHON_VERSION: "3.13"

jobs:
run_test_docker_image:
Expand All @@ -30,19 +30,19 @@ jobs:
os: [ubuntu-latest]

steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v3

- name: Enable experimental docker features
run: |
echo $'{\n "experimental": true\n}' | sudo tee /etc/docker/daemon.json
sudo systemctl restart docker.service
- name: Enable experimental docker features
run: |
echo $'{\n "experimental": true\n}' | sudo tee /etc/docker/daemon.json
sudo systemctl restart docker.service

- name: Run shell script
env:
VERSION_NUMBER: ${{ env.VERSION_NUMBER }}
PYTHON_VERSION: ${{ env.LATEST_PYTHON_VERSION }}
SC2_VERSION: ${{ env.LATEST_SC2_VERSION }}
run: sh dockerfiles/test_docker_image.sh
- name: Run shell script
env:
VERSION_NUMBER: ${{ env.VERSION_NUMBER }}
PYTHON_VERSION: ${{ env.LATEST_PYTHON_VERSION }}
SC2_VERSION: ${{ env.LATEST_SC2_VERSION }}
run: sh dockerfiles/test_docker_image.sh

run_test_new_python_version:
name: Run test_new_python_candidate.sh
Expand All @@ -54,20 +54,20 @@ jobs:
os: [ubuntu-latest]

steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v3

- name: Enable experimental docker features
run: |
echo $'{\n "experimental": true\n}' | sudo tee /etc/docker/daemon.json
sudo systemctl restart docker.service
- name: Enable experimental docker features
run: |
echo $'{\n "experimental": true\n}' | sudo tee /etc/docker/daemon.json
sudo systemctl restart docker.service

- name: Run shell script
continue-on-error: true
env:
VERSION_NUMBER: ${{ env.VERSION_NUMBER }}
PYTHON_VERSION: ${{ env.EXPERIMENTAL_PYTHON_VERSION }}
SC2_VERSION: ${{ env.LATEST_SC2_VERSION }}
run: sh dockerfiles/test_new_python_candidate.sh
- name: Run shell script
continue-on-error: true
env:
VERSION_NUMBER: ${{ env.VERSION_NUMBER }}
PYTHON_VERSION: ${{ env.EXPERIMENTAL_PYTHON_VERSION }}
SC2_VERSION: ${{ env.LATEST_SC2_VERSION }}
run: sh dockerfiles/test_new_python_candidate.sh

docker_build:
name: Build docker image
Expand All @@ -77,59 +77,64 @@ jobs:
fail-fast: false
matrix:
os: [ubuntu-latest]
python-version: ['3.9', '3.10', '3.11', '3.12', '3.13']
sc2-version: ['4.10']
python-version: ["3.9", "3.10", "3.11", "3.12", "3.13"]
sc2-version: ["4.10"]
env:
IMAGE_NAME: burnysc2/python-sc2-docker:py_${{ matrix.python-version }}-sc2_${{ matrix.sc2-version }}
BUILD_ARGS: --build-arg PYTHON_VERSION=${{ matrix.python-version }} --build-arg SC2_VERSION=${{ matrix.sc2-version }}

steps:
- uses: actions/checkout@v3

- name: Build docker image
run: docker build -t $IMAGE_NAME-v$VERSION_NUMBER $BUILD_ARGS - < dockerfiles/Dockerfile

- name: Enable experimental docker features
run: |
echo $'{\n "experimental": true\n}' | sudo tee /etc/docker/daemon.json
sudo systemctl restart docker.service

- name: Build squashed image
run: docker build -t $IMAGE_NAME-v$VERSION_NUMBER-squashed --squash $BUILD_ARGS - < dockerfiles/Dockerfile

- name: Run test bots on squashed image
if: matrix.python-version != '3.7'
run: |
echo "Start container, override the default entrypoint"
docker run -i -d \
--name test_container \
--mount type=bind,source="$(pwd)",destination=/root/python-sc2,readonly \
--entrypoint /bin/bash \
$IMAGE_NAME-v$VERSION_NUMBER-squashed
echo "Install python-sc2"
docker exec -i test_container bash -c "pip install poetry \
&& cd python-sc2 && poetry install --no-dev"
echo "Run various test bots"
docker exec -i test_container bash -c "cd python-sc2 && poetry run python test/travis_test_script.py test/autotest_bot.py"
docker exec -i test_container bash -c "cd python-sc2 && poetry run python test/run_example_bots_vs_computer.py"

- name: Login to DockerHub
if: github.ref == 'refs/heads/develop'
uses: docker/login-action@v2
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}

- name: Upload docker image
if: github.ref == 'refs/heads/develop' && github.event_name == 'push'
run: docker push $IMAGE_NAME-v$VERSION_NUMBER

- name: Upload squashed docker image
if: github.ref == 'refs/heads/develop' && github.event_name == 'push'
run: docker push $IMAGE_NAME-v$VERSION_NUMBER-squashed

- name: Upload squashed docker image as latest tag
if: github.ref == 'refs/heads/develop' && github.event_name == 'push' && matrix.python-version == env.LATEST_PYTHON_VERSION && matrix.sc2-version == env.LATEST_SC2_VERSION
run: |
docker tag $IMAGE_NAME-v$VERSION_NUMBER-squashed burnysc2/python-sc2-docker:latest
docker push burnysc2/python-sc2-docker:latest
- uses: actions/checkout@v3

- name: Build docker image
run: docker build -t $IMAGE_NAME-v$VERSION_NUMBER $BUILD_ARGS - < dockerfiles/Dockerfile

- name: Enable experimental docker features
run: |
echo $'{\n "experimental": true\n}' | sudo tee /etc/docker/daemon.json
sudo systemctl restart docker.service

- name: Build squashed image
run: docker build -t $IMAGE_NAME-v$VERSION_NUMBER-squashed --squash $BUILD_ARGS - < dockerfiles/Dockerfile

- name: Run test bots on squashed image
if: matrix.python-version != '3.7'
run: |
echo "Start container, override the default entrypoint"
docker run -i -d \
--name test_container \
--entrypoint /bin/bash \
$IMAGE_NAME-v$VERSION_NUMBER-squashed
echo "Install python-sc2"
docker exec -i test_container mkdir -p /root/python-sc2
docker cp pyproject.toml test_container:/root/python-sc2/
docker cp uv.lock test_container:/root/python-sc2/
docker cp sc2 test_container:/root/python-sc2/sc2
docker cp test test_container:/root/python-sc2/test
docker cp examples test_container:/root/python-sc2/examples
docker exec -i test_container bash -c "pip install uv \
&& cd python-sc2 && uv sync --frozen --no-cache --no-install-project"
echo "Run various test bots"
docker exec -i test_container bash -c "cd python-sc2 && uv run python test/travis_test_script.py test/autotest_bot.py"
docker exec -i test_container bash -c "cd python-sc2 && uv run python test/run_example_bots_vs_computer.py"

- name: Login to DockerHub
if: github.ref == 'refs/heads/develop'
uses: docker/login-action@v2
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}

- name: Upload docker image
if: github.ref == 'refs/heads/develop' && github.event_name == 'push'
run: docker push $IMAGE_NAME-v$VERSION_NUMBER

- name: Upload squashed docker image
if: github.ref == 'refs/heads/develop' && github.event_name == 'push'
run: docker push $IMAGE_NAME-v$VERSION_NUMBER-squashed

- name: Upload squashed docker image as latest tag
if: github.ref == 'refs/heads/develop' && github.event_name == 'push' && matrix.python-version == env.LATEST_PYTHON_VERSION && matrix.sc2-version == env.LATEST_SC2_VERSION
run: |
docker tag $IMAGE_NAME-v$VERSION_NUMBER-squashed burnysc2/python-sc2-docker:latest
docker push burnysc2/python-sc2-docker:latest
7 changes: 3 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,10 +26,9 @@ pip install --upgrade burnysc2
```
or directly from develop branch:
```
pip install poetry
pip install --upgrade --force-reinstall https://github.com/BurnySc2/python-sc2/archive/develop.zip
```
Both commands will use the `sc2` library folder, so you will not be able to have Dentosal's and this fork installed at the same time, unless you use virtual environments or poetry.
Both commands will use the `sc2` library folder, so you will not be able to have Dentosal's and this fork installed at the same time, unless you use virtual environments.

## StarCraft II
You'll need a StarCraft II executable. If you are running Windows or macOS, just install SC2 from [blizzard app](https://starcraft2.com/).
Expand Down Expand Up @@ -186,6 +185,6 @@ Git commit messages use [imperative-style messages](https://stackoverflow.com/a/

To run pre-commit hooks (which run autoformatting and autosort imports) you can run
```sh
poetry run pre-commit install
poetry run pre-commit run --all-files --hook-stage push
uv run pre-commit install
uv run pre-commit run --all-files --hook-stage push
```
5 changes: 2 additions & 3 deletions dockerfiles/Dockerfile
Original file line number Diff line number Diff line change
@@ -1,9 +1,8 @@
# Set up StarCraft II Test Environment for python-sc2 bots (not pysc2 bots!)
ARG PYTHON_VERSION=3.10

# Use an official debian stretch slim release as a base image
FROM python:$PYTHON_VERSION-slim
MAINTAINER Burny <[email protected]>
# https://docs.astral.sh/uv/guides/integration/docker/#available-images
FROM ghcr.io/astral-sh/uv:python$PYTHON_VERSION-bookworm-slim

ARG SC2_VERSION=4.10

Expand Down
32 changes: 19 additions & 13 deletions dockerfiles/test_docker_image.sh
Original file line number Diff line number Diff line change
Expand Up @@ -17,11 +17,11 @@ BUILD_ARGS="--build-arg PYTHON_VERSION=$PYTHON_VERSION --build-arg SC2_VERSION=$
# Allow image squashing by enabling experimental docker features
# https://stackoverflow.com/a/21164441/10882657
# https://github.com/actions/virtual-environments/issues/368#issuecomment-582387669
file=/etc/docker/daemon.json
if [ ! -e "$file" ]; then
echo $'{\n "experimental": true\n}' | sudo tee /etc/docker/daemon.json
sudo systemctl restart docker.service
fi
# file=/etc/docker/daemon.json
# if [ ! -e "$file" ]; then
# echo $'{\n "experimental": true\n}' | sudo tee /etc/docker/daemon.json
# sudo systemctl restart docker.service
# fi

# Build image without context
# https://stackoverflow.com/a/54666214/10882657
Expand All @@ -33,21 +33,27 @@ docker build -t $IMAGE_NAME $BUILD_ARGS - < dockerfiles/Dockerfile
docker rm -f test_container

# Start container, override the default entrypoint
# https://docs.docker.com/storage/bind-mounts/#use-a-read-only-bind-mount
docker run -i -d \
--name test_container \
--volume ./:/root/python-sc2:ro \
--entrypoint /bin/bash \
$IMAGE_NAME

docker exec -i test_container mkdir -p /root/python-sc2
docker cp pyproject.toml test_container:/root/python-sc2/
docker cp uv.lock test_container:/root/python-sc2/
docker cp sc2 test_container:/root/python-sc2/sc2
docker cp test test_container:/root/python-sc2/test

# Install python-sc2, via mount the python-sc2 folder will be available
docker exec -i test_container bash -c "pip install poetry \
&& cd python-sc2 && poetry install --no-dev"
docker exec -i test_container bash -c "pip install uv && cd python-sc2 && uv sync --no-cache --no-install-project"

# Run various test bots
docker exec -i test_container bash -c "cd python-sc2 && poetry run python test/travis_test_script.py test/autotest_bot.py"
docker exec -i test_container bash -c "cd python-sc2 && poetry run python test/travis_test_script.py test/queries_test_bot.py"
#docker exec -i test_container bash -c "cd python-sc2 && poetry run python test/travis_test_script.py test/damagetest_bot.py"
docker exec -i test_container bash -c "cd python-sc2 && poetry run python test/run_example_bots_vs_computer.py"
docker exec -i test_container bash -c "cd python-sc2 && uv run python test/travis_test_script.py test/autotest_bot.py"
docker exec -i test_container bash -c "cd python-sc2 && uv run python test/travis_test_script.py test/queries_test_bot.py"
#docker exec -i test_container bash -c "cd python-sc2 && uv run python test/travis_test_script.py test/damagetest_bot.py"

docker cp examples test_container:/root/python-sc2/examples
docker exec -i test_container bash -c "cd python-sc2 && uv run python test/run_example_bots_vs_computer.py"

# Command for entering the container to debug if something went wrong:
# docker exec -it test_container bash
8 changes: 4 additions & 4 deletions dockerfiles/test_new_python_candidate.sh
Original file line number Diff line number Diff line change
Expand Up @@ -30,9 +30,9 @@ docker run -i -d \


# Install python-sc2, via mount the python-sc2 folder will be available
docker exec -i test_container bash -c "pip install poetry \
&& cd python-sc2 && poetry install --no-dev"
docker exec -i test_container bash -c "pip install uv \
&& cd python-sc2 && uv sync --frozen --no-cache --no-install-project"

# Run various test bots
docker exec -i test_container bash -c "cd python-sc2 && poetry run python test/travis_test_script.py test/autotest_bot.py"
docker exec -i test_container bash -c "cd python-sc2 && poetry run python test/run_example_bots_vs_computer.py"
docker exec -i test_container bash -c "cd python-sc2 && uv run python test/travis_test_script.py test/autotest_bot.py"
docker exec -i test_container bash -c "cd python-sc2 && uv run python test/run_example_bots_vs_computer.py"
4 changes: 2 additions & 2 deletions docs_generate/text_files/docker.rst
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ The following command launches a new container in interactive mode, which means

Install bot requirements
-------------------------
The command ``docker exec -i app poetry add "burnysc2>=0.12.12"`` installs the ``burnysc2`` dependencies in the docker container. Add more libraries as needed. You can also create your custom docker image so you do not have to re-install the dependencies every time you create a new container.
The command ``docker exec -i app uv add "burnysc2>=0.12.12"`` installs the ``burnysc2`` dependencies in the docker container. Add more libraries as needed. You can also create your custom docker image so you do not have to re-install the dependencies every time you create a new container.

Since the linux SC2 binary is usually outdated (last update as of this writing was summer of 2019), you will likely have to replace your IDs with older IDs, which can be found here: https://github.com/BurnySc2/python-sc2/tree/linux-4.10/sc2/ids

Expand Down Expand Up @@ -62,7 +62,7 @@ Running the match(es)
---------------------
Now you are ready to let docker run your matches (headless)::

docker exec -i app poetry run python /root/aiarena-client/arenaclient/run_local.py
docker exec -i app uv run python /root/aiarena-client/arenaclient/run_local.py

Copying the replay from container to host machine
--------------------------------------------------------------
Expand Down
1 change: 0 additions & 1 deletion docs_generate/text_files/introduction.rst
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,6 @@ Install through pip using ``pip install burnysc2`` if Python is in your environm

Alternatively (of if the command above doesn't work) you can install a specific branch directly from github, here the develop branch::

pip install poetry
pip install --upgrade --force-reinstall https://github.com/BurnySc2/python-sc2/archive/develop.zip

Creating a bot
Expand Down
Loading
Loading