Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: switch to Github Actions #1933

Merged
merged 33 commits into from
Aug 6, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
33 commits
Select commit Hold shift + click to select a range
4410d5a
add github actions
LexLuthr Jun 25, 2024
728ffa3
fix attempt 1
LexLuthr Jun 25, 2024
fe15332
fix attempt 2
LexLuthr Jun 25, 2024
d334e34
fix attempt 3
LexLuthr Jun 25, 2024
8aa3640
fix attempt 4
LexLuthr Jun 25, 2024
4a9e544
fix attempt 5
LexLuthr Jun 25, 2024
1f8a396
fix attempt 6
LexLuthr Jun 25, 2024
bd939b7
fix attempt 7
LexLuthr Jun 25, 2024
d3f3163
fix attempt 8
LexLuthr Jun 25, 2024
ee7494f
fix attempt 9
LexLuthr Jun 25, 2024
539436a
fix attempt 10
LexLuthr Jun 25, 2024
a081aa7
fix attempt 11
LexLuthr Jun 25, 2024
7708efb
fix attempt 12
LexLuthr Jun 25, 2024
a319107
fix attempt 13
LexLuthr Jun 25, 2024
1140845
fix attempt 14
LexLuthr Jun 25, 2024
2d072b1
fix attempt 15
LexLuthr Jun 25, 2024
5514baa
fix attempt 16
LexLuthr Jun 25, 2024
2d200a9
fix attempt 17
LexLuthr Jun 25, 2024
5a60dc6
fix lint errors
LexLuthr Jun 25, 2024
2fd9958
make deps for lint
LexLuthr Jun 25, 2024
8bf84db
fix fmt
LexLuthr Jun 25, 2024
db646bc
test self-hosted
LexLuthr Jul 12, 2024
9ee733a
self-hosted runner 1
LexLuthr Jul 26, 2024
08694da
self-hosted runner 2
LexLuthr Jul 26, 2024
1c540e2
self-hosted runner 3
LexLuthr Jul 26, 2024
f2ef486
self-hosted runner 4
LexLuthr Jul 26, 2024
5ee396b
self-hosted runner 5
LexLuthr Jul 26, 2024
b776c38
add CI linter, yugabyte cleanup
LexLuthr Jul 29, 2024
1423559
add setup files
LexLuthr Aug 6, 2024
6c88e6a
disable shellcheck
LexLuthr Aug 6, 2024
61873d0
fix systemd file
LexLuthr Aug 6, 2024
a55a73b
update setup instructions
LexLuthr Aug 6, 2024
900da6a
test without go cache
LexLuthr Aug 6, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .github/actionlint.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
self-hosted-runner:
# Labels of self-hosted runner in array of string
labels:
- docker
48 changes: 0 additions & 48 deletions .github/actions/container-builder/action.yaml

This file was deleted.

24 changes: 24 additions & 0 deletions .github/actions/install-deps/action.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
name: 'Install Dependencies'
description: 'Install common dependencies'

runs:
using: 'composite'
steps:
- name: Install dependencies
run: |
sudo apt-get update
sudo apt-get install -y openssh-client git ocl-icd-opencl-dev libhwloc-dev
shell: bash

- name: Fetch all tags
run: git fetch --all
shell: bash

- name: Sync submodules
run: git submodule sync
shell: bash

- name: Update submodules
run: git submodule update --init
shell: bash

16 changes: 16 additions & 0 deletions .github/actions/setup-go/action.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
name: 'Setup Go'
description: 'Setup Go environment'

inputs:
go-version:
description: 'Go version to use'
required: true

runs:
using: 'composite'
steps:
- name: Set up Go
uses: actions/setup-go@v5
with:
go-version: ${{ inputs.go-version }}
cache: false
74 changes: 74 additions & 0 deletions .github/image/Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,74 @@
## This image is used to github action runner for this repo
# Use the official Golang image as the base image
FROM golang:1.22-bullseye AS builder

# Set the working directory inside the container
WORKDIR /app

# Install Git
RUN apt-get update && apt-get install -y git

# Clone the repositories
RUN git clone https://github.com/filecoin-project/boost.git
RUN git clone https://github.com/filecoin-project/curio.git

# Download Go dependencies for Boost
WORKDIR /app/boost
RUN git submodule update --init
RUN go mod download

# Download Go dependencies for Curio
WORKDIR /app/boost
RUN git submodule update --init
RUN go mod download

# Stage 2: Install Lotus binary
FROM ghcr.io/filecoin-shipyard/lotus-containers:lotus-v1.28.1-devnet AS lotus-test

# Stage 3: Build the final image
FROM myoung34/github-runner AS curio-github-runner

# Copy Go dependencies from the builder stage
COPY --from=builder /go/pkg /go/pkg
COPY --from=builder /go/bin /go/bin
COPY --from=lotus-test /usr/local/bin/lotus /usr/local/bin/

RUN apt update && apt install -y \
build-essential \
bzr pkg-config \
clang \
curl \
gcc git \
hwloc \
jq \
libhwloc-dev wget \
mesa-opencl-icd \
ocl-icd-opencl-dev

ENV RUSTUP_HOME=/usr/local/rustup \
CARGO_HOME=/usr/local/cargo \
PATH=/usr/local/cargo/bin:$PATH \
RUST_VERSION=1.76.0

RUN set -eux; \
dpkgArch="$(dpkg --print-architecture)"; \
case "${dpkgArch##*-}" in \
amd64) rustArch='x86_64-unknown-linux-gnu'; rustupSha256='0b2f6c8f85a3d02fde2efc0ced4657869d73fccfce59defb4e8d29233116e6db' ;; \
armhf) rustArch='armv7-unknown-linux-gnueabihf'; rustupSha256='f21c44b01678c645d8fbba1e55e4180a01ac5af2d38bcbd14aa665e0d96ed69a' ;; \
arm64) rustArch='aarch64-unknown-linux-gnu'; rustupSha256='673e336c81c65e6b16dcdede33f4cc9ed0f08bde1dbe7a935f113605292dc800' ;; \
i386) rustArch='i686-unknown-linux-gnu'; rustupSha256='e7b0f47557c1afcd86939b118cbcf7fb95a5d1d917bdd355157b63ca00fc4333' ;; \
*) echo >&2 "unsupported architecture: ${dpkgArch}"; exit 1 ;; \
esac; \
url="https://static.rust-lang.org/rustup/archive/1.26.0/${rustArch}/rustup-init"; \
wget "$url"; \
echo "${rustupSha256} *rustup-init" | sha256sum -c -; \
chmod +x rustup-init; \
./rustup-init -y --no-modify-path --profile minimal --default-toolchain $RUST_VERSION --default-host ${rustArch}; \
rm rustup-init; \
chmod -R a+w $RUSTUP_HOME $CARGO_HOME; \
rustup --version; \
cargo --version; \
rustc --version;

# Allow attaching a volume for the specified path
VOLUME /var/tmp/filecoin-proof-parameters
7 changes: 7 additions & 0 deletions .github/utils/boost-github-actions-runner.env
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
RUNNER_SCOPE=repo
REPO_URL=https://github.com/filecoin-project/boost
LABELS=docker
ACCESS_TOKEN=<ADD TOKEN HERE>
RUNNER_WORKDIR=/tmp/runner/work
DISABLE_AUTO_UPDATE=1
EPHEMERAL=1
33 changes: 33 additions & 0 deletions .github/utils/create-runner-image.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
#!/bin/bash

# Directory where the GitHub Action runner will be set up
RUNNER_DIR="/tmp/github-runner"

# Clone the necessary repositories and set up the runner
setup_github_runner() {
# Remove any existing directory
rm -rf $RUNNER_DIR

# Create the directory
mkdir -p $RUNNER_DIR

# Navigate to the directory
# shellcheck disable=SC2164
cd $RUNNER_DIR

# Clone the required repositories
git clone https://github.com/filecoin-project/curio.git
git clone https://github.com/filecoin-project/boost.git

# Copy necessary files
cp -r boost/.github/utils/* .

# Copy the Dockerfile
cp boost/.github/image/Dockerfile .

# Build the Docker image
docker buildx build -t curio/github-runner:latest .
}

# Execute the function
setup_github_runner
7 changes: 7 additions & 0 deletions .github/utils/curio-github-actions-runner.env
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
RUNNER_SCOPE=repo
REPO_URL=https://github.com/filecoin-project/curio
LABELS=docker
ACCESS_TOKEN=<TOKEN>
RUNNER_WORKDIR=/tmp/runner/work
DISABLE_AUTO_UPDATE=1
EPHEMERAL=1
19 changes: 19 additions & 0 deletions .github/utils/github-actions-runner.service
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
[Unit]
Description=Ephemeral GitHub Actions Runner Container
After=docker.service
Requires=docker.service
[Service]
TimeoutStartSec=0
Restart=always
ExecStartPre=-/usr/bin/docker stop github-runnerNUM
ExecStartPre=-/usr/bin/docker rm github-runnerNUM
#ExecStartPre=-/usr/bin/docker pull curio/github-runner:latest
ExecStart=/usr/bin/docker run --rm \
--env-file /etc/curio-github-actions-runner.env \
-e RUNNER_NAME=docker-runnerNUM \
-v /var/run/docker.sock:/var/run/docker.sock \
-v /opt/filecoin-proof-parameters:/var/tmp/filecoin-proof-parameters \
--name github-runnerNUM \
curio/github-runner:latest
[Install]
WantedBy=multi-user.target
77 changes: 77 additions & 0 deletions .github/utils/setup.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,77 @@
## How to set up Github Action runner
1. Clone this repo and copy out the files in this directory to one level above
```shell
mkdir github-runner

cd github-runner

git clone https://github.com/filecoin-project/curio.git
git clone https://github.com/filecoin-project/boost.git

cp -r boost/.github/utils/* .
```
2. Copy the Dockerfile

```shell
copy boost/.github/image/Dockerfile .
```

3. Create new image

```shell
docker buildx build -t curio/github-runner:latest .
```

4. Create systemd file. Please ensure to equal number files for Boost and Curio. If server can host 10 runner then 5 should be for Boost and 5 for Curio.

```shell
for i in {1..5}; do cat github-actions-runner.service | sed "s/NUM/$i/g" > github-actions-runner$i.service; done
for i in {6..10}; do cat github-actions-runner.service | sed 's/curio-/boost-/g' | sed "s/NUM/$i/g" > github-actions-runner$i.service; done
for i in {1..10}; do install -m 644 github-actions-runner$i.service /etc/systemd/system/ ; done
systemctl daemon-reload
```
5. Add the token to ENV files
6. Copy the ENV files to /etc

```shell
cp boost-github-actions-runner.env
cp curio-github-actions-runner.env
```

7. Start and Enable the services
```shell
for i in {1..10}; do systemctl start github-actions-runner$i.service; done
for i in {1..10}; do systemctl status github-actions-runner$i.service; done
for i in {1..10}; do systemctl enable github-actions-runner$i.service; done
```

8. Verify that new runners are visible in the repo.

## Set up docker image creator
1. Make the script executable
```shell
cd github-runner
chmod +x create-runner-image.sh
```
2. Create a cron job to update the image every day
```shell
crontab -e
```

```shell
0 0 * * * /root/github-runner/create-runner-image.sh
```

## Github Token
Creating GitHub personal access token (PAT) for using by self-hosted runner make sure the following scopes are selected:

```text
repo (all)
admin:public_key - read:public_key
admin:repo_hook - read:repo_hook
admin:org_hook
notifications
workflow
```

## This setup is based in the https://github.com/myoung34/docker-github-actions-runner
Loading