Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add build-in-devcontainer.yaml workflow #122

Merged
merged 32 commits into from
Sep 21, 2023
Merged
Changes from all commits
Commits
Show all changes
32 commits
Select commit Hold shift + click to select a range
ba10e18
add build-in-devcontainer.yaml workflow
trxcllnt Aug 22, 2023
baf79cf
limit the devcontainers build matrix to amd64 + CUDA 12.0 + conda/pip
trxcllnt Aug 23, 2023
06e59d2
add the conents: read permission
trxcllnt Aug 31, 2023
cc06c68
move contents:read into build job
trxcllnt Aug 31, 2023
75549d5
copy permissions list from other reusable workflows
trxcllnt Aug 31, 2023
131f089
set up ssh-agent
trxcllnt Aug 31, 2023
8dd95be
install openssh-client
trxcllnt Aug 31, 2023
6ec1817
don't adjust SSH config
trxcllnt Aug 31, 2023
1cbde1a
update ssh-agent action, print git and ssh configs for debugging
trxcllnt Aug 31, 2023
62ec6b8
change to the repo dir before copying
trxcllnt Aug 31, 2023
e6d12f0
concat and move git and ssh configs into place
trxcllnt Aug 31, 2023
cf85375
set ssh dir permissions and ownership
trxcllnt Aug 31, 2023
2c7849d
start the ssh-agent inside the container if it's not mounted
trxcllnt Aug 31, 2023
f5a1379
pass in extra repo key instead of the actual secret
trxcllnt Aug 31, 2023
349e330
fix typo
trxcllnt Aug 31, 2023
6be7bfa
actually add the keys
trxcllnt Aug 31, 2023
d21f7c4
manually load ssh keys and update git and ssh configs
trxcllnt Aug 31, 2023
21f5b1d
cleanup
trxcllnt Aug 31, 2023
15e3b3c
openssh-client is unnecessary now
trxcllnt Sep 1, 2023
4445292
use an 8-core machine by default
trxcllnt Sep 1, 2023
4624d97
support pulling pip packages from our staging pypi index
trxcllnt Sep 1, 2023
8e24944
move secret to envvar
trxcllnt Sep 1, 2023
b8b02fc
inline into script again because the value is multiline
trxcllnt Sep 2, 2023
b6fa5c1
move git/ssh config logic into devcontainer-utils-init-ssh-deploy-key…
trxcllnt Sep 12, 2023
0b9da7f
clone repo into subdir, use actions/cache to cache devcontainer features
trxcllnt Sep 12, 2023
7bfa25a
make data/empty-folder
trxcllnt Sep 12, 2023
8cfbf8d
absolute path to data
trxcllnt Sep 13, 2023
436c7ca
append with tee
trxcllnt Sep 13, 2023
cdcc02b
use >>
trxcllnt Sep 13, 2023
2032857
mkdir -p first
trxcllnt Sep 13, 2023
878be9c
don't cache userDataFolder
trxcllnt Sep 13, 2023
e9029bc
Merge branch 'branch-23.10' into fea/devcontainers
trxcllnt Sep 20, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
103 changes: 103 additions & 0 deletions .github/workflows/build-in-devcontainer.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,103 @@
on:
workflow_call:
inputs:
sha:
type: string
repo:
type: string
node_type:
type: string
default: "cpu8"
build_command:
type: string
required: true
# Note that this is the _name_ of a secret containing the key, not the key itself.
extra-repo-deploy-key:
required: false
type: string
default: ''

permissions:
actions: read
checks: none
contents: read
deployments: none
discussions: none
id-token: write
issues: none
packages: read
pages: none
pull-requests: read
repository-projects: none
security-events: none
statuses: none

jobs:
build:
strategy:
fail-fast: false
matrix:
arch: ["amd64"]
cuda: ["12.0"]
pkgr: ["conda", "pip"]
runs-on: "linux-${{ matrix.arch }}-${{ inputs.node_type }}"
steps:
- uses: actions/checkout@v3
with:
repository: ${{ inputs.repo }}
ref: ${{ inputs.sha }}
fetch-depth: 0
- name: Check if repo has devcontainer
run: |
if test -f .devcontainer/cuda${{ matrix.cuda }}-${{ matrix.pkgr }}/devcontainer.json; then
echo "HAS_DEVCONTAINER=true" >> "${GITHUB_ENV}";
else
echo "HAS_DEVCONTAINER=false" >> "${GITHUB_ENV}";
fi
- if: ${{ env.HAS_DEVCONTAINER == 'true' }}
uses: aws-actions/configure-aws-credentials@v2
with:
role-to-assume: ${{ vars.AWS_ROLE_ARN }}
aws-region: ${{ vars.AWS_REGION }}
role-duration-seconds: 43200 # 12h
- if: ${{ env.HAS_DEVCONTAINER == 'true' }}
name: Setup Node.js
uses: actions/setup-node@v3
with:
node-version: '16'
- if: ${{ env.HAS_DEVCONTAINER == 'true' }}
name: Copy devcontainer.json file up one level
run: |
echo "REPOSITORY=$(basename $(pwd))" | tee -a "${GITHUB_ENV}";
cp .devcontainer/cuda${{ matrix.cuda }}-${{ matrix.pkgr }}/devcontainer.json \
.devcontainer/devcontainer.json;
- if: ${{ env.HAS_DEVCONTAINER == 'true' }}
name: Run build in devcontainer
uses: devcontainers/[email protected]
with:
push: never
env: |
REPOSITORY=${{ env.REPOSITORY }}
SCCACHE_REGION=${{ vars.AWS_REGION }}
AWS_ACCESS_KEY_ID=${{ env.AWS_ACCESS_KEY_ID }}
AWS_SESSION_TOKEN=${{ env.AWS_SESSION_TOKEN }}
AWS_SECRET_ACCESS_KEY=${{ env.AWS_SECRET_ACCESS_KEY }}
runCmd: |
set -e;

mkdir -p ~/.config/pip/;
cat <<EOF >> ~/.config/pip/pip.conf
[global]
extra-index-url = https://cibuildwheel:${{ secrets.RAPIDSAI_PYPI_CI_PASSWORD }}@pypi.k8s.rapids.ai/simple
EOF

rapids-make-${PYTHON_PACKAGE_MANAGER}-env || true;

if ! grep -qE "^$" <<< "${{ inputs.extra-repo-deploy-key }}"; then
if ! pgrep ssh-agent >/dev/null 2>&1; then eval "$(ssh-agent -s)"; fi;
ssh-add - <<< "${{ secrets[inputs.extra-repo-deploy-key] }}";
devcontainer-utils-init-ssh-deploy-keys || true;
fi

cd ~/"${REPOSITORY}";
${{ inputs.build_command }}