Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add k8s launcher #352

Open
wants to merge 76 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
76 commits
Select commit Hold shift + click to select a range
3be037b
add lazyllm before group
Aug 8, 2024
9337e4c
Merge remote-tracking branch 'upstream/main'
Aug 8, 2024
67f086c
Merge remote-tracking branch 'upstream/main'
Aug 9, 2024
dbf3280
Merge remote-tracking branch 'upstream/main'
Aug 12, 2024
d2421d2
Merge remote-tracking branch 'upstream/main'
Aug 13, 2024
b1169fe
Merge remote-tracking branch 'upstream/main'
Aug 14, 2024
2d73df9
Merge remote-tracking branch 'upstream/main'
Aug 14, 2024
e3193b1
Merge remote-tracking branch 'upstream/main'
Aug 14, 2024
80fc6c5
Merge remote-tracking branch 'upstream/main'
Aug 14, 2024
8a1b54b
Merge remote-tracking branch 'upstream/main'
Aug 15, 2024
ef51ad1
Merge remote-tracking branch 'upstream/main'
Aug 16, 2024
3d39b4e
Merge remote-tracking branch 'upstream/main'
Aug 19, 2024
63acf1a
Merge remote-tracking branch 'upstream/main'
Aug 20, 2024
5d8be06
Merge remote-tracking branch 'upstream/main'
Aug 20, 2024
a0cdd74
Merge remote-tracking branch 'upstream/main'
Aug 23, 2024
c44f5ec
Merge remote-tracking branch 'upstream/main'
Aug 23, 2024
5093af2
Merge remote-tracking branch 'upstream/main'
Aug 24, 2024
214ec1b
Merge remote-tracking branch 'upstream/main'
Aug 28, 2024
2224adc
Merge remote-tracking branch 'upstream/main'
Aug 29, 2024
ec59048
Merge remote-tracking branch 'upstream/main'
Aug 29, 2024
92e9c6b
Merge remote-tracking branch 'upstream/main'
Aug 30, 2024
e079af8
Merge remote-tracking branch 'upstream/main'
Aug 30, 2024
02892f2
Merge remote-tracking branch 'upstream/main'
Aug 30, 2024
9348f06
Merge remote-tracking branch 'upstream/main'
Aug 30, 2024
3bd6c6a
Merge remote-tracking branch 'upstream/main'
Sep 2, 2024
e11d29d
Merge remote-tracking branch 'upstream/main'
Sep 2, 2024
1494288
Merge remote-tracking branch 'upstream/main'
Sep 3, 2024
2a9c4fe
Merge remote-tracking branch 'upstream/main'
Sep 3, 2024
61cfb0c
Merge remote-tracking branch 'upstream/main'
Sep 3, 2024
4558336
Merge remote-tracking branch 'upstream/main'
Sep 4, 2024
407b84e
Merge remote-tracking branch 'upstream/main'
Sep 4, 2024
ce3e287
Merge remote-tracking branch 'upstream/main'
Sep 5, 2024
4546f52
Merge remote-tracking branch 'upstream/main'
Sep 5, 2024
6357946
Merge remote-tracking branch 'upstream/main'
Sep 10, 2024
306c115
Merge remote-tracking branch 'upstream/main'
Sep 12, 2024
9a92680
Merge remote-tracking branch 'upstream/main'
Sep 13, 2024
706a734
Merge remote-tracking branch 'upstream/main'
Sep 14, 2024
ab90f96
Merge remote-tracking branch 'upstream/main'
Sep 14, 2024
66ef831
Merge remote-tracking branch 'upstream/main'
Sep 14, 2024
015591d
Merge remote-tracking branch 'upstream/main'
Sep 18, 2024
51e5b4d
Merge remote-tracking branch 'upstream/main'
Sep 18, 2024
7b89394
Merge remote-tracking branch 'upstream/main'
Sep 20, 2024
c0ffa78
Merge remote-tracking branch 'upstream/main'
Sep 23, 2024
d2a86ab
Merge remote-tracking branch 'upstream/main'
Sep 24, 2024
b307398
Merge remote-tracking branch 'upstream/main'
Sep 25, 2024
1fd7bad
Merge remote-tracking branch 'upstream/main'
Sep 26, 2024
1c89278
Merge remote-tracking branch 'upstream/main'
Sep 26, 2024
41f11f8
Merge remote-tracking branch 'upstream/main'
Sep 27, 2024
b4dc5ac
Merge remote-tracking branch 'upstream/main'
Sep 27, 2024
e6cef20
Merge remote-tracking branch 'upstream/main'
Oct 8, 2024
97ae7e3
Merge remote-tracking branch 'upstream/main'
Oct 21, 2024
8a5a3c6
Merge remote-tracking branch 'upstream/main'
Oct 22, 2024
b026348
Merge remote-tracking branch 'upstream/main'
Oct 28, 2024
8956e9b
Merge remote-tracking branch 'upstream/main'
Nov 1, 2024
a30ee05
Merge remote-tracking branch 'upstream/main'
Nov 19, 2024
2fc5666
Merge remote-tracking branch 'upstream/main'
Nov 20, 2024
da52fdc
add kubernetes launcher
Nov 20, 2024
3d6fd0f
Delete comments
Nov 20, 2024
e779b63
1、modify launcher config file; 2、add test units
Nov 20, 2024
903e7d9
Merge remote-tracking branch 'upstream/main'
Nov 25, 2024
ccf8007
Merge branch 'main' into add_k8s_launcher
Nov 25, 2024
a3a84bb
k8s launcher supports nfs mounting
Nov 27, 2024
98b0ee8
modify kubernetes package import path
Dec 9, 2024
fc75ba4
modify k8s launcher for ecs
Dec 13, 2024
5b0cd77
add k8s test units
Dec 16, 2024
46f8026
modify k8s test yaml
Dec 17, 2024
e31aea4
add shell config for k8s test
Dec 17, 2024
91368bd
add install requirments
Dec 17, 2024
0190105
modify k8s_test.yaml
Dec 17, 2024
d83b638
add debug code for k8s test
Dec 17, 2024
3050511
config launcher volume path
Dec 17, 2024
dde1576
del sleep for debug
Dec 17, 2024
cc0a08e
modify k8s test yaml
Dec 18, 2024
eae2709
delete the command to print cmd
Dec 18, 2024
f315c65
Merge remote-tracking branch 'upstream/main'
Dec 18, 2024
1f960cb
Merge branch 'main' into add_k8s_launcher
Dec 18, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
65 changes: 65 additions & 0 deletions .github/workflows/k8s_test.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
name: Test on k8s

on:
workflow_dispatch:
push:
branches:
- main
pull_request:
branches:
- main
paths-ignore:
- "**.md"

env:
CI_PATH: '/mnt/nfs_share/GitHub/${{ github.repository }}/${GITHUB_RUN_NUMBER}'

concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true

jobs:
k8s_tests:
runs-on: tps_k8s

container:
image: registry.cn-sh-01.sensecore.cn/ai-expert-service/lazyllm_ci:latest
options: --mount type=bind,source=/mnt/nfs_share,target=/mnt/nfs_share --mount type=bind,source=/mnt/lustre/share_data,target=/mnt/lustre/share_data

steps:
- name: Checkout code
uses: actions/checkout@v4

- name: Create custom directory
run: |
set -ex
echo ${{ env.CI_PATH }}
mkdir -p ${{ env.CI_PATH }}

- name: Clean custom directory
run: |
set -ex
if [ -d "${{ env.CI_PATH }}" ]; then
rm -rf ${{ env.CI_PATH }}/*
fi

- name: Move code to custom directory
run: |
set -ex
mv $GITHUB_WORKSPACE/* ${{ env.CI_PATH }}/

- name: k8s_test
shell: bash
run: |
cd ${{ env.CI_PATH }}
source activate lazyllm
pip install -r tests/requirements.txt
export PYTHONPATH=$PWD:$PYTHONPATH
export LAZYLLM_DEFAULT_LAUNCHER="k8s"
export LAZYLLM_DATA_PATH="/mnt/lustre/share_data/lazyllm/data"
export LAZYLLM_K8S_ENV_NAME="lazyllm"
export LAZYLLM_K8S_CONFIG_PATH="/mnt/nfs_share/k8s_config.yaml"
export LAZYLLM_HOME="${{ env.CI_PATH }}/${{ github.run_id }}-${{ github.job }}"
mkdir -p $LAZYLLM_HOME
source /mnt/nfs_share/env.sh
python -m pytest --lf --last-failed-no-failures=all --durations=0 --reruns=2 -v tests/k8s_tests
Loading