Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Set up GitHub Actions Workflow for Testing Parsl with Flux #3159

Merged
merged 73 commits into from
Jun 10, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
73 commits
Select commit Hold shift + click to select a range
b3bcdef
Added flux ci
mercybassey Mar 6, 2024
1b6e399
removed tests
mercybassey Mar 6, 2024
5c7a6c0
configured the CI to install Parsl
mercybassey Mar 6, 2024
2bf4de1
Merge branch 'master' into parsl+flux
mercybassey Mar 6, 2024
a5fbe78
Merge branch 'master' into parsl+flux
mercybassey Mar 6, 2024
b1eda08
Install checked-out version of Parsl in CI
mercybassey Mar 6, 2024
9fa40ef
Merge remote-tracking branch 'origin/parsl+flux' into parsl+flux
mercybassey Mar 6, 2024
efacad0
Add basic Parsl verification to CI
mercybassey Mar 6, 2024
177944c
Fixed indentation
mercybassey Mar 6, 2024
f714af6
Add step to install pytest
mercybassey Mar 6, 2024
a71e86c
Installed pytest and pytest-random-order for CI tests
mercybassey Mar 6, 2024
8447148
Install dependencies from test-requirements.txt for CI tests
mercybassey Mar 6, 2024
54d21e8
Adjust CI test command to resolve pytest failure
mercybassey Mar 6, 2024
d8b073b
Merge branch 'master' into parsl+flux
mercybassey Mar 6, 2024
15d280e
Combining steps
mercybassey Mar 6, 2024
f1c8105
Merge branch 'parsl+flux' of github.com:mercybassey/parsl into parsl+…
mercybassey Mar 6, 2024
961561c
Added --random-order
mercybassey Mar 6, 2024
a2f0923
Configured CI to install python3-dev
mercybassey Mar 7, 2024
fe5921e
Added a test for writing to non-writable directory
mercybassey Mar 7, 2024
c934f15
Merge branch 'master' into parsl+flux
mercybassey Mar 7, 2024
89c294f
Added logging in the test
mercybassey Mar 7, 2024
00741ba
Merge branch 'parsl+flux' of github.com:mercybassey/parsl into parsl+…
mercybassey Mar 7, 2024
636a1ce
Merge branch 'master' into parsl+flux
mercybassey Mar 7, 2024
a9d8afa
Merge branch 'master' into parsl+flux
mercybassey Mar 7, 2024
e696cc0
commented out failing test
mercybassey Mar 8, 2024
c44adfc
Merge branch 'parsl+flux' of github.com:mercybassey/parsl into parsl+…
mercybassey Mar 8, 2024
bb03950
Merge branch 'master' into parsl+flux
mercybassey Mar 8, 2024
c79c0ec
commented out failing tests
mercybassey Mar 8, 2024
48d4e9a
fixed flake8 issue
mercybassey Mar 8, 2024
d15ee18
Added a step to test Parsl with flux
mercybassey Mar 8, 2024
778fc8f
Added local_threads.py config
mercybassey Mar 8, 2024
542e74a
Configured the CI to start flux, before running tests
mercybassey Mar 8, 2024
e6db9b9
Streamlined parl+flux in CI
mercybassey Mar 8, 2024
9916252
Reconfigured the 'Start Flux and Test Parsl with Flux'
mercybassey Mar 8, 2024
15f0763
Added verbosity since the run doesn't exits
mercybassey Mar 8, 2024
3bd14ad
removed concurrency for CI
mercybassey Mar 8, 2024
a1b9923
Added timeout of 5minutes and without any verbosity flag
mercybassey Mar 8, 2024
4536aa9
Running CI with verbosity level -v
mercybassey Mar 8, 2024
668caaa
Running CI with verbosity level -vvv
mercybassey Mar 8, 2024
939f79f
Removed the std.out and test.memo.stdout.x files
mercybassey Mar 8, 2024
e66b069
Removed the std.out and test.memo.stdout.x files
mercybassey Mar 8, 2024
627dccc
fixed flux test
mercybassey Mar 11, 2024
8401356
Merge branch 'master' into parsl+flux
mercybassey Mar 12, 2024
3a5628f
running ci again
mercybassey Mar 12, 2024
448ae1f
fix makefile typo
mercybassey Mar 12, 2024
0b8676f
Added flux test config
mercybassey Mar 12, 2024
c9bab6b
minor
mercybassey Mar 12, 2024
45b4032
fixed flake8 error and seperated the flux+parsl test in ci
mercybassey Mar 12, 2024
ee3af80
Merge branch 'master' into parsl+flux
mercybassey Mar 12, 2024
d661fc1
running tests
mercybassey Mar 12, 2024
791c268
Merge branch 'master' into parsl+flux
mercybassey Mar 12, 2024
9df54fc
Running tests
mercybassey Mar 12, 2024
8bc0bfd
Merge branch 'parsl+flux' of github.com:mercybassey/parsl into parsl+…
mercybassey Mar 12, 2024
0903435
fixed flake8 error
mercybassey Mar 12, 2024
2583075
remove a mistake in the makefile
mercybassey Mar 18, 2024
b514001
added a comment at 'flux_local_test'
mercybassey Mar 18, 2024
b2b96fd
Merge branch 'master' into parsl+flux
benclifford Mar 18, 2024
c7761aa
Merge branch 'master' into parsl+flux
mercybassey Mar 18, 2024
91d1539
took test_stdout.py to how it was and corrected ci
mercybassey Mar 18, 2024
215a035
Omitted tests marked as as well in 'Test Parsl with Flux Config'
mercybassey Mar 18, 2024
3bb44d3
Merge branch 'master' into parsl+flux
benclifford Mar 18, 2024
155d8e1
removed unwanted files
mercybassey Mar 18, 2024
d03cd52
Merge branch 'parsl+flux' of github.com:mercybassey/parsl into parsl+…
mercybassey Mar 18, 2024
f332bb1
Merge branch 'master' into parsl+flux
benclifford Jun 10, 2024
0662487
Remove changes to .gitignore
benclifford Jun 10, 2024
f5d1630
Use 1 line between targets, like elsewhere
benclifford Jun 10, 2024
d7312d1
mark tests that need unix filesystem permissions enforced, which is n…
benclifford Jun 10, 2024
b4d5d85
Another bad file test that requires unix fs permission enforcement
benclifford Jun 10, 2024
882f0e8
flake8
benclifford Jun 10, 2024
4167a42
Increase workflow timeout - it was being reached in normal operation
benclifford Jun 10, 2024
26c29e5
Increase timeout more - hit 10 minutes and i'd like to see what kind …
benclifford Jun 10, 2024
49f210f
Upgrade base ubuntu to jammy
benclifford Jun 10, 2024
e990f08
Merge branch 'master' into parsl+flux
benclifford Jun 10, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
47 changes: 47 additions & 0 deletions .github/workflows/parsl+flux.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
name: Test Flux Scheduler
on:
pull_request: []

jobs:
build:
runs-on: ubuntu-22.04
permissions:
packages: read
strategy:
fail-fast: false
matrix:
container: ['fluxrm/flux-sched:jammy']
timeout-minutes: 30

container:
image: ${{ matrix.container }}
options: "--platform=linux/amd64 --user root -it --init"

name: ${{ matrix.container }}
steps:
- name: Make Space
run: |
rm -rf /usr/share/dotnet
rm -rf /opt/ghc

- name: Checkout
uses: actions/checkout@v3

- name: Install Dependencies and Parsl
run: |
apt-get update && apt-get install -y python3-pip curl
pip3 install . -r test-requirements.txt

- name: Verify Parsl Installation
run: |
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In case you don't know, for single run lines you don't need the pipe, and can just do:

run: make local_thread_test

It's just visual, but makes the file a little more tidy.

pytest parsl/tests/ -k "not cleannet and not unix_filesystem_permissions_required" --config parsl/tests/configs/local_threads.py --random-order --durations 10

- name: Start Flux and Test Parsl with Flux
run: |
flux start pytest parsl/tests/test_flux.py --config local --random-order

- name: Test Parsl with Flux Config
run: |
flux start pytest parsl/tests/ -k "not cleannet and not unix_filesystem_permissions_required" --config parsl/tests/configs/flux_local.py --random-order --durations 10


5 changes: 5 additions & 0 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -127,3 +127,8 @@ coverage: ## show the coverage report
.PHONY: clean
clean: ## clean up the environment by deleting the .venv, dist, eggs, mypy caches, coverage info, etc
rm -rf .venv $(DEPS) dist *.egg-info .mypy_cache build .pytest_cache .coverage runinfo $(WORKQUEUE_INSTALL)

.PHONY: flux_local_test
flux_local_test: ## Test Parsl with Flux Executor
pip3 install .
pytest parsl/tests/ -k "not cleannet" --config parsl/tests/configs/flux_local.py --random-order --durations 10
11 changes: 11 additions & 0 deletions parsl/tests/configs/flux_local.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
from parsl.config import Config
from parsl.executors import FluxExecutor


def fresh_config():
return Config(
executors=[FluxExecutor()],
)


config = fresh_config()
Loading