Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WIP fix CI #180

Draft
wants to merge 5 commits into
base: dev
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion .github/workflows/checks.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,8 @@ jobs:
uses: actions/setup-python@v4
with:
python-version: "3.10"
- name: Pin pip version
run: python -m pip install --upgrade "pip<24.1"
- name: Install requirements
run: |
pip install .
Expand All @@ -40,12 +42,14 @@ jobs:
uses: actions/setup-python@v4
with:
python-version: "3.10"
- name: Pin pip version
run: python -m pip install --upgrade "pip<24.1"
- name: Install requirements
run: |
pip install .
pip install -r requirements_dev.txt
- name: Deploy app
run: docker-compose up -d --build
run: docker compose up -d --build
- name: Wait for app startup
run: sleep 20
- name: Run integration tests
Expand Down
17 changes: 7 additions & 10 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ environments composed of GA4GH TES nodes.

proTES gateway may serve as a crucial component in federated compute networks
based on the GA4GH Cloud ecosystem. Its primary purpose is to provide
centralized features to a federated network of independently operated GA4GH TES
centralized features to a federated network of independently operated GA4GH TES
instances. As such, it can serve, for example, as a compatibility layer, a load
balancer workload distribution layer, a public entry point to an enclave of
independent compute nodes, or a means of collecting telemetry.
Expand All @@ -38,9 +38,9 @@ proof-of-concept examples for different task distribution scenarios:
evenly (actually: randomly!) distributes workloads across a network of TES
endpoints
* **Bringing compute to the data**: The
`pro_tes.middleware.task_distribution.distance` plugin selects TES endpoints
`pro_tes.middleware.task_distribution.distance` plugin selects TES endpoints
to relay incoming requests to in such a way that the distance the (input) data
of a task has to travel across the network of TES endpoints is minimized.
of a task has to travel across the network of TES endpoints is minimized.

### Implementation notes

Expand Down Expand Up @@ -76,7 +76,7 @@ Ensure you have the following software installed:

### Prerequisites

Create data directory and required subdiretories
Create data directory and required subdirectories

```bash
export PROTES_DATA_DIR=/path/to/data/directory
Expand Down Expand Up @@ -135,7 +135,7 @@ interactions with the community.

## Versioning

The project adopts the [semantic versioning][semver] scheme for versioning.
The project adopts the [semantic versioning][res-sem-ver] scheme for versioning.
Currently the service is in beta stage, so the API may change and even break
without further notice. However, once we deem the service stable and "feature
complete", the major, minor and patch version will shadow the supported TES
Expand Down Expand Up @@ -174,22 +174,19 @@ thread in our [Q&A forum][contact-qa], or send us an [email][contact-email].
[docs-contributing]: <https://elixir-cloud-aai.github.io/guides/guide-contributor/>
[docs-deploy]: deployment/README.md
[docs-license]: LICENSE
[GA4GH TES OpenAPI specification]:<https://github.com/ga4gh/task-execution-schemas>
[image-protes-overview]: <images/overview.svg>
[res-celery]: <http://www.celeryproject.org/>
[res-connexion]: <https://github.com/zalando/connexion>
[res-docker]: <https://www.docker.com/>
[res-docker-compose]: <https://docs.docker.com/compose/>
[res-elixir-cloud-aai]: <https://elixir-cloud.dcc.sib.swiss/>
[res-flask]: <http://flask.pocoo.org/>
[res-foca]: <https://github.com/elixir-cloud-aai/foca>
[res-ga4gh]: <https://www.ga4gh.org/>
[res-ga4gh-cloud]: <https://www.ga4gh.org/work_stream/cloud/>
[res-ga4gh-tes]: <https://github.com/ga4gh/task-execution-schemas>
[res-git]: <https://git-scm.com/>
[res-helm]: <https://helm.sh/>
[res-kubernetes]: <https://kubernetes.io/>
[res-mondodb]: <https://www.mongodb.com/>
[res-ouath2]: <https://oauth.net/2/>
[res-mongodb]: <https://www.mongodb.com/>
[res-oauth2]: <https://oauth.net/2/>
[res-rabbitmq]: <https://www.rabbitmq.com/>
[res-sem-ver]: <https://semver.org/>
4 changes: 2 additions & 2 deletions pro_tes/ga4gh/tes/models.py
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ class TesCreateTaskResponse(CustomBaseModel):

class TesExecutor(CustomBaseModel):
image: str = Field(
default=[""],
default="",
description=(
"Name of the container image. The string will be passed as "
" the image\nargument to the containerization run command. "
Expand Down Expand Up @@ -626,7 +626,7 @@ class TesTask(CustomBaseModel):
)
resources: Optional[TesResources] = None
executors: list[TesExecutor] = Field(
default=[TesExecutor],
default=[TesExecutor()],
description=(
"An array of executors to be run. Each of the executors "
" will run one\nat a time sequentially. Each executor is a"
Expand Down
2 changes: 1 addition & 1 deletion pro_tes/gunicorn.py
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@
forwarded_allow_ips = "*" # pylint: disable=invalid-name

# Set Gunicorn bind address
bind = f"{app_config.server.host}:{app_config.server.port}"
BIND = f"{app_config.server.host}:{app_config.server.port}"

# Source environment variables for Gunicorn workers
raw_env = [
Expand Down
7 changes: 5 additions & 2 deletions pro_tes/tasks/track_task_progress.py
Original file line number Diff line number Diff line change
Expand Up @@ -20,14 +20,17 @@

# pylint: disable-msg=too-many-locals
# pylint: disable=unsubscriptable-object
# pylint: disable=too-many-arguments
# pylint: disable=too-many-positional-arguments
# pylint: disable=unused-argument
@celery.task(
name="tasks.track_run_progress",
bind=True,
ignore_result=True,
track_started=True,
)
def task__track_task_progress( # pylint: disable=too-many-arguments
self, # pylint: disable=unused-argument
def task__track_task_progress(
self,
worker_id: str,
remote_host: str,
remote_base_path: str,
Expand Down
2 changes: 1 addition & 1 deletion requirements.txt
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
celery-types>=0.20.0
connexion>=2.11.2,<3
foca>=0.12.1
foca==0.12.1
geopy>=2.2.0
gunicorn>=20.1.0,<21
ip2geotools>=0.1.6
Expand Down
Loading