A utility that generates distributable items that help challenge organizers set up their challenge more easily on Grand-Challenge.org.
Install via PyPi:
pip install grand-challenge-forge
grand-challenge-forge --help
A challenge pack consists of challenge-tailored examples for the following:
- A script to automate uploading data to an archive
- A submission algorithm that can be submitted to a challenge phase
- An evaluation method that evaluates algorithm submissions and generates performance metrics for ranking
grand-challenge-forge pack pack-context.json
Will use the context found in pack-context.json
and generate a pack at the current working directory in
a directory dist/
(default).
Example of the content of pack-context.json
{
"challenge": {
"slug": "challenge-slug",
"phases": [
{
"slug": "phase-slug",
"archive": {
"url": "https://grand-challenge.org/archives/archive-slug/"
},
"algorithm_inputs": [
{
"slug": "input-ci-slug",
"kind": "Segmentation",
"super_kind": "Image",
"relative_path": "images/input-value"
},
{
"slug": "another-input-ci-slug",
"kind": "Anything",
"super_kind": "File",
"relative_path": "another-input-value.json"
}
],
"algorithm_outputs": [
{
"slug": "output-ci-slug",
"kind": "Image",
"super_kind": "Image",
"relative_path": "images/output-value"
},
{
"slug": "another-output-ci-slug",
"kind": "Anything",
"super_kind": "File",
"relative_path": "output-value.json"
}
]
},
{
"slug": "another-phase-slug",
"archive": {
"url": "https://grand-challenge.org/archives/another-archive-slug/"
},
"algorithm_inputs": [
{
"slug": "input-ci-slug",
"kind": "Image",
"super_kind": "Image",
"relative_path": "images/input-value"
}
],
"algorithm_outputs": [
{
"slug": "another-output-ci-slug",
"kind": "Anything",
"super_kind": "File",
"relative_path": "output-value.json"
}
]
}
]
}
}
Alternatively, you generate a pack by providing a JSON string directly:
grand-challenge-forge pack --output-dir /tmp '{ "challenge": { "slug": "a-slug"...'
This will output a pack directory in the /tmp
directory.
Note
By default, the forge does quality checks on the pack that may require docker.
You can disable these via -n
Via API pack generation can be done via:
from grand_challenge_forge.forge import generate_challenge_pack
from Pathlib import Path
qc = []
generate_challenge_pack(
context={"challenge": {...}}
output_path=Path("dist/"),
quality_control_registry=qc,
delete_existing=False,
)
for check in qc:
check()
grand-challenge-forge algorithm algorithm-context.json
Will use the context found in algorithm-context.json
and generate a algorith-template directory at the current working directory in a directory dist/
(default).
Example of the content of algorithm-context.json
{
"algorithm": {
"title": "An Algorithm Title",
"slug": "an-algorithm-title",
"url": "https://grand-challenge.org/algorithms/an-algorithm/",
"inputs": [
{
"slug": "input-ci-slug",
"kind": "Segmentation",
"super_kind": "Image",
"relative_path": "images/input-value"
},
{
"slug": "another-input-ci-slug",
"kind": "Anything",
"super_kind": "File",
"relative_path": "another-input-value.json"
}
],
"outputs": [
{
"slug": "output-ci-slug",
"kind": "Image",
"super_kind": "Image",
"relative_path": "images/output-value"
},
{
"slug": "another-output-ci-slug",
"kind": "Anything",
"super_kind": "File",
"relative_path": "output-value.json"
}
]
},
}
Alternatively, you generate an algorithm template by providing a JSON string directly:
grand-challenge-forge algorithm --output-dir /tmp '{ "algorithm": { ... } }'
This will output an algorithm-template directory in the /tmp
directory.
Note
By default, the forge does quality checks on the template that may require docker.
You can disable these via -n
Via API the algorithm-template generation can be done via:
from grand_challenge_forge.forge import generate_algorithm_template
from Pathlib import Path
qc = []
generate_algorithm_template(
context={"algorithm": { ... }}
output_path=Path("dist/"),
quality_control_registry=qc,
delete_existing=False,
)
for check in qc:
check()
Install grand-challenge-forge locally (requires poetry
):
git clone https://github.com/DIAGNijmegen/rse-grand-challenge-forge.git
cd rse-grand-challenge-forge
poetry install
poetry run grand-challenge-forge --help
Several linters and stylers run to check the formatting during continuous integration. Ensure they are run before committing by installing pre-commit.
use tox
to run all tests across all supported python versions:
pip install tox
tox
Under the hood grand-challenge-forge uses: