Welcome to the FRX Challenges project! This is an open source repository that provides key software components for running competitive data science challenges.
FRX stands for Frictionless Reproducibility eXchange. Inspired by Donoho, 2023. This project enables communities to host data challenges with live evaluation. It is designed to be run either on local infrastructure, or via cloud-hosted infrastructure.
This is a young project that is under active development. Below are the core workflows that we wish to support:
- prompts: Allow organizers to create a website that describes the data challenge and provides instructions for participants.
- submissions: Allow one or more participants to create one or more submissions for evaluation.
- evaluators: Leverage cloud infrastructure to run submissions against standardized environments and datasets, and allow organizers to define their own evaluation scripts, criteria, and metrics.
- feedback: Provide information to participants about how their submissions scored relative to others.
We aim for this project to work locally as well as via cloud infrastructure, so that users may prototype and make submissions via their own hardware and environments, or via community-hosted infrastructure like JupyterHub and BinderHub.
This project was built in collaboration with the HHMI CellMap Segmentation challenge, which funded its original development. Our goal is to generalize the infrastructure that enabled this challenge to be used for other communities, datasets, and workflows.
It builds heavily upon the Jupyter ecosystem and is designed to be interoperable with community-based cloud infrastructure like JupyterHub and BinderHub.
It is currently developed and maintained by 2i2c, a non-profit dedicated to providing communities with interactive computing infrastructure to create and share knowledge.
This repository contains the core software that powers the FRX Challenges platform. It consists of a Django application that is meant to leverage cloud infrastructure as part of the evaluation system.
The Helm chart lets a user create a reproducible and maintainable deployment of FRX Challenges on a Kubernetes cluster in a cloud environment. The released charts are made available in our Helm chart repository.