Migrating v1.5 competition to v2

In this example I will be migrating this Iris competition to v2: https://github.com/madclam/m2aic2019

Baseline

I first made sure the bundle I was working with worked on v1.5 by uploading the bundle produced by make_bundle then making a submission included with the Iris example. It worked.

Then I downloaded Pisano period v2 competition here: https://github.com/codalab/competition-examples/tree/master/v2/pisano_period

And made sure that worked in a similar way, using the task + solution provided.

How a v1.5 program runs on compute worker

How a v2 program runs on compute worker

Convert the YAML

Overall

I went through the old YAML and many fields are not implemented yet, for example:

• force_submission_to_leaderboard
• disallow_leaderboard_modifying
• etc.

I commented these out for now, we can actually use this competition to test them later!

HTML

HTML section has a new format, old:

html:
  data: data.html
  evaluation: evaluation.html
  overview: overview.html
  terms: rules.html
  #notebook: README.html

New:

pages:
  - title: Data
    file: data.html
  - title: Evaluation
    file: evaluation.html
  - title: Overview
    file: overview.html
  - title: Rules
    file: rules.html

Phases

In v2 we leverage "tasks and solutions" instead of putting data directly on phases. Phases keep their main properties like start_date, end_date although some are named more simply i.e. start, end

Old:

phases:
  1:
    phasenumber: 1
    label: Development Phase
    description: 'Development phase: tune your models and submit prediction results, trained model, or untrained model.'
    start_date: 2018-11-15
    is_scoring_only: False    
    execution_time_limit: 500
    max_submissions_per_day: 5   
    force_best_submission_to_leaderboard: True      # Participants will see their best submission on the leaderboard
    starting_kit: starting_kit.zip                  # The starting kit you prepared
    ingestion_program: ingestion_program.zip        # The ingestion program (the same for both phases)
    public_data: input_data.zip                     # Same as input data (available for download by the participants)
    input_data: input_data.zip                      # The data used by the ingestion program (and the code of the participants) in both phases
    scoring_program: scoring_program.zip            # The scoring program (the same for both phases)
    reference_data: reference_data_1.zip            # The truth values (solution) for phase 1 used by the scoring program
    color: green   

New:

tasks:
  - index: 0
    name: Iris Development Phase Task
    input_data: input_data.zip
    scoring_program: scoring_program.zip
    reference_data: reference_data_1.zip

# No solutions included in this example, but it's possible to do it like so...
# solutions:
# - index: 0
# - tasks:
#   - 0
# - path: solution.zip

phases:
  - name: Development Phase
    description: 'Development phase: tune your models and submit prediction results, trained model, or untrained model.'
    start: 2018-11-15
    tasks:
      - 0
    # if we had solutions..
    # solutions:
    #   - 0

Leaderboard

Old:

leaderboard:
    leaderboards:
        Results: &RESULTS
            label: RESULTS
            rank: 1
    columns:
        set1_score:
            leaderboard: *RESULTS
            label: Prediction score
            numeric_format: 4
            rank: 1
        Duration:
            leaderboard: *RESULTS
            label: Duration 
            numeric_format: 2
            rank: 2
            

New:

leaderboards:
  - title: Results
    key: main
    columns:
      - title: Prediction score
        key: set1_score
        index: 0
        sorting: desc
      - title: Duration
        key: Duration
        index: 1
        sorting: desc

Convert the programs

Then I examined the scoring program