Labels are wrongly associated with scores

[marcverhagen]

Because

Somewhere along the line we started assigning the wrong label, probably because the way we dealt with configurations has changed several times.

The config file modeling/config/trainer.yml has:

bins:
  pre:
    slate:
      - "S"
    chyron:
      - "I"
      - "N"
      - "Y"
    credit:
      - "C"

And the classifier config in /modeling/models/20231214-193543.convnext_tiny.kfold_config.yml has

labels:
- slate
- chyron
- credit

But the method train.get_final_label_names() returns

['chyron', 'credit', 'slate', 'NEG']

and since that method is eventually used to set labels on an instance of Prediction we get a mismatch.

[marcverhagen]

This was fixed in 71b57a2 by using the labels field from the model config file. An alternative would be to change

bins:
  pre:
    slate:
      - "S"
    chyron:
      - "I"
      - "N"
      - "Y"
    credit:
      - "C"

into

bins:
  pre:
    - slate:
      - "S"
    - chyron:
      - "I"
      - "N"
      - "Y"
    - credit:
      - "C"

and update train.get_final_label_names().

[keighrim]

This is probably because of this; yaml/pyyaml#110.

Trainer config:

app-swt-detection/modeling/config/trainer.yml

Lines 43 to 52 in f033551

	bins:
	pre:
	slate:
	- "S"
	chyron:
	- "I"
	- "N"
	- "Y"
	credit:
	- "C"

After yaml.dump-ed (exported config), keys are sorted:

app-swt-detection/modeling/models/20231214-193543.convnext_tiny.kfold_config.yml

Lines 1 to 10 in f033551

	bins:
	pre:
	chyron:
	- I
	- N
	- Y
	credit:
	- C
	slate:
	- S

Considering added label mapping complexity when we add postbin handling in a very near future, I don't think classifier relying on the labels values from the model_config is a good idea, simply because that value is the "final labels" after pre+post binning. To that end, I think we should remove the labels key all together from export process to eliminate a big source of confusion.

app-swt-detection/modeling/train.py

Lines 243 to 244 in f033551

	configs_copy = copy.deepcopy(configs)
	configs_copy['labels'] = get_final_label_names(configs)

To fix the original wrong association issue, just adding sort_keys=False argument to yaml.dump call should be enough.

[marcverhagen]

Agreed on both counts. I hadn't looked closely at the new export code you had added. And indeed having both bins and labels in the model configuration is redundant. I am doing some cleanup of the code now and will include those fixes.

[marcverhagen]

Fixed in 0ab78e0.