Transparency increase through using explainability analysis on ML models.

[pomodoren]

Is your feature request related to a problem? Please describe.
It is of value to add a layer of understanding on how the models are making a decision, and to try to follow the decision (of classification) for the inner layers of the deep learning models (like U-Net or ResNet). This can increase the transparency and understanding on how the models are acting. This is more common in use-cases of medicine, but probably can be transferable to fAIr models. Not creating it can create a disruption between the contributors and model-creators, making "magic" models.

Describe the solution you'd like
Together with the models, a downloadable report on the segmentation. Practically, you see that a model does well on the buildings that have a specific property (say - round shape), and less well on the other buildings. This can be done through Shapley values and similar methodology.

Describe alternatives you've considered

• NA

Additional context

• this issue is based on the discussion on FOSS4G conference: https://2023.foss4g.org/
• check shap package for insight (Image examples): https://shap.readthedocs.io/en/latest/index.html

Tasks

• Take example model, example data
• Test shap, find misclassification reasoning
• Create sample report
• Update report on model

[kshitijrajsharma]

Related discussion : https://forum.openlabs.cc/t/hot-osm-style-models-a-discussion/2869