Reworking the measurement and metrics part

[thuiop]

During the last months, we have mainly focused on image generation which is only one part of BTK (albeit an important one). I believe the next big pull request should be about the measurement part which have not been updated for a while (but should still be functional).
I'm putting here the up-to-date flowchart of BTK to show how it works.

The user is supposed to create a subclass of MeasurementParams, with custom get_deblended_images and make_measurement methods (though make_measurement is basically unused in the default implementation), which should return respectively the deblended images and the measurements on the blended images. They give this to the MeasureGenerator, as well as a DrawBlendsGenerator, which when called generates the images and executes the two methods on them (and returns all of it).
For the metrics (actually measuring the performance of the algorithm), they should create a subclass of MetricsParams, several function such as get_detections, get_segmentation, get_flux... which are used to recover the results from the MeasureGenerator (which is given to the MetricsParams). The MetricsParams is then given to the run function, which uses a bunch of utility functions to evaluate the performance.
As you can see, it lacks a bit of symmetry. I think this part definitely need some reworking to get something people can actually and understand and use. Please put your ideas under this issue if you have any.

What I think : the generator structure seems appropriate to me ; I am ok with giving the DrawBlendsGenerator to the MeasureGenerator. I believe there should also be a MetricsGenerator receiving the MeasureGenerator. The MeasurementParams seems a bit heavy to me ; basically it's there to gather several functions made by the users to run on the generated data. I think this could be achieved using a namedtuple (to keep consistency with other parts of BTK), with several "special" attributes corresponding to specific measurements which will be used for the metrics part (eg deblended images, detections, segmentation,...), where users would provide functions they made, and possibly an additionnal attribute if users want to run some other measurements on the data not covered by the metrics part. EDIT : I realized that this would not be a very good way to do it, as often the same algorithm gives both segmentation, deblended images, centers or whatever. Instead, maybe the user should provide an unique function returning the results as a dictionnary with specific keys ?
In the same fashion, the MetricsParams is too heavy, and actually I do not think it really is useful (the fact that it takes in a MeasureGenerator is also very weird.
Anyway, there is some work to be done to sort out the metrics part as there are a lot of functions in it right now, and it is unclear to me right now which are useful and which are not.

[thuiop]

Some other ideas I had, but more secondary ones :

• Currently there is no clean way to run different deblending methods at the same time ; this might be an interesting thing to implement though I am not totally sure how it would work.
• I think there are some functions to plot the results but not much and you have to think about how to use them (or copy and paste some example) ; we may want to make this easier for people, though this is probably more of a long-term goal

[ismael-mendoza]

Thanks @thuiop ! I will start working on the measurement part first really soon.

Is there a way you can link your really excellent diagram to the current README ? since the current has been outdated for a while, thanks!

[thuiop]

I'm making a PR for this very soon.

[ismael-mendoza]

Thank you :) Then I'll contribute/follow-up there depending