UAI reference comparison tests often fail

[gdalle]

As a result they are commented out. Do we know why?

openjournals/joss-reviews#5700

[mroavi]

Hi @gdalle. Let me give you a summary of how our package performs with the UAI 2014 benchmarks, available here.

MAR:

Of the 142 benchmark problems, which are divided into 10 problem sets, 137 pass for the MAR task. The 5 failing tests belong to the 'relational' problem set. The reason our package cannot solve these problems is that it runs out of memory due to the very high tree width of these problems. This is an example where approximate methods become necessary in favor of exact ones.

PR:

After solving an issue related to arithmetic overflow in this PR, our package now correctly calculates the partition function for all problems in the benchmark, excluding those in the 'relational' data set, for the same reasons described earlier. The exact issue was that the partition function could become a very large number, beyond what a computer can normally represent. To address this, we've changed our approach to return the partition function in log space, effectively solving the overflow issue.

MAP:

Because UAI 2014 does not offer reference solutions for the MAP task, we are using the Subproblem-Tree Calibration solver made by Huayan Wang, which is available here.

We've noticed that the induced tree width of the tree decompositions generated by our package is too large to handle some of the UAI 2014 MAP problems using an exact approach. When we do manage to provide a solution, it aligns correctly with the reference solution. Despite this, our package runs out of memory for several of these problems. This is a clear case where methods like Huayan Wang's really shine. They're approximate but get the job done when you're dealing with these kinds of limitations. According to the README of his tool, his method 'returns approximate solutions and bounds for arbitrary time budgets.' Trading off accuracy for scalability is a known trade-off in the field of probabilistic inference. We are continuing to explore possibilities that will enable us to offer tractable solutions using an exact method for this problem.

MMAP:

As mentioned above, the UAI 2014 competition benchmark does not offer reference solutions for this task either. Therefore, we are validating our results with those produced by the Merlin solver, which can be found here.

The solutions produced by our tool and those generated by Merlin are inconsistent for several problems. We are currently investigating the reason for this discrepancy (GitHub issue link). One possible explanation for the inconsistency between our results and Merlin's could be the algorithms used. While our method produces exact results, Merlin supports the following algorithms for the MAP task, all of which are approximate: Weighted mini-bucket elimination, Join graph linear programming, Iterative join graph propagation, and Gibbs sampling

I hope this gives a better understanding of how our package performs with the UAI 2014 benchmarks. We're actively working on improving this.

[gdalle]

Sounds good to me! Thanks for the detailed rundown