Data processor with uncertainties package

[nkanazawa1989]

Summary

This PR updates internal data representation within the data processor. Namely, the computation of error propagation is offloaded to uncertainties package, and Tuple[Any, Any], i.e. tuple of nominal values and standard errors, appearing everywhere in the processor is simplified to np.ndarray.

Details and comments

Manually handling error propagation has been big pain for writing new node (one may induce numerical error, or one might be lazy to compute error propagation, or to write its unittest, etc...). Offloading this to uncertainties package will improve this complexity in coding, and, in addition, now we can integrate nominal values and standard errors into a single numpy array. Thus we can cut the amount of code into roughly half.

This package supports ufloat, and you can apply most of standard math operation as if to use a python float. In the processor data is usually represented by numpy array version of it rather than ufloat.

Hint for review:

ufloat array can be converted into standard float array of nominal and error part by

data = unumpy.uarray(nominal_values=[1., 2., 3.], std_devs=[0.1, 0.2, 0.3])

>>> unumpy.nominal_values(data)
array([1., 2., 3.])

>>> unumpy.std_devs(data)
array([0.1, 0.2, 0.3])

In addition, this convenient functions support non ufloat array

data = np.array([1., 2., 3.])

>>> unumpy.nominal_values(data)
array([1., 2., 3.])

>>> unumpy.std_devs(data)
array([0., 0., 0.])

so we can support error free data without writing branching logic.

[nkanazawa1989]

With unumpy module, we can initialize the processed data as a single numpy array instead of tuple of list and None. Thus we don't need to typecast data to array (asarray), and frequently appearing code in the format method

if error is not None:
   ...

can be removed too. This means we are no longer need DataAction._format logic, and now it is almost looking like _validate method. If we can rename the method, the branching if self._validate: can be moved to __call__ method, and we can further simplify the logic in subclasses.

[eggerdj]

This looks like a good direction to follow. I would like to see a very short explanation of the uncertainties package in the main doc in particular how covariances between errors are accounted for.

[eggerdj]

Looks good to me. Some minor suggestions. One last question to double check: this will still allow us to construct nodes that process memory (e.g. a sequences of shots like ["01", "01", "11", "11"]), right?

[eggerdj]

💯

[eggerdj]

Perhaps best to show the math that lead to this number to make the test more transparent.

[nkanazawa1989]

Wrote some comment in 5f49f5f

I did't turn this into math formula since this means we evaluate the code with another (same) code.

[yaelbh]

@nkanazawa1989 @eggerdj I see that this PR is approved, can we merge it?

[eggerdj]

I'm fine with merging but first consider the suggestions on the docs.

[nkanazawa1989]

this will still allow us to construct nodes that process memory (e.g. a sequences of shots like ["01", "01", "11", "11"]), right?

good catch! typecast to float array can break data structure of level2 memory. The logic is fixed and data preparation test has been added for all input data types.

[eggerdj]

LGTM. Thanks for adding the extra tests.