balance transactions with their amounts' precisions

[simonmichael]

• lib: move commodityStylesFromAmounts to Hledger.Data.Amount
• lib: use a transaction's amount precisions when balancing it

[simonmichael]

"Surprising developments in old behaviour, as a consequence of #931:
now that print shows amounts with all of their decimal places, we had
better balance transactions using all of those visible digits
(so that hledger and a user will agree on whether it's balanced).

So now when transaction balancing compares amounts to see if they look
equal, it uses (for each commodity) the maximum precision seen in just
that transaction's amounts - not the precision from the journal's
commodity display styles. This makes it more localised, which is a
nice simplification.

In journalFinalise, applying commodity display styles to the journal's
amounts is now done as a final step (after transaction balancing, not
before), and only once (rather than twice when auto postings are
enabled), and seems slightly more thorough (affecting some inferred
amounts where it didn't before).

Inferred unit transaction prices (which arise in a two-commodity
transaction with 3+ postings, and can be seen with print -x) may now
be generated with a different number of decimal places than before.
Specifically, it will be the sum of the the number of decimal places
in the amounts being converted to and from. (Whereas before, it was..
some other, perhaps larger number.) Hopefully this will always be a
suitable number of digits such that hledger's & users' calculation of
balancedness will agree.

Lib changes:

Hledger.Data.Journal
added:
journalInferCommodityStyles
journalInferAndApplyCommodityStyles
removed:
canonicalStyleFrom"

[simonmichael]

May need a bit more tweaking. It's visibly more picky about unit transaction prices (@), sometimes requiring them to be more precise than previously.

[simonmichael]

PS here's the original motivation for this: after #931, I hoped that print's output would always be parseable, but sometimes it still wasn't, because transaction balancing was still influenced by commodity declarations, which print currently does not preserve. And secondly now that print is showing all decimal places, it seems simpler and more intuitive for hledger to use those same decimal places when checking transaction balancedness.

[simonmichael]

As mentioned, the new balancing is more picky, requiring more decimals in unit prices for certain transactions (where you have more decimals than usual, eg from investment transactions). It's arguably more correct and simpler (not affected by far-off commodity directives), but this does seem like it would be pretty annoying breakage for folks upgrading. Unfortunately I think this means the old behaviour must be kept around as an option, at least for a while. So I'm thinking possibly:

Add a --balancing=exact|styled option.
exact is the new balancing using the transaction's precisions. (In each commodity, the sum of amounts, when rendered with the max precision of the amounts being summed, looks like exactly zero.)
styled is the old balancing using the global display precisions. (In each commodity, the sum of amounts, when rendered with the journal's inferred/declared display precision, looks like exactly zero.)
exact is the new default. As a migration aid, transaction-balancing errors will suggest correct entries if possible, and mention --balanced=styled as a fallback.
styled is advertised as being deprecated and liable to be dropped eventually.

[simonmichael]

Repushed with --balancing option, a test and docs. I think it's ready for merge, any review welcome.

"A surprising development in old behaviour: as a consequence of #931,
print now shows amounts with all of their decimal places, so we had
better balance transactions using all of those visible digits
(so that hledger and a user will agree on whether it's balanced).

So now when transaction balancing compares amounts to see if they look
equal, it uses (for each commodity) the maximum precision seen in just
that transaction's amounts - not the precision from the journal's
commodity display styles.

This makes it more localised - therefore simpler - and more robust,
when print-ing transactions to be re-parsed by hledger (previously,
print-ed transactions could be unparseable because they were dependent
on commodity directives).

However, the new behaviour can break existing journals, so we provide
a --balancing=exact|styled option to select the new (default) or old
balancing behaviour. (The old behaviour may not be perfectly
replicated, but it's hopefully close enough to be unnoticeable.)
This is intended as a temporary migration aid, hopefully to be removed
eventually.

In journalFinalise, applying commodity display styles to the journal's
amounts is now done as a final step (after transaction balancing, not
before), and only once (rather than twice when auto postings are
enabled), and seems slightly more thorough (affecting some inferred
amounts where it didn't before).

As a consequence of this change, inferred unit transaction
prices (which arise in a two-commodity transaction with 3+ postings,
and can be seen with print -x) may in some cases be generated with a
different (greater) precision than before. Specifically, it will now
be the sum of the number of decimal places in the amounts being
converted to and from. (Whereas before, it was.. something else.)
Hopefully this will always be a suitable number of digits such that
hledger's & users' calculation of balancedness will agree.

Lib changes:

Hledger.Data.Journal
added:
journalInferCommodityStyles
journalInferAndApplyCommodityStyles
removed:
canonicalStyleFrom"

[apauley]

I ran my standard set of reports using hledger from this branch - I had one transaction from 2018 that was off, and it was easy enough to adjust it.

real postings' sum should be 0 but is: $-0.00000000000000832

[simonmichael]

Here's an entry that's hard to adjust for the new code (minimised from a cost basis adjustment transaction) :

commodity $0.00
2020-01-01
    a     A -11.0 @ $0.093735
    b     A  10.4 @ $0.099143

The amounts are always converted to cost before summing. For --balancing=styled, the above is close enough, since $ amounts are rounded to two decimal places. For --balancing=exact, it fails no matter how precise we make the price, eg:

2020-01-01
    a     A -11.0 @ $0.093735
    b     A  10.4 @ $0.09914278846153846

Why ? Roughly speaking, it converts to cost, transforming the above to:

2020-01-01
    a     AAA -$1.031085
    b     AAA  $1.031084999999999984

It sums these, and would round the result to $'s max precision inferred from the transaction, but there is no precision inferred for $ since the original entry had no $ amounts (price amounts do not influence display styles, remember) so no rounding happens and the result is not zero.

Would we want $ to be rounded to the price amounts' precision here (6 or 17) ? Or to the original commodity's precision (1) ?

[simonmichael]

(I think 1.)

[simonmichael]

Update: this is stalled, awaiting more testing and fixes for balancing precision issues.