Market history dust orders mess up charts

[svk31]

I'm sure you've all seen charts that look like this in the GUI:

The main reason for this is partial fills leaving behind dust orders with only 1 satoshi remaining on the sell or buy side, which can only execute at one specific price due to the price being defined as the ratio of buy/sell quantities. This price can be quite different from the originally defined price of the order, causing these jumps in the price seen in the chart.

One way of improving on this is to filter out these dust orders when constructing the buckets, which I've implemented in this commit: aaa82a8

Now I'm no C programmer so that code is probably terrible so if someone can clean it up that's great, but it works and proves that filtering can improve the display of the market history. Compare the first screenshot to this second one of the same market and period with my local node running my fix:

There are still some weird spikes, but it's much improved.

[oxarbitrage]

ill take a look, i had the exact same problem making charts for the explorer, thanks for bringing it in because i think it need to be fixed from the plugin. it is maybe possible that your commit fix one of the problems and the remaining spikes are by some other issue, not sure but i will check to see if i can get some more info.

pls next time if you can send code in the form of pull request to the bitshares develop branch even if you know it will not be accepted as it is.

[abitmore]

Thanks for finally opened this ticket, so we'll have a dedicated place to discuss this issue.

PR #287 was a trial to solve this, but I think the approach is not the best.

As @pmconrad mentioned in your commit, it's not good to rely on the precision.

Issue #342 is related.

My idea to this issue is: for each match/filling, we should record an extra price field (the maker price), but not only the amounts paid/received, then this price can be used to produce charts. This approach may need to change the structure of fill_order_operation.

[svk31]

If you have an idea for how to solve it at the root then that's obviously more desireable, my solution is more of a band-aid that fixes the issue in most cases for most assets.

Relying on the precision was the only way I could see to make sure it wasn't applied to low precision assets like BEYONDBIT which is 0 precision and will thus have many trades of 1 satoshi value; so the precision criteria is just one of exclusion not inclusion.

Anyway, I'm sure you can figure out a better way to do this.

[pmconrad]

I agree that adding the maker price to the fill_order_op would be the best way to go. The downside is that this would increase the storage size of the operation history object by 25%, therefore I'd postpone this change until the ES plugin is in a usable state.

[abitmore]

The implementation I submitted (#455) requires more than 25% of additional storage size, but IMHO it's easier to read/understand. @svk31 you can take a look and/or test.

[abitmore]

Merged. Closing.

[vanissoft]

As this problem manifests in OHLC results, couldn't better to adapt OHLC generator to ignore minimun quantities transaction? Thinking loud.