Improve fee estimation

[ManfredKarrer]

As we have seen in the recent spam attack and blockchain congestion the fee estimation is difficult to get right.
We use an API from https://bitcoinfees.earn.com but it was usually overpaid quite a lot. So we reduced a while back to get more to an optimum. That worked pretty well until recently when in 1 or 2 days the mempool grew from 20k to 200k unconfirmed transactions. That rapid rise was not handled well by the current model.
One reason was the average function. We used the average of the last hour, which kicked us always to the lower end and with rising fees the chance to get those txs confirmed decreased.

We changed now to more aggressive and expensive fees (lower max blocks param and only last 10 min taken for average) but I think that will lead to overpay again. Why we should pay the Bitmain and co more as absolutely needed? They do a great disservice for Bitcoin since long and it really hurts to pay them extra. So we should find a way to get back to the min. fee required to get the tx confirmed in < 3 blocks.

We could manually adjust the parameters and once we observe mempool getting filled up we have to be more aggressive again. But that manual adjustment is not a good solution of course.
So we should find a way how to influence the parameters based on those observations we would do manually, namely the size of the mempool as well the dynamics (fast rise or more organic behavior).

First approach could be to simply request the mempool size and use that as influence for adjusting our parameters.

Anyone who is expert in that field would be very welcome to give input. Alternative estimations services would be great as well as ear.com does not work well from my experience (e.g. I often observed the they suggest a high fee and I got txs with much less confirmed immediately).

io.bisq.provider.fee.providers.BtcFeesProvider is the class which implements our estimation.
It simply takes the maxDelay as param and adds the average function to not be too much exposed to spikes.

[ripcurlx]

Besides optimizing the fee adjustments, we should also add the information within the trading process of the expected confirmation time to reduce the number of disputes opened.

[montvid]

I use https://jochen-hoenicke.de/queue/#2h I enter manually 300 200 140 or 100 depending on the chart - works for me. Now it would be 300.

[mrosseel]

@montvid based on what do you take 300? You just take the 'second most expensive' fee interval from that page?

[montvid]

500 - 300 tx 300 - 15000 tx 200 - 25000 tx so i choose the best for me. If I want to wait i choose 200 if i want it quite fast i choose 300 if i want instant i can do earn.com or 500.

[ManfredKarrer]

Here you can see our current fees: http://37.139.14.34:8080/getFees
It is on the very high side atm and we need to get that lower with a smart tool to not risk to get caught by the next spam attack.

[montvid]

I don't know from where are you getting litecoin fees ""ltcTxFee": 500" but they are the minimum 0.001ltc/kb and will be in the future (from 0.16 the devs say it could be 0.0001ltc/kb) as the blocks are never full always half empty. Litecoin blocks are capable 2.5min/1mb so 10min/4mb - with segwit almost 10min/8mb.

I was saying we could use discovery how many transactions are in what range of satoshi as earn.com or https://jochen-hoenicke.de/queue/#2h and make our own conclusions as to what fee is optimal. How to automate that I don't know but I never use the earn.com recommended price as it is too big - I decide from the charts a more optimal price. of course there is the bitcoin core client algorithm to predict fees but i don't think it is very accurate too.

Good luck!

[montvid]

See here for litecoin block space info https://chain.so/ltc

[cbeams]

Of interest from an article posted today: https://coinjournal.net/charlie-lee-wants-test-new-solution-bitcoin-fee-estimation-litecoin/

Lee’s solution is to have miners place the lowest fee included in a mined block in that block’s header. A soft-forking upgrade to Litecoin would ensure that miners do not include transactions with fees lower than the minimum signalled in the block header.

According to Lee, litecoin wallet users will then be able to calculate the median fee based on miner signalling from the last 100 blocks, which should allow them to ensure that their transaction gets confirmed in the next couple of blocks. Since the minimum fee is included in the block header, this data is also available to Simple Payment Verification (SPV) clients.

[sneurlax]

A good fee estimate is important, but we should also be able to use RBF to increase fee if the "mempool landscape" changes, as it were.

If this is a bounty, would you make #1139 a bounty, as well? I would claim this bounty, but do so via #1139, ensuring that Bech32 (P2WKPH) and RBF are both covered by bitcoinj, if necessary.

Fees would be lowered significantly via SegWit already. Then we could use a relatively-low initial fee estimate (or better yet, give the users a simple slider to choose between a 25-block and next-block estimate) and just increase it later.

[cbeams]

Yes, all of this stuff is bounty potential. Please come join us in Slack where we can discuss this and related work in more detail. https://bisq.network/slack-invite.

[montvid]

I want to suggest using a bitcoin node with a limited mempool to estimate fees. The mempool holds the highest fee/fastest to confirm transactions first. Let's say the mempool is 10 or 20 MB so the node would report what is the minimum fee to get into the 10 MB mempool thereby predicting the tx cost in realtime like https://dedi.jochen-hoenicke.de/queue/#2h (see Mempool size in MB and Unconfirmed tx count)

[ManfredKarrer]

There are 2 dev working on better fee estimation (met them in 34c3).
On is Felix Weis (https://twitter.com/felixweis, http://blog.bitcoin-traveler.com/) and the other one is https://github.com/dlebrecht.

Can anyone contact them and check out possibilities to use their service? Felix at least told me that he has his service open source and should be out mid January.

[ManfredKarrer]

The guys behind that page seem to be other good candidates: https://b10c.me/

[ManfredKarrer]

Electrum announced mempool based fee estimation:
https://twitter.com/ElectrumWallet/status/962417376510824448

[montvid]

Unfortunately there is no big use in that if bisq fee setting is broken now #1325 but looking bright in the future!

[ManfredKarrer]

@montvid Your bug report is the only I am aware of. Fee calculation basically works but seems some edge cases are not covered correctly (your report).

[ManfredKarrer]

David Vorick's idea: https://gist.github.com/DavidVorick/ba17fb7db50b9270d1c2131ff425422b

[ManfredKarrer]

I just had a conversation with Felix Weiss and he told me he does not have time to help us. The API is still not public but we can use that url: https://whatthefee.io/data.json

curl https://whatthefee.io/data.json
{"index": [3, 6, 9, 12, 18, 24, 36, 48, 72, 96, 144], "columns": ["0.0500", "0.2000", "0.5000", "0.8000", "0.9500"], "data": [[260, 260, 280, 300, 340], [220, 240, 260, 280, 320], [180, 240, 260, 260, 300], [140, 200, 240, 260, 280], [20, 140, 220, 260, 260], [0, 100, 200, 240, 260], [0, 0, 140, 200, 240], [0, 0, 40, 180, 240], [0, 0, 0, 120, 200], [0, 0, 0, 20, 180], [0, 0, 0, 0, 40]]}

To get to satoshi/byte just calculate exp(feelvl/100), e.g. exp(300/100) = 20.8.

He will update us if the format changes.
I think we could start to build on that and use the 90-95% range.
We shoud have some fallback / cross check so probably good to keep the earn.com data as well and
make some sanity checks (e.g. diff. must not be larger the x).

Any dev want to work on that? It is not lot of work, just a request, parsing the json and making that sanity check.

I suggest a bounty of 1000 BSQ for that.

[ghost]

@ManfredKarrer
I guess this has to be done in java, and integrated in the pricenode code ? or I'm wrong ?

[ManfredKarrer]

Yes. But its is pretty simple.

[ghost]

I'll give it a look.
(I noticed java has similarities with C++, and I did already code in C++).
I'm however slow, so anybody faster has priority.

[cbeams]

Please study the design of the relevant pricenode code, particularly BitcoinFeeRateProvider and its service provider interface FeeRateProvider before implementing anything. A first step would be to implement and test a WhatTheFeeFeeRateProvider. A second step would be to look at how to integrate it into the app, compare it against results with the existing (earn.com) BitcoinFeeRateProvider, etc.

[mrosseel]

yet another fee estimator, looks well thought out, leaving it here for reference: https://bitcoiner.live/

[bitbug42]

Hi, developer of bitcoiner.live here.
I just recently added an API to the service, feel free to use it if it can help.
It's described here: https://bitcoiner.live/doc/api
I plan to keep it online and the format stable for the foreseeable future.

[ManfredKarrer]

@bitbug42 Ah great thanks!

[huey735]

For DAO transactions there's little reason to use the same values used in the trade's transactions. A better solution seems to be to use instead the custom value that we can set for withdraws.

[wiz]

My self-hosted mempool fee estimation API service is now live at http://xxh4i4lbbrh6mzy2.onion/api/v1/fees/recommended

After a few more btcnode operators start this service as well, I will make a PR to add those hostnames to the pricenode code and remove earn's API. The API functionality should be the same.

[mrosseel]

looks good wiz, where can I find the source for this?
Edit: never mind, found it at https://github.com/mempool-space/mempool.space
any opinion on bitcoiner.live as backup/sanity check?

[wiz]

@mrosseel I think if we all self-host we'll be good, but let's discuss in #btcnode channel on keybase

[Emzy]

My self-hosted mempool fee estimation API service is up at
http://22qaao7zikvzqehg.onion/api/v1/fees/recommended

[wiz]

Created a project for this bisq-network/projects#27

[cd2357]

Have you considered using the estimatesmartfee call from Bitcoin Core?

[wiz]

Fixed by bisq-network/projects#27