Mitigate issue with not confirmed deposit transactions

[chimp1984]

This is a Bisq Network proposal. Please familiarize yourself with the submission and review process.

Problem description

If the required miner fee increases very fast our fee estimation service (earn.com) is not reflecting that properly and the Bisq trader uses a too low fee which might cause that the published transaction will never get confirmed and will be removed from mempool after a while. In case it was a trade tx it can affect the other trader as well and therefor spread out like a virus.

Bisq allows spending unconfirmed outputs as otherwise usability would be terrible. There is no security risk involved with that as if one input for the deposit tx was not valid the deposit tx is also invalid and the buyer needs to wait until it is confirmed before starting the fiat/altcoin transfer. But it causes inconvenience as the application is not detecting and acting automatically to that situation and users need to do a SPV chain resync which can be very time and CPU consuming if the wallet is old and contains many transactions.

There are further potential problems but those have been never observed so far, so I will keep those out from a more detailed discussion (fee for payout tx is defined at take offer time -> can be too low as we don't know when the payout tx will be published).

BitcoinJ does only know about transactions related to one's wallet but not about the trade peers trade fee tx. Technically it would be possible but it would require a SPV resync which is not feasible to do from a usability point of view.

This problem is a conceptual problem and can never be prevented to 100% (even if we would have a full node instead of BitcoinJ) but we can try to reduce the reasons which triggers it and adding tools to reduce its negative effects.

Beside low miner fee there might be some reasons causing similar effects: Bugs in the trade process (probably related to the wallet management with AddressEntryList) which could cause invalid transactions. We need to elimintate those bugs (edge cases and so far we never could reproduce it) and this reason will not be discussed in that proposal. Those are rather rare as well.

Proposed solutions

1. Get a better fee estimation service
   There is an old bounty task for that and we should prioritize to work on that. Our current service (earn.com) does not work well in times of high volatility and we are overpaying all the time to be on the safe side, so there is potential to make miner fees also cheaper.

We should add quality control for the fee estimation using historical data to compare how well our estimated fees have matched the fees used at real blocks. There are a few services out there and we should try to build on what is already developed and studied. It is a non-trivial topic which lacks of perfect solutions, so we should not under-estimate the effort to get that really well done. To get in touch with those who are working in that field is highly recommend (e.g. Stefan Weiss, Laurnet, https://bitcoiner.live,... ).

We could run that service on our provide BTC nodes and Bisq users are connecting via Tor hidden service so privay is protected or we could proxy it over our exisisting price node which provides that data already now. Combining the provided BTC nodes with the fee estimation service only does not increase privacy risks but if we use the same service for lookup of trade peers transactions it potentially worsen the privacy exposure to those provided nodes as with the bloom filter the nodes learns also about with whom one is trading. So we should be careful by mixing too many of those services.

2. Add a check if a transaction is in the mempool
   In case the taker only took a partial part of the trade amount the seller (maker) will get a change output from the deposit tx. If either the taker fee tx of the deposit tx never gets confirmed because of too low miner fee, that change output on the makers side will also become invalid. If it was used for other offers or trades we have polluted those offers or trades as well and if another trader takes such an offer he might get polluted as well.
   To mitigate that we can add a check to see if a tx is in the mempool before we use it's outputs for funding another tx. If not we can isolate that utxo until it is confirmed from being spent.
   This will only cover the already older transactions which got wiped out of the mempool but not cover the more recent transaction with too low fee still sitting in the mempool in the hope to get confirmed. So it is not clear if this will help us much. Avoiding the causes need to be the primary goal.

3. Don't allow spending unconfirmed outputs if they are not from the own transaction
   We could limit the allowance to spend unconfirmed transaction to those utxos which are funded exclusively from own utxos. The deposit tx would therefor not fall into that and a potential change output would be unspendable until the deposit tx is confirmed. I think that is rather easy to do in BitcoinJ and would only affect usability in a few cases. This might be a preferred way to deal with the problem compared to the solution discussed above. But it will require more analysis if that still might cause severe usability drawbacks and if it is true that it is rather trivial from the BitcoinJ side to implement that.

4. Add a metric or the estimated number of blocks it will take until a transaction will get confirmed
   We could use the fee estimation service this. If we see that a transaction will probably not get incuded in a certain time frame we can act on it and alert the user to double spend the tx with a higher fee and therefore canceling out the pending transaction. This approach might be more difficult and complex as it initially sounds. Our BitcoinJ version does not support RBF (replace by fee) as far i am aware ;-( (I think it is also not supported in v0.15). We still can do a forced double spend (and there is a hidden feature in Bisq to do that) but it requires a SPV resycn and we need to communicate that to the other trader and force him to a SPV resync as well. But that will only work after the double spend tx is confirmed. Also there might be race conditions that both traders do that at the same time and then their 2 different double spend txs are in a race which one will get into the block and the loser has essentially 2 txs with a double spend. So this might at the end cause more new problems as it is trying to solve or at least causes quite some complexity and UX challenges. We have to keep in mind that some users hardly understand the technology behind Bitcoin and terms like mempool or doublespend are either unknown to them or might stress them. Relying on traders to make a good decision here is a bad strategy. Forcing them from some automatiation into a stressful user experience (SPV resync can take very long) is problematic as well.

Discussion

We have to keep in mind that this problem exists since day one and rather rarely it had caused big problems. If we would have a "perfect" fee estimation service (considering overpaying fee as a valid option to be on the safe side) this problem should nearly never happen and if it appears users can fix it by getting help from mediators or support.

Adding the solution #2 and solution #4 suggestions seems to be rather complex and problematic and I would suggest to keep them as a secondary step if that problem is not getting solved by a high quality fee estimation service and rather put the effort into that work.

Lets do one step at a time and be aware that the challenges we are trying to solve are not trivial.

EDIT: Have referred to wrong solutions in the discussion part. I find solution #2 and #4 rather complex and problematic.

[wiz]

Thanks for typing this up, just to update from my side I'm working on 1 and 2 already:

For 1, we have a self-hosted mempool fee estimation API service ready for all btcnode operators to deploy immediately, it's the backend we run on https://mempool.space/api/v1/fees/recommended and http://xxh4i4lbbrh6mzy2.onion/api/v1/fees/recommended

For 2, we can have btcnode operators run the self-hosted electrs backend on their btcnode and serve the block explorer REST API over a tor hidden service so Bisq can query these btcnodes if the TXID is in the Bitcoin mempool or not.

For 3 and 4 are you working on implement these already?

[chimp1984]

@wiz

Have referred to wrong solutions in the discussion part. I find solution #2 and #4 rather complex and problematic.

Was a mistake in initial posting....

I think we should in parallel develop some tool to collect historical data from different services to be able to analyse and compare which strategy works best. There are some services out there and I doubt we can outperform them when we just start new in that field. I consider quality fee estimation a hard problem without perfect solutions and we should build on top of prior art.

[wiz]

I agree that fee estimation is a hard problem, and that's why I've been contributing to the Mempool.space project for the past several months, which is one of the most respected Bitcoin mempool analysis websites available - it's much better than 21 / earn.

Mempool.space updates its database every few seconds to project the next X blocks that will be mined, and uses this data to estimate fees in the next X blocks. I actually observe the mempool all day every day on a dedicated display, to constantly check if the "next block fee" is accurate or not. What you consider "best" may not be what I consider "best", but since this API updates in realtime from the live mempool state, I think this is a very good starting point, and much better than the current centrally hosted CPOF at earn. This way we can all self-host our fee estimation backend.

[chimp1984]

Sure earn.com is crap.

But how does Mempool.space relates to those 2 projects:
https://whatthefee.io/
https://bitcoiner.live/

At least whatthefee is using mempool for estimation as well. I just would like to learn which strategies are behind to see the differences. Would be great to have a comparision page of the best candidate services and then check it with historical data to see which service delivered the most accurate estimations, specially in times of fee volatility. Do you have any historical data of Mempool.space? If so we could check already how good the estimation have been during the recent fee spike.

[chimp1984]

@wiz Any progress on the fee estimation service?

[wiz]

@chimp1984 Yes, in the @mempool-space backend we are adding a new fee estimation API endpoint for Bisq, basically we just implement the same API endpoint spec that we are using now, so the Bisq pricenode code doesn't need to change at all, just the API endpoint hostname will change to the self-hosted IP address instead of earn.com - then the Bisq pricenodes will continue to use the same the "highest fee transaction" in the "2nd projected block" like it is now, but mempool has a better implementation of this because the data updates every few seconds in realtime. additionally the fee estimation API will be self-hosted on each Bisq btcnode so we eliminate the CPOF using earn. PR coming soon!

[mpolavieja]

Hi @wiz, @chimp1984. Could you please update the status of this proposal?

[wiz]

Recently @chimp1984 added several mitigations, especially the new concept of "failed trades", which reduce the severity of this issue quite a bit. We still need to move away from earn to a self-hosted fee estimation service, but that can be done with the usual PR review process and pricenode maintainer role and doesn't require a proposal.

So I think we can close this proposal for now as completed, and re-open again in the future if it becomes necessary.

[mpolavieja]

Ok, thanks. I´ll close it as approved (completed).