Remove the need for Bisq to trust Bitcoin Average by querying exchange APIs directly and calculating our own weighted average for fiat and altcoin prices #35
Comments
wiz
added
a:proposal
|
I like this idea in principle although I'm not familiar with the different feeds and how much work it would be to use all the different data feeds. It is certainly a weakness to rely on BitcoinAverage for all our market data. I do see a risk in inconsistent price feeds though using an aggregate of feeds from different exchanges calculated at each node. We have seen people abusing the tolerance margin introduced to allow for difference in price feed before. |
Since @cd2357 worked on the Pricenode PR that implemented #27, I've asked him to work on the PR for this project as well. I understand he already has a WIP that enables Pricenodes to directly query the 10+ centralized exchange APIs directly without adding any jar deps to Bisq. Since the jar dep library that Bisq already uses to query Exchange APIs has now added support for over 80 exchanges, it's not much work to add these new exchanges: https://github.com/knowm/XChange
While this is possible, in practice I don't think it will be an issue as we have price thresholds to resolve this. In any case, it will require lots of testing and a slow phase-in to the new system.
Yes, but AFAIK the only cases of this in the past were caused by BitcoinAverage flaws, which would be resolved by this proposal. For example, BA used LocalBitcoins for SEK prices instead of the largest Bitcoin Exchange in Sweden called Safello, which made Bisq vulnerable to someone trading a small amount of LocalBitcoins to move the Bisq price and buy Bitcoin below market: https://bisq.community/t/btc-sek-is-way-off/6558 |
|
The issue I'm talking about is for buyer and seller to agree on price. We have a very narrow allowance to avoid someone tries to game it. If we have multiple sources it might not be possible to have such a narrow spread, or it won't be possible to take offers. I'm not sure how a phase-in would work considering the buyer and seller has to come to the same price, or they won't be able to trade. If they have different sources it's much more likely that they will disagree on price and taking the trade will fail. |
|
My understanding is that
There are about 6 pricenodes now which basically run the same code and should provide the same exchange rate data (within some margin of error, since the pricenodes could poll BitcoinAverage within a few seconds from each other, so they could get slightly different data). The Bisq clients then poll these pricenodes (one of them? several of them? not sure) and for each currency pair, the exchange rate with the latest timestamp gets used in the client. So if I understand this correctly, the only ways two Bisq clients can have massively different exchange rates is either:
I guess this is possible now as well, but Bisq defends against this by using this narrow allowance (which you mentioned @sqrrm ) for an acceptable exchange rate difference between traders. So even if a Bisq client tries to game the system and use favorable exchange rates at the expense of someone they trade with, the protocol would not allow a big difference between their rates. Again, assuming my understanding is correct, what this project proposes is to have the pricenodes use more providers to get exchange rate data from multiple exchanges. Then each pricenode would aggregate the data and calculate some kind of "weighted average" (prioritizing values from exchanges with more liquidity, or the more established ones, or the more reputable, etc; specific logic TBD). When Bisq clients poll the pricenodes, instead of getting the BitcoinAverage rates, they would get these weighted average rates. And if all 6 Bisq pricenodes use the same code, meaning
then they should basically report very similar exchange rates to the Bisq clients. Basically it's a similar situation to now, cause the pricenodes could report slightly different BitcoinAverage rates (cause they query BitcoinAverage at different times). So I think this risk exists only to the same degree that it already exists now, and the narrow allowance check in the Bisq clients already mitigate against it. But please correct me if I'm wrong, it's quite a complex system so it's easy to overlook smth. |
|
As the ops team lead, I am assigning this project to @cd2357 and I have allocated budget for his compensation for work under this project from the ops team budget. |
|
Now that @cd2357's PR seems to be working well, we can proceed to have the pricenode operators setup new instances for testing. Can all the @bisq-network/pricenode-operators please make a fresh Ubuntu 20.04 VM and download the pricenode installation script from @cd2357's PR branch and replace these 4 lines then run it locally: After you get your new Tor V3 onion, please comment here and after we have several new pricenodes running we can begin full testing of the new pricenode code:
Also @bisq-network/pricenode-operators be advised this project is a 🚨 high risk change 🚨 and that we might call for a rollback (or upgrade to the new, reverted version) in short order after the migration. |
|
I just installed the service and it seems to be running. The onion is ro7nv73awqs3ga2qtqeqawrjpbxwarsazznszvr6whv7tes5ehffopid.onion |
|
aprcndeiwdrkbf4fq7iozxbd27dl72oeo76n7zmjwdi4z34agdrnheyd.onion |
Pending merge of bisq-network#4315 and rough consensus to proceed with migration plan in bisq-network/projects#35 wizpriceje6q5tdrxkyiazsgu7irquiqjy2dptezqhrtu7l2qelqktid (@wiz) emzyprwrcz22h2fzhsbqd3deoe5hkqdm5yfy4geuqcdqqicgoor65iad (@Emzy) 6b7jpqiy2ejq3m7jskem6cuverg7xelhhuf3d2nvooucayoeapo2m3qd (@devinbileck) aprcndeiwdrkbf4fq7iozxbd27dl72oeo76n7zmjwdi4z34agdrnheyd (@mrosseel) ro7nv73awqs3ga2qtqeqawrjpbxwarsazznszvr6whv7tes5ehffopid (@alexej996)
|
Okay, now that we have 5 pricenodes online the staging environment is ready for full testing. The branch I'll be using to test is bisq-network/bisq#4403 |
|
My node is now running with the following onion address: |
Pending merge of bisq-network#4315 and rough consensus to proceed with migration plan in bisq-network/projects#35 wizpriceje6q5tdrxkyiazsgu7irquiqjy2dptezqhrtu7l2qelqktid (@wiz) emzyprwrcz22h2fzhsbqd3deoe5hkqdm5yfy4geuqcdqqicgoor65iad (@Emzy) devinpndvdwll4wiqcyq5e7itezmarg7rzicrvf6brzkwxdm374kmmyd (@devinbileck) aprcndeiwdrkbf4fq7iozxbd27dl72oeo76n7zmjwdi4z34agdrnheyd (@mrosseel) ro7nv73awqs3ga2qtqeqawrjpbxwarsazznszvr6whv7tes5ehffopid (@alexej996)
|
@sqrrm @cbeams Okay I think we are ready for review. Please see the following PRs: |
|
I've been doing a lot of testing and found no issues so far. As for the migration to repoint old V2 onions to the new pricenodes, I would say after the above 3 PRs are merged and there are no issues found, once v1.3.8 gets pre-released the pricenode operators should all coordinate a time for everyone to switch at the same time. When there are no issues from the migration, we can safely release v1.3.8 with the new V3 onions hard-coded. @ripcurlx what's the tentative schedule for the v1.3.8 release timeline? |
|
I would like to change my HS to: |
|
lol ok. will amend PR. |
Pending merge of bisq-network#4315 and rough consensus to proceed with migration plan in bisq-network/projects#35 wizpriceje6q5tdrxkyiazsgu7irquiqjy2dptezqhrtu7l2qelqktid (@wiz) emzypricpidesmyqg2hc6dkwitqzaxrqnpkdg3ae2wef5znncu2ambqd (@Emzy) devinpndvdwll4wiqcyq5e7itezmarg7rzicrvf6brzkwxdm374kmmyd (@devinbileck) aprcndeiwdrkbf4fq7iozxbd27dl72oeo76n7zmjwdi4z34agdrnheyd (@mrosseel) ro7nv73awqs3ga2qtqeqawrjpbxwarsazznszvr6whv7tes5ehffopid (@alexej996)
|
@devinbileck are you planning to do full release testing for v1.3.8 ? obviously we will want to emphasize anything that uses prices |
|
@m52go Since the new Bisq Price Indices are different from Bitcoin Average prices, in the case of BTC/USD about $100 or 1%, we should probably give some advance warning to users about the migration and @ripcurlx maybe an in-app broadcast message reminding users of the change a few days/hours/minutes in advance. What do you think? 💰Bisq Pricenode Data Check on Sun Aug 9 21:05:01 UTC 2020 |
wiz
removed
a:proposal
wiz
added
has:approval
|
Since the Bisq Pricenodes were all migrated to the new code, and Bisq is no longer using Bitcoin Average as a price oracle, calculating our own weighted averages (documented at https://bisq.wiki/Bisq_Price_Indices), I am closing this project as 💰Bisq Pricenode Price Check on Thu Aug 20 09:05:02 UTC 2020 BTC/EUR: BTC/GBP: BTC/BRL: BTC/CAD: BTC/AUD: BTC/JPY: BTC/SEK: BTC/XMR: BTC/ZEC: BTC/ETH: BTC/ETC: BTC/DOGE: Thanks a lot to @cd2357 @sqrrm @cbeams and @bisq-network/pricenode-operators for all the hard work for this project, another one for the ops team win list! 🎉 |
|
It appears this change has rolled out quite smoothly. Congrats, all! |
Approved in bisq-network/proposals#163 and fixes bisq-network/bisq#1074
Description
This project will remove the need for Bisq network to trust BitcoinAverage as a data oracle and provide fiat and altcoin prices, by implementing our own weighted average price index for fiat and altcoin prices calculated from querying 10 or more Bitcoin Exchange APIs directly.
Rationale
Currently the Bisq Pricenode Operators all subscribe to expensive Bitcoin Average API subscription plans, which is a large recurring expense for the Ops team budget. It's a no-brainer to do this project financially, since we will no longer have to pay these recurring monthly expenses for API subscriptions, but much more importantly in terms of our goals to improve reliability and censorship-resistance, this project will remove a very centralized TTP and CPOF for Bisq by decentralizing the data sources. If BitcoinAverage were to suddenly ban Bisq, we would be in trouble.
Criteria for delivery
When all the items in Part 1 below are completed, we can consider this project to have been delivered.
Measures of success
After we have shut down all Bisq Pricenodes using Bitcoin Average, and Bisq is running stable using the new decentralized providers, we can consider this project to be successful.
Risks
The new code could crash our Pricenodes.
The new code could calculate prices incorrectly.
The pricenodes might disagree on the price for a certain fiat or crypto asset, causing network issues.
More risks probably exist which we cannot know.
Tasks
Part 1: Implementation
Part 2: Documentation
Part 3: Testing
Part 4: Migration
Estimates for budget allocation
Dev: $5000
Ops: $3000
The text was updated successfully, but these errors were encountered: