Fatal exception: No path found from AR to USD

[kiates]

Received No path found for AR to USD. Please open an issue at https://github.com/eprbell/dali-rp2/issues. when processing input from Pionex using the csv.pionex plug-in:

2023-04-01 14:13:18,834/dali/DEBUG: Fetched high conversion rate 1.0005 for 2022-01-13 15:22:14+00:00/USDT->USD from cache of plugin Historic-Crypto: HistoricalBar(duration=datetime.timedelta(seconds=60), timestamp=Timestamp('2022-01-13 15:22:00+0000', tz='UTC'), open=Decimal('1.0004'), high=Decimal('1.0005'), low=Decimal('1.0004'), close=Decimal('1.0005'), volume=Decimal('31751.38'))
2023-04-01 14:13:18,834/dali/DEBUG: Unresolvable transaction (no unique_id): InTransaction:
plugin=Pionex_CSV
unique_id=__unknown
raw_data=01/13/2022 15:22:14,0.25261294,USDT,0.00490807,AR,0,USDT,
timestamp=2022-01-13 15:22:14+0000
asset=USDT
exchange=Pionex
holder=Chad Yates
transaction_type=Buy
spot_price=1.0005
crypto_in=0.25261294
fiat_fee=None
fiat_in_no_fee=None
fiat_in_with_fee=None
notes=high spot_price read from Historic-Crypto plugin;
2023-04-01 14:13:19,325/CCXT-converter/high/DEBUG: Converting AR to USD
2023-04-01 14:13:19,325/CCXT-converter/high/DEBUG: Using default exchange Kraken type for Pionex.
2023-04-01 14:13:19,325/CCXT-converter/high/DEBUG: No path found for AR to USD. Please open an issue at https://github.com/eprbell/dali-rp2/issues.
2023-04-01 14:13:19,325/dali/ERROR: Fatal exception occurred:
Traceback (most recent call last):
File "C:\Users\cyates\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\dali\dali_main.py", line 169, in _dali_main_internal
resolved_transactions: List[AbstractTransaction] = resolve_transactions(transactions, dali_configuration, args.read_spot_price_from_web)
File "C:\Users\cyates\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\dali\transaction_resolver.py", line 246, in resolve_transactions
transaction = _update_spot_price_from_web(transaction, global_configuration)
File "C:\Users\cyates\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\dali\transaction_resolver.py", line 144, in _update_spot_price_from_web
raise RP2RuntimeError(
rp2.rp2_error.RP2RuntimeError: Spot price for __unknown:2022-01-13 15:22:14+00:00:AR->USD not found on any pair converter plugin
2023-04-01 14:13:19,328/dali/INFO: Log file: ./log/rp2_2023_04_01_14_02_25_617960.log
2023-04-01 14:13:19,328/dali/INFO: Generated output directory: dali-output
2023-04-01 14:13:19,328/dali/INFO: Done

[kiates]

On another run (the same file), I got this one instead for a different coin...

No path found for MBTC to USD. Please open an issue at https://github.com/eprbell/dali-rp2/issues.

Not sure why it's non-deterministic. Does the csv importer use multiple threads by default?

[eprbell]

Looks like you're using the default pair converter (Historic-Crypto), which doesn't have all markets. You'll want to try @macanudo527's CCXT-based pair converters, which implement a fancy price routing algorithm for cases where the pair is not directly available.

[macanudo527]

I think he is using the CCXT converter but MBTC and AR aren't listed on Kraken the default source for pricing. I'll have to add alternative exchanges for them. I'll try to get to it soon.

[kiates]

AR -> USD

I was able to work around the AR -> USD case by just specifying for it to use Gate as the default ccxt exchange:

[dali.plugin.pair_converter.ccxt]
historical_price_type = nearest
default_exchange = Gate

I tried just adjusting the nested dictionaries that seem to be designed to allow overriding which exchange to fetch the spot price from. So I added a "AR": "USD" key/value to the 1st layer, and a "ARUSD": _GATE to the 2nd layer, but that didn't seem to work. Don't know why.

MBTC -> USD

The MBTC -> USD case is more difficult as it appears to be a Pionex-only thing. An MBTC is 1/1000th of a BTC by definition, and is probably something they made internally. I remember making BTC grid bots there and they would suggest switching behind the scense to MBTC to "improve fund utilization or something".

For the moment, I'm attempting to hack around it by just watching for MBTC/USD from/to pairs and then fetching the BTC/USD price instead and then adjusting the result by dividing by 1000.

Going a bit further, one could envision a set of rules that are just based on expressions. So for example, given a from asset of MBTC, the software could automatically know that it is 1/1000 of a BTC and register that conversion factor for MBTC -> BTC in the graph system -- which I don't fully understand yet, and it doesn't help that I'm not really a python programmer. Then in theory the current design would be able to walk from MBTC -> BTC -> USD or whatever else it needed to get to.

[kiates]

Ran into another that it couldn't resolve: NEO->USD (using Kraken instead of Gate)

[eprbell]

AR -> USD

I was able to work around the AR -> USD case by just specifying for it to use Gate as the default ccxt exchange:

[dali.plugin.pair_converter.ccxt]
historical_price_type = nearest
default_exchange = Gate

I tried just adjusting the nested dictionaries that seem to be designed to allow overriding which exchange to fetch the spot price from. So I added a "AR": "USD" key/value to the 1st layer, and a "ARUSD": _GATE to the 2nd layer, but that didn't seem to work. Don't know why.

MBTC -> USD

The MBTC -> USD case is more difficult as it appears to be a Pionex-only thing. An MBTC is 1/1000th of a BTC by definition, and is probably something they made internally. I remember making BTC grid bots there and they would suggest switching behind the scense to MBTC to "improve fund utilization or something".

For the moment, I'm attempting to hack around it by just watching for MBTC/USD from/to pairs and then fetching the BTC/USD price instead and then adjusting the result by dividing by 1000.

Going a bit further, one could envision a set of rules that are just based on expressions. So for example, given a from asset of MBTC, the software could automatically know that it is 1/1000 of a BTC and register that conversion factor for MBTC -> BTC in the graph system -- which I don't fully understand yet, and it doesn't help that I'm not really a python programmer. Then in theory the current design would be able to walk from MBTC -> BTC -> USD or whatever else it needed to get to.

That's an interesting case. Basically it would be an exchange-specific map of "artificial assets":
"artificial asset name" -> ("real asset", transformation())

So on Pionex, given an artificial asset like MBTC the map would point to a tuple (BTC, divide_by_1000()). This is just to point out the functionality, but the implementation would use an ArtificialAsset class, with a get_real_asset() and transform_value() methods. Note that this mapping would only be defined in the context of Pionex (it's not global). So if in the future an actual new asset called MBTC appears on the market, the above mapping would not be applied outside of Pionex.

[kiates]

And here is another pair that couldn't resolve: BETA->USD

At this point I'm using the the following in the pair converter portion:

[dali.plugin.pair_converter.ccxt]
historical_price_type = nearest
default_exchange = Gate

[dali.plugin.pair_converter.historic_crypto]
historical_price_type = nearest

It's my understanding that you can have two as long as they aren't the same type.

I'm wondering if there is a way to make it put some kind of Keyword.UNKNOWN state instead of hard failing. Then at least it could finish processing, and leave you with a list of things that you may have to manually figure out and adjust.

[eprbell]

@macanudo527, what is the process of adding new routes? I'm wondering if we can make it easy for people to add their own routes when this kind of problem happens or if we can document the process of adding routes.

@kiates, you can probably hack your local tree to output UNKNOWN by modifying this small function:
https://github.com/eprbell/dali-rp2/blob/main/src/dali/transaction_resolver.py#L171

I'm not sure we'll want to merge that into the code though: the idea is that DaLI tries hard to generate correct files or it fails. This reduces support effort for the dev team.

[eprbell]

Also, yes, you can have multiple pair converters and they will be checked in order until a price is found (or until the list of pair converters is exhausted).

[macanudo527]

@macanudo527, what is the process of adding new routes? I'm wondering if we can make it easy for people to add their own routes when this kind of problem happens or if we can document the process of adding routes.

Right now it is ad-hoc addition of markets, but there will have to be an automatic process of doing this since the lists will quickly become long and unwieldy. Right now the priority is replacing the graph system with djikstra and a few more optimizations like lazy load. Once it is cleaned up we can work on automatic discovery of markets.

@kiates the reason the AR-> USD market is failing is because it doesn't exist. You can add AR -> USDT though. Or, if you are in Japan, you can just make Binance your default and it will resolve a lot of these issues, I think. The market with the 2nd highest volume is Kucoin so you should add that one in if you can't use Binance. Be sure to check CoinMarketCap to see which exchange has the best market.

[macanudo527]

That's an interesting case. Basically it would be an exchange-specific map of "artificial assets":
"artificial asset name" -> ("real asset", transformation())

So on Pionex, given an artificial asset like MBTC the map would point to a tuple (BTC, divide_by_1000()). This is just to point out the functionality, but the implementation would use an ArtificialAsset class, with a get_real_asset() and transform_value() methods. Note that this mapping would only be defined in the context of Pionex (it's not global). So if in the future an actual new asset called MBTC appears on the market, the above mapping would not be applied outside of Pionex.

I don't think it needs to be that complicated actually. You can just add aliases with set prices to the graph. So, you add the market MBTC -> BTC locked at the spot price of 0.00000001 for pionex. This same mechanism can be used to resolve the issues with Kraken's X-assets (xbtc, xeth, etc...) By adding a xbtc -> BTC market and a locked spot price of 1. As well as renamed assets.

[eprbell]

That's an interesting case. Basically it would be an exchange-specific map of "artificial assets":
"artificial asset name" -> ("real asset", transformation())
So on Pionex, given an artificial asset like MBTC the map would point to a tuple (BTC, divide_by_1000()). This is just to point out the functionality, but the implementation would use an ArtificialAsset class, with a get_real_asset() and transform_value() methods. Note that this mapping would only be defined in the context of Pionex (it's not global). So if in the future an actual new asset called MBTC appears on the market, the above mapping would not be applied outside of Pionex.

I don't think it needs to be that complicated actually. You can just add aliases with set prices to the graph. So, you add the market MBTC -> BTC locked at the spot price of 0.00000001 for pionex. This same mechanism can be used to resolve the issues with Kraken's X-assets (xbtc, xeth, etc...) By adding a xbtc -> BTC market and a locked spot price of 1. As well as renamed assets.

Nice: simpler and more elegant.

[macanudo527]

@kiates It appears you are doing some of your trading with Binance.com. If that is the case, you should just set the default exchange to Binance. That would resolve most of the issues except mBTC. BETA appears to be only available on the Binance ecosystem, so it is hard to get a reliable price without using Binance.com

There shouldn't be issues using Binance.com for pricing unless you are in the US.

I'll work on adding AR for now, since it is a pretty popular asset, but I can't really add BETA without using Binance.com

[kiates]

I can't use binance as I'm USA. When I've tried their API rejects any requests as they are geo-locked. I might be able to use a VPN temporarily to get around it.

[macanudo527]

I can't use binance as I'm USA. When I've tried their API rejects any requests as they are geo-locked. I might be able to use a VPN temporarily to get around it.

I wouldn't risk it. It seems like US authorities have made it their mission to seal off Binance.com in every way possible. And there is that new bill that is trying to make it illegal to access off-limits websites through VPNs.

The CSV files are still available for US citizens at least for a little while, so a CSV download plugin would help. It's on my to-do list, but not anytime soon. In the meantime, I don't think there is a way to accurately price BETA.

There was talk of implementing a Coingecko API pair converter that could be used in a last-resort situation like this. That is one way you could resolve it.

[kiates]

I do have access to Binance.US but I expect the coin list is very limited and not useful for this situation, plus Binance.US isn't supported yet.