Tracking issue: DEX state legibility for API consumers

[zbuc]

Is your feature request related to a problem? Please describe.

To enable ecosystem participants like ZKV to begin building tooling on top of the Penumbra chain, we need to build out the node APIs (ObliviousQueryService and SpecificQueryService) to provide necessary data.

Describe the solution you'd like

It's currently unclear what the specific needs from ZKV will be, but Natalia shared a high-level description with us:

Objective - to give a user a complete info about the health of the ecosystem, what’s going on and how profitable it is to interact with it

Features:

- Check the history of the growth TVL
- Test price for swap i would want to make
- Check profitability of put funds
- Check liquidity of the pools
- check in on growth of diff pools
- Check available pools
- Check which tokens are supported
- see which token is most popular
- see Governance and votes

Possible further down the line:

- check in on anonymity set size
- check personal funds (via wallet widget)
- check your own transaction history
so something similar to https://info.uniswap.org/#/ but for Penumbra

The specifics for ZKV's dashboard need to be re-evaluated as some of the things here don't make much sense for Penumbra (TVL as a metric, "available pools", etc.)

We should: start with high-level use cases and determine the RPCs necessary to make them work and come up with a diff against our currently available RPCs.

Re: depth chart, what does it mean to have a depth chart in the Penumbra context, where you're not restricted to liquidity on a single pair? i.e. there are multiple ways to trade from A -> B through intermediary assets that would not be represented in a traditional depth chart.

Possibly represent the "direct" depth curve as well as one that takes routing into account in the same chart?

[hdevalence]

Comments during a discussion with @zbuc @erwanor:

• Check the history of the growth TVL

There are two points here:

1. The historical data part: how do we expose historical data to consumers? We can do historical queries for the JMT state, but only if we don't prune/compact the database state. We also don't want to have state bloat of the on-chain data. Probably the best option in the near term is to use Tendermint events to feed data to an external indexer which can process it or answer other queries. Alternatively, we could build custom GRPC interfaces for state updates, but while this would be very powerful, it would require custom tooling.
2. The TVL part: we can definitely record the TVL, and it's probably good to do so. But measuring DeFi protocols by TVL is like measuring aircraft by weight -- we expect Penumbra to be significantly more capital-efficient than alternatives, so the TVL number probably won't be a very good reflection of things users care about, like prices / slippage.

• Test price for swap i would want to make

We think the best way to implement this would be to have an RPC that takes a trade as input and simulates execution against the current chain state. Because of the way we've built the Penumbra internals (all execution is simulated, we just selectively commit it), this can run the exact same code we would use to execute a trade, so should take essentially no extra development effort on top of the effort needed to implement execution.

However, it's possible that this RPC might be expensive to compute, especially after we do routing, so we might want to have pd be able to configure access to it or rate-limit it.

• Check profitability of put funds

Assuming this means being able to compute the profitability of a specific LP position (identified by PositionId, this probably involves:

1. Looking up the height at which a position was created and what its initial reserves were
2. Looking up price information for those assets at that height
3. Combining the initial reserves with the price information to compute a cost basis in some numeraire (which may not be any on-chain asset, e.g., the numeraire might be "fiat USD")
4. Looking up the most recent reserves and the height of those reserves

• If the position is open, the height is the current height and the most recent reserves are the current ones
• If the position is closed, the most recent reserves are the ones at time of closing and the height is the closing height

5. Looking up price information for those assets at the relevant height
6. Combining the most recent reserves with the price data to compute the current portfolio value in some numeraire
7. Looking up any retroactively issued rewards for the position
8. Looking up price information for those rewards at the relevant height
9. Combining the reward amounts with the price data to compute the reward amount.

The question is, which of this information should be on-chain, which should be indexed separately, and which should be externally provided?

• The most recent reserves are stored on-chain, since they're needed for the dex to function, and can be queried directly.
• The initial reserves are not stored on-chain, but could be indexed using Tendermint events
• The reward mechanics are TBD dex: design details of LP reward claim mechanism #2167 but, depending on the design, could either be stored on chain or indexed using Tendermint events
• The price information could potentially come from the chain, but it might be better to consult an external price oracle (e.g., if USDC depegs, what happens? Or, what if the prices on Penumbra are out-of-line with external reference markets?)

• Check liquidity of the pools

This is a bit tricky because Penumbra doesn't have "pools" as such, we have (effectively) an orderbook for each pair.

We could expose the order book state for each pair in a few different ways. We could provide streams of the raw positions on that pair, ordered by their effective price. Or we could provide summarized depth charts, say in 1bps price increments.

This might also not be the most meaningful representation, either, because we don't execute on individual pairs, we execute on the graph. Let's call liquidity on other pairs that can be used to execute a trade on the target pair "synthetic liquidity". It seems useful to provide a synthetic depth chart that shows the order book as if synthetic liquidity were included. How should this be represented? Probably we want the underlying data to include information about the routes (e.g., this section of the synthetic order book runs along path P1, this section along P2, etc).

• check in on growth of diff pools

Penumbra doesn't have pools, but it could be possible to represent aggregates of individual positions for trading pairs. This would be the same as the previous item ("liquidity of the pools") AFAIK. Tracking growth would likely be best handled by using Tendermint events or custom indexing on ZKV's part rather than tracking historical data on-chain.

• Check available pools

Similar to checking liquidity.

• Check which tokens are supported

We have an oblivious API already that returns the known asset listing.

• see which token is most popular

Most popular by which metrics? Transfers, swaps, most liquidity available in positions? This is likely not an API we'd build into the chain, and would be better implemented by ZKV on top of raw data we can provide once we have a better understanding.

• see Governance and votes

[zbuc]

We should fill in the missing bullet points and do a round-trip with ZKV to get their feedback on our proposed changes

[plutoegg]

Thank you for the comments, and first I want to just address some of the terminology around pools, Penumbra liquidity, and goals of our dashboard:

The specifics for ZKV's dashboard need to be re-evaluated as some of the things here don't make much sense for Penumbra (TVL as a metric, "available pools", etc.)

Re: depth chart, what does it mean to have a depth chart in the Penumbra context, where you're not restricted to liquidity on a single pair? i.e. there are multiple ways to trade from A -> B through intermediary assets that would not be represented in a traditional depth chart.

• We understand your points here, and the use of the term 'pools' is perhaps not correct, however from the future DEX user of Penumbra, we think you are overestimating the difference that users (both LPs and traders) will see compared to UniswapV3 + aggregator layer (i.e. 1inch and Paraswap)
• We are trying to help users and traders visualise the available liquidity, and ultimately the depth chart on a particular pair still matters to them (even though trades will be routed via multiple paths to get the best price
• Uniswap v3 visualisations of liquidity are relevant here (see screenshot below). Yes, there is no single pool, and this profile is made up of many different individual liquidity positions, to create the overall orderbook, but to a trader it functions as a single 'pool' (or order-book if you prefer), and this is what we are referring to and aiming to visualise:

• We believe that visualising the liquidity on a specific pair (i.e. ATOM:USDC) is still useful and interesting, even though a trade could be routed via all available best liquidity
• If you look at routing visualisation on popular Ethereum DEX aggregators, for small trades, they will almost always go via just the most popular route, and even large trades will usually be split across max 2-3 different routes. See examples below:

• Ultimately liquidity concentrates on the most used pairs. It is very uncommon even for large trades via aggregators on Ethereum that you see routes that involve more than 3 splits to get the best price (i.e. see screenshots), and this is because LPs are reactive to where they can earn the most fees, and provide more liquidity on these popular pairs. The same effect may be more limited on Penumbra, because LPs won't have the same visibility of where trades are routed, but they should be able to at least backwards engineer this information based on their fee revenues.

We think the best way to implement this would be to have an RPC that takes a trade as input and simulates execution against the current chain state. Because of the way we've built the Penumbra internals (all execution is simulated, we just selectively commit it), this can run the exact same code we would use to execute a trade, so should take essentially no extra development effort on top of the effort needed to implement execution.

• We definitely like the idea of simulating the trade, in order to get the best price and using that to show users the overall price across multiple routes and all available individual positions
• However we don't think this should be the only visualisation of liquidity on Penumbra, and think we can come up with better visual representations of available liquidity, more akin to Uniswap v3

---

I'll pause there, and separately leave comments on the other aspects from ZKV's perspective, and on ZKV's endpoint and historical data needs in a separate post.

But hope that clarifies why we think that visualisation of Penumbra liquidity in a way that is similar to UniswapV3 will still be useful to LPs and traders, despite trades being executed against the full graph and not just individual routes / positions.

[hdevalence]

• We understand your points here, and the use of the term 'pools' is perhaps not correct, however from the future DEX user of Penumbra, we think you are overestimating the difference that users (both LPs and traders) will see compared to UniswapV3 + aggregator layer (i.e. 1inch and Paraswap)

• We are trying to help users and traders visualise the available liquidity, and ultimately the depth chart on a particular pair still matters to them (even though trades will be routed via multiple paths to get the best price

Yes, to clarify, we weren't suggesting that the depth chart on a particular pair doesn't matter to users, it definitely does. The concern was just about how we can convey to users that the liquidity used to execute a trade from asset A to B is not restricted to the liquidity directly between A and B.

• Uniswap v3 visualisations of liquidity are relevant here (see screenshot below). Yes, there is no single pool, and this profile is made up of many different individual liquidity positions, to create the overall orderbook, but to a trader it functions as a single 'pool' (or order-book if you prefer), and this is what we are referring to and aiming to visualise:

• We believe that visualising the liquidity on a specific pair (i.e. ATOM:USDC) is still useful and interesting, even though a trade could be routed via all available best liquidity

Yes, we also think this would be great to display. One idea we'd discussed about how to convey the availability of routing was to display two overlaid charts, one with direct liquidity and the other with synthetic liquidity. This would allow users to see both where liquidity is allocated, and also what liquidity is available for them to trade — and to see the spread between those curves, which will vary by pair.

In rendering the dashboard, do you think it would be better to have a liquidity-concentration-style rendering like the Univ3 one in the screenshot, or a more traditional market depth chart?

• If you look at routing visualisation on popular Ethereum DEX aggregators, for small trades, they will almost always go via just the most popular route, and even large trades will usually be split across max 2-3 different routes. See examples below:

Yes, we also expect this to be an emergent behavior, and it's why our routing algorithm (#2283) alternates between a graph-traversal "spill phase" whose complexity is independent of the number of positions/orders on each pair and a straight-line "fill phase" that walks up the joint orderbook of a specific route. This way, in the common case where there are only a few relevant routes, we can spend most of our execution time filling along a specific route.

• Ultimately liquidity concentrates on the most used pairs. It is very uncommon even for large trades via aggregators on Ethereum that you see routes that involve more than 3 splits to get the best price (i.e. see screenshots), and this is because LPs are reactive to where they can earn the most fees, and provide more liquidity on these popular pairs. The same effect may be more limited on Penumbra, because LPs won't have the same visibility of where trades are routed, but they should be able to at least backwards engineer this information based on their fee revenues.

All of the routing and execution is public, so in principle LPs on Penumbra should have exactly the same visibility about how trades were routed.

From a visualization standpoint, it seems like a Sankey diagram could be a nice way to display this data. For each block, it should be possible to see the actually-executed route for each batched trade, as well as to see any arbitrage trades performed by the chain. This seems like useful information to display retrospectively.

We just need to ensure that as we do that routing and execution on-chain, we produce the data to make it legible to external consumers. So if displaying routes seems useful, it would be helpful to settle on a data structure for recording them.

We think the best way to implement this would be to have an RPC that takes a trade as input and simulates execution against the current chain state. Because of the way we've built the Penumbra internals (all execution is simulated, we just selectively commit it), this can run the exact same code we would use to execute a trade, so should take essentially no extra development effort on top of the effort needed to implement execution.

• We definitely like the idea of simulating the trade, in order to get the best price and using that to show users the overall price across multiple routes and all available individual positions

• However we don't think this should be the only visualisation of liquidity on Penumbra, and think we can come up with better visual representations of available liquidity, more akin to Uniswap v3

Yes, we agree, sorry for the confusion. What we were thinking of here was not the frontend visualization, but the backend RPCs we could use to deliver the required data:

• For liquidity/depth charts, simulating routing allows rendering both direct and synthetic liquidity on the same chart, showing users how much liquidity is directly provisioned on the pair, and how much liquidity is available by routing through other pairs

• For routing charts / Sankey diagrams, it allows the visualization to be done not just retrospectively ("I did a trade in this block, show me how it was actually executed") but prospectively ("If I did do a trade in the next block, show me what its route might be").

Of these two use cases, the second one is probably a lower priority, because it is both less important and also more complicated from a deployment perspective, since it would mean having user-driven queries. The data for the first one can be cached, so it's computed at most once per block for each visualized pair.

Hope this clarifies things — I think we are actually much closer aligned with your perspective than we may have appeared!

[erwanor]

We should create a tracking issue from this design issue, so that we can breakdown this effort into action items.

[hdevalence]

xref #2576