Optimize input and output storage

[yanguoyu]

Now, every input cell and output cell in the transaction that includes the user's cell will save in SQLite.
If a user deposits ckb from an exchange, the transaction will include many inputs and outputs. I found that a user deposits many times from an exchange. And then the input entity and output entity have 500,000 rows and 1000,000 rows, but most of them are not belong to the user. It may affect SQL execution time.
So we can save input cells and output cells that only belong to the users, and if we need to get the transaction's detail, we can get it by RPC service.

[yanguoyu]

This scheme does not work, because in the history page, users can search by other's address.
https://github.com/nervosnetwork/neuron/blob/v0.106.0/packages/neuron-wallet/src/services/tx/transaction-service.ts#L74-L98

[yanguoyu]

How about searching by https://github.com/nervosnetwork/ckb-indexer#get_transactions when the user search's address is not theirs? @Keith-CY
And I need to check whether there are exist any else that needs other's cells.

[Keith-CY]

How about searching by nervosnetwork/ckb-indexer#get_transactions when the user search's address is not theirs? @Keith-CY And I need to check whether there are exist any else that needs other's cells.

It seems work

[yanguoyu]

How about searching by nervosnetwork/ckb-indexer#get_transactions when the user search's address is not theirs? @Keith-CY And I need to check whether there are exist any else that needs other's cells.

It seems work

There may exist a problem if the user enters an address(e.g. a miner's address) that has many txs, the RPC may use more times to search txs.
It's better to get the current wallet's all tx details, then we should use batch request get_transaction RPC to get all transactions' detail, and then filter by lockArgs.

[yanguoyu]

It will implement by these steps:

1. Write migrate to clear InputEntity, OutputEntity, TransactionEntity, IndexerTxHashCache
2. When calling https://github.com/nervosnetwork/neuron/blob/v0.106.0/packages/neuron-wallet/src/block-sync-renderer/sync/queue.ts#L103, remove the inputs and outputs that not belong to the current wallet.
3. When searching transactions by address, if this address belongs to the current wallet, keep the old logic, else get all transactions' detail in the current wallet and then filter with the address.
4. When getting transaction detail, use call get_transaction to replace.

[Keith-CY]

2. When calling https://github.com/nervosnetwork/neuron/blob/v0.106.0/packages/neuron-wallet/src/block-sync-renderer/sync/queue.ts#L103, remove the inputs and outputs that not belong to the current wallet.

This method will be called multiple times and most of them are useless, how about remove the historical data by clear cache

[yanguoyu]

This method will be called multiple times and most of them are useless, how about remove the historical data by clear cache

I don't get this point. This method is used to sync transactions to DB. I mean remove input and ouputs that don't belong to the current wallet before saving them to DB.

[Keith-CY]

This method will be called multiple times and most of them are useless, how about remove the historical data by clear cache

I don't get this point. This method is used to sync transactions to DB. I mean remove input and ouputs that don't belong to the current wallet before saving them to DB.

I see, I thought it's used to remove historical data

[yanguoyu]

This method will be called multiple times and most of them are useless, how about remove the historical data by clear cache

I don't get this point. This method is used to sync transactions to DB. I mean remove input and ouputs that don't belong to the current wallet before saving them to DB.

I see, I thought it's used to remove historical data

I will write migrate to remove historical data in step 1, it's the same as clear cache.

[yanguoyu]

I found another problem, when we get the transaction list we need to get all input and output to calculate receive ckb or send ckb.
It seems that there is no need to optimize because getting the transaction list is a frequent operation.
@Keith-CY What do you think?

[Keith-CY]

I found another problem, when we get the transaction list we need to get all input and output to calculate receive ckb or send ckb. It seems that there is no need to optimize because getting the transaction list is a frequent operation. @Keith-CY What do you think?

But the input/output owned by other addresses are not used in the computation, they are still removable

[yanguoyu]

But the input/output owned by other addresses are not used in the computation, they are still removable

Sound like that, I make a mistake to think it needs another's cell to calculate.

[yanguoyu]

Here https://github.com/nervosnetwork/neuron/blob/develop/packages/neuron-wallet/src/block-sync-renderer/sync/indexer-cache-service.ts#L81 also can be optimized, it gets all transactions from the node every time. We can cache the last block number and get it from the last block number next time.

[yanguoyu]

nervosnetwork/neuron#2672