tip: 383
title: Optimize transaction cache loading
author:[email protected]
discussions to: https://github.com/tronprotocol/TIPs/issues/383
status: Final
type: Standards Track
category: Core
created: 2022-02-28
This TIP is to describes how to optimize transaction cache loading.
When start the node with the official suggested machine configuration (16cores, 32G, SSD) and the provided startup script. It takes a long time to start syncing blocks each time. Through analysis, it is found that transaction cache loading consumes too much time. This article details the optimization of the transaction cache loading.
Node startup speed is too slow, 80% of the time is spent on transaction cache loading. In order to speed up node startup and provide a better user experience, it is necessary to reduce the transaction loading time.
The storage unit corresponding to each block, each storage unit consists of two fields: block number and transaction id list, The data structure is defined as follows.
public class RecentTransactionItem {
private long num;
private List<String> transactionIds;
}
The traversal of the database is time-consuming. If there are too many keys, the consumption time will be very long. In order to speed up the traversal speed, the amount of data should be reduced.
The following is the time-consuming statistics of transaction cache loading before and after optimization. It can be found that the loading speed is significantly improved after optimization.
| loading cost | |
|---|---|
| before optimization | 52,104ms |
| after optimization | 26,448ms |
In the current design, the transaction cache database stores the transaction hashes of the last 70,000 blocks, and each transaction stores one piece of data. When processing blocks, each transaction will be written to the database, and the implementation logic is as follows.
Optional.ofNullable(transactionCache)
.ifPresent(t -> t.put(trxCap.getTransactionId().getBytes(),
new BytesCapsule(ByteArray.fromLong(trxCap.getBlockNum()))));
If there are 200 transactions per block, there are a total of 14,000,000 pieces of data, which is relatively large , when looping through the database, it takes nearly 30 seconds. The transaction cache database loading logic is as follows.
private void init() {
DBIterator iterator = (DBIterator) persistentStore.iterator();
while (iterator.hasNext()) {
Entry<byte[], byte[]> entry = iterator.next();
byte[] key = entry.getKey();
byte[] value = entry.getValue();
if (key == null || value == null) {
return;
}
Key k = Key.copyOf(key);
Long v = Longs.fromByteArray(value);
blockNumMap.put(v, k);
db.put(k, v);
}
}
It used to store one piece of data per transaction, but now it can be designed to store one piece of data per block, the key is block number, and the value is RecentTransactionItem, the implementation logic is as follows. This greatly reduces the amount of data and shortens the database traversal time.
public void updateRecentTransaction(BlockCapsule block) {
List list = new ArrayList<>();
block.getTransactions().forEach(capsule -> {
list.add(capsule.getTransactionId().toString());
});
RecentTransactionItem item = new RecentTransactionItem(block.getNum(), list);
chainBaseManager.getRecentTransactionStore().put(
ByteArray.subArray(ByteArray.fromLong(block.getNum()), 6, 8),
new BytesCapsule(JsonUtil.obj2Json(item).getBytes()));
}