Extra: Optimize contract data storage

[acuarica]

After measuring gas usage by contract (see #165), we might want to optimize how contract stores data.

We could improve data storage by using a LookupMap instead of an UnorderedMap. As described in [1], LookupMap does not permit to traverse its elements. However, since we are using a linked list to traverse those elements in insertion order, we could be better of with LookupMap.

[1] https://docs.near.org/docs/concepts/data-storage#rust-collection-types

[acuarica]

Another possibility would be to use the intrinsic linked list of blocks to retrieve Corgis.

If we go with this alternative, we will need to query the RPC for linked transactions. The CI deploy script query previous transactions to get the previous created accounts, it might be useful:

corgis/ci/deploy.mjs

Lines 86 to 138 in 20f55f9

	async function giveMeMyMoneyBack(accountId) {
	class Explorer {
	constructor() {
	this.connection = new autobahn.Connection({
	realm: "near-explorer",
	transports: [
	{
	url: 'wss://near-explorer-wamp.onrender.com/ws',
	type: "websocket",
	},
	],
	retry_if_unreachable: true,
	max_retries: Number.MAX_SAFE_INTEGER,
	max_retry_delay: 10,
	});
	}
	open() {
	return new Promise((resolve, reject) => {
	this.connection.onopen = (session) => resolve(session);
	this.connection.onclose = (reason) => reject(reason);
	this.connection.open();
	});
	}
	close() {
	this.connection._transport.onclose = () => { };
	this.connection.close();
	}
	}
	const c = new Explorer();
	const s = await c.open();
	const sqlQuery = `SELECT DISTINCT t.receiver_account_id FROM transactions t LEFT JOIN transaction_actions ta ON ta.transaction_hash=t.transaction_hash WHERE ta.action_kind='CREATE_ACCOUNT' AND t.signer_account_id='${accountId}'`;
	const result = await s.call(`com.nearprotocol.${'testnet'}.explorer.select:INDEXER_BACKEND`, [sqlQuery]);
	c.close();
	for (const { receiver_account_id } of result) {
	process.stdout.write(` > Fetching contract account ${receiver_account_id} .. `);
	try {
	const contractAccount = await near.account(receiver_account_id);
	try {
	await contractAccount.deleteAccount(accountId);
	console.log(`account succefully deleted`);
	} catch (e) {
	console.log(`account could not be deleted`);
	}
	} catch (e) {
	console.log(`account does does not exist`);
	}
	}
	}

From Sherif@NEAR in the Discord chat on how use RPC as a linked list:

the video includes some very relevant discussion about storage strategies for apps with high frequency data updates and moderately large data sets.

Eugene is taking advantage of NEAR's low cost of overwriting data (vs writing new data) to replace the previous post with each new one.

He finds older posts by following a linked list of posts using block height as the reference to the next post.

Assuming a constraint of one post per block per user (blocks are 1 second so that's very reasonable), this means each post can be queried directly via RPC using user account and block height to traverse the tree

His frontend grabs the most recent post from the blockchain then hits validator RPC nodes for previous 10 posts before switching to hit archival RPC nodes until 100 posts are loaded

also discussed in this video:

• dynamically estimating storage costs of a contract
• upgradeable contracts

https://youtu.be/1i-HnhwoNWg

source code: https://github.com/evgenykuzyakov/near-fm

[acuarica]

Just for reference, Rust contracts binaries are bloated.

See near/near-sdk-rs#167

[acuarica]

Dict is now using LookupMap. Storage was reduced almost 50% when creating a corgi.