feat: add support for creating version 0 transactions

[jstarry]

Problem

Can't create version 0 transactions with web3

Summary of Changes

• Added VersionedMessage type which can be used to represent legacy or v0 messages
• Added VersionedTransaction class which is message version agnostic by using VersionedMessage
• Added MessageV0 class to serialize/deserialize v0 messages
• Implemented serialization/deserialization for v0 transactions and messages
• Introduced MessageCompiledInstruction type as a successor to the CompiledInstruction type because it's inefficient and unnecessary to encode instruction data as a base58 string

Fixes #

[codecov]

Codecov Report

Merging #27142 (e92391c) into master (a97346a) will increase coverage by 0.1%.
The diff coverage is n/a.

@@            Coverage Diff            @@
##           master   #27142     +/-   ##
=========================================
+ Coverage    76.9%    77.0%   +0.1%     
=========================================
  Files          48       51      +3     
  Lines        2505     2649    +144     
  Branches      355      360      +5     
=========================================
+ Hits         1927     2041    +114     
- Misses        448      476     +28     
- Partials      130      132      +2     

[jordaaash]

Looks great, easy to follow, I had only minor feedback.

[jordaaash]

Should we add this to https://github.com/solana-labs/buffer-layout-utils/blob/master/src/web3.ts and consume it here? Could move most or all of these into it and eliminate some duplication

[jstarry]

I'd be happy with that but I don't feel it's a big priority for simple layouts for signatures and public keys

[jordaaash]

Feels like this should be accounts if it represents the same thing as in CompiledInstruction? MessageCompiledInstruction is a bit confusing as a name (it's just a CompiledInstruction with the data raw instead of encoded)

[jstarry]

MessageCompiledInstruction is meant to be the successor to CompiledInstruction. Encoding the data is unnecessary and inefficient so that was the main reason I wanted to move away from CompiledInstruction. The accounts -> accountKeyIndexes isn't as big of a deal but I feel that it's more clear and descriptive.

[jordaaash]

Conceptually feels a little weird that a TransactionVersion is referenced by a Message, instead of say a MessageVersion. While the transaction is versioned, isn't it because the message is versioned rather than the other way around?

[jstarry]

I personally feel that it's valid to talk about the version of a transaction when you really mean the version of a transaction's message. The only thing that makes this a "message version" to me is the need to encode the version in the signed message, but it really refers to the structure of the transaction as a whole. It's likely that in the future, the structure of the signatures portion of the transaction is updated and then versioning won't be specific to the message structure as it appears to be right now since v0 transactions didn't update how signatures are encoded.

[jordaaash]

Yeah, MessageVersion makes way more sense to me here. wdyt?

[jordaaash]

All this looks good. I'm imagining from #27213 that there will later be some higher level interface than using lookup table indexes directly?

[jstarry]

Yup, that higher level interface looks like this:

const txMessage = new TransactionMessage({
  instructions,
  recentBlockhash,
  payerKey,
});

const versionedTx = new VersionedTransaction(
  txMessage.compile({
    version: 0,
    addressLookupTableAccounts,
  });
);

[steveluscher]

I was looking for a way that you could architect this type such that it was impossible to screw it up. As written, you could create malformed data like this:

const matl: MessageAddressTableLookup = {
  accountKey: /* ... */,
  writableIndexes: [0, 1],
  readonlyIndexes: [1, 2],
};

If the storage was, instead, implemented as a sparse array, it would be impossible to overlap a readable/writeable index.

type Readable = 0;  // or 'r'
type Writeable = 1;  // or 'w'
type IndexesWithWriteability = (Readable | Writeable | undefined)[];
const indexes: IndexesWithWriteability = [];
indexes[0] = 1;
indexes[1] = 1;
indexes[1] = 0;  // OVERWRITTEN! Note how overlapping index 1 is impossible here.
indexes[2] = 0;

Then you'd do stuff like this at the point of serialization:

const readonlyIndexes = [];
const writeableIndexes = [];
indexes.forEach((writeability, index) => {
  if (writeability === 0) {
    readonlyIndexes.push(index);
  } else if (writeability === 1) {
    writeableIndexes.push(index);
  }
});
const writeableIndexesLength = shortvec.encodeLength(
  encodedWriteableIndexesLength,
  writeableIndexes.length,
);

[jstarry]

Oh yeah great point, it's definitely possible to create invalid transaction messages with the MessageV0 constructor. This MessageAddressTableLookup type directly maps to the serialization format of v0 messages (MessageV0 is just a barebones data class) so it's not meant to be created or handled directly by devs. In the draft PR here you can see that the construction of v0 messages has a high level API MessageV0.compile(..) which will build these lookup data structures internally in a valid way:

solana/web3.js/src/message/v0.ts

Lines 71 to 80 in 25d2566

	const lookupTableAccounts = args.addressLookupTableAccounts || [];
	for (const lookupTable of lookupTableAccounts) {
	const extractResult = compiledKeys.extractTableLookup(lookupTable);
	if (extractResult !== undefined) {
	const [addressTableLookup, loadedAddresses] = extractResult;
	addressTableLookups.push(addressTableLookup);
	allLoadedAddresses.writable.push(...loadedAddresses.writable);
	allLoadedAddresses.readonly.push(...loadedAddresses.readonly);
	}
	}

solana/web3.js/src/message/compiled-keys.ts

Lines 104 to 123 in 25d2566

	extractTableLookup(
	lookupTable: AddressLookupTableAccount,
	): [MessageAddressTableLookup, LoadedAddresses] | undefined {
	const [writableIndexes, drainedWritableKeys] =
	this.drainKeysFoundInLookupTable(
	lookupTable.state.addresses,
	keyMeta =>
	!keyMeta.isSigner && !keyMeta.isInvoked && keyMeta.isWritable,
	);
	const [readonlyIndexes, drainedReadonlyKeys] =
	this.drainKeysFoundInLookupTable(
	lookupTable.state.addresses,
	keyMeta =>
	!keyMeta.isSigner && !keyMeta.isInvoked && !keyMeta.isWritable,
	);
	// Don't extract lookup if no keys were found
	if (writableIndexes.length === 0 && readonlyIndexes.length === 0) {
	return;
	}

[steveluscher]

Yeah, only thing is that MessageV0 is exported in #27213, so by Murphy's Law someone will definitely eventually create one with hand rolled indices that overlap.

[jstarry]

I think that's fine. The same holds for any other field in MessageV0... staticAccountKeys could have dup keys, instructions could reference invalid account indexes, the header values could be contradictory, etc.

[steveluscher]

It's really stressing me out that we're going to have to hard-require the implementation for every version from here to the end of time, but I don't have anything helpful to suggest at this point :)

I wonder if, at some point, we let people create builds (hard forks? 😅) of web3.js that no longer include old transaction implementations, if their use case is to produce new transactions and not to interpret old ones.

That said, the existence of this module definitely encourages hard-requiring the whole universe. What would you think about getting rid of this, and instead offering a lightweight getSerializedMessageVersion() utility? That way the end user can decide if they want to:

• Hard require every version and switch between them, or
• Fatal/skip if they encounter a version they are not interested in supporting.

[jstarry]

I'm totally in favor of adding a utility to get the serialized message version but I think it's important to provide the VersionedMessage.deserialize utility in addition since anyone deserializing an arbitrary message would need this logic anyways. I think it only makes sense to forgo this utility if you don't need to deserialize messages at all and are only concerned with creating/serializing a particular set of transaction versions. I'm trying to modularize the code that's specific to a particular transaction version so I think we're in decent position for that already.

[jstarry]

Here's a PR to add that utility function you suggested: #27415