Merge remote-tracking branch 'upstream/master'

Makefile

@@ -49,7 +49,15 @@ build:
 	GOARCH=wasm GOOS=js go build -o ./runtime/cmd/parse/parse.wasm ./runtime/cmd/parse
 	go build -o ./runtime/cmd/check/check ./runtime/cmd/check
 	go build -o ./runtime/cmd/main/main ./runtime/cmd/main
-	cd ./languageserver && make build
+	(cd ./languageserver && make build && cd -)
+	make build-tools
+
+.PHONY: build-tools
+build-tools:
+	(cd ./tools/analysis && go build . && cd -)
+	(cd ./tools/batch-script && go build . && cd -)
+	(cd ./tools/constructorcheck && make plugin && cd -)
+	(cd ./tools/docgen && make build && cd -)
 
 .PHONY: lint-github-actions
 lint-github-actions: build-linter

meetings/2022-07-22.md

@@ -0,0 +1,106 @@
+
+# July 22, 2022
+
+## External Mutability
+
+- FLIP that prevents external mutation was approved, implemented, and released in Secure Cadence:
+  [https://github.com/onflow/flow/pull/703](https://github.com/onflow/flow/pull/703)
+- Open problems:
+  - Composites were not covered by the FLIP, only arrays and dictionaries
+  - Externally taking a reference to a field can circumvent the external mutability restriction
+- New FLIP: [https://github.com/onflow/flow/pull/1056](https://github.com/onflow/flow/pull/1056)
+  - Purity/mutability modifier for functions
+  - Automatic analysis to determine if function has side-effects
+
+### Condition can currently have side effects:
+- https://github.com/onflow/cadence/issues/1805
+- Might be used for underhanded code: Post-condition might perform mutation. Example: FT withdrawal
+- Might be solved by purity FLIP
+- Can a pure function emit events?
+  - Does affect on-chain state, but is not observable
+- Are other side-effecting functons like debugging logging allowed?
+- "Pure" vs "no side-effects"
+  - "Pure" usually means no side-effects at all
+  - Have to be pragmatic
+- Alternatives:
+  - Prevent calls in general
+  - Run-time errors if there are side-effects
+
+## Remove deprecated key API
+- We currently have two key management APIs:
+  - `AuthAccount.addPublicKey`/`AuthAccount.removePublicKey`:
+    - Low-level, accepts encoded key as byte array
+    - Deprecated a year ago
+  - [`AuthAccount.keys.add`](https://docs.onflow.org/cadence/language/accounts/#add-account-keys)/`AuthAccount.keys.remove`
+    - High-level, accepts key as object (`PublicKey`)
+    - Added a year ago
+- We attempted to remove the deprecated in the Secure Cadence release
+  - Rolled back, as it will break client applications if they use SDK which uses deprecated API
+- Also: Events that are emitted when a key is added is inconsistent, depends on which API is used: https://github.com/onflow/cadence/issues/1796
+- Do we have metrics on how many SDKs are still using the deprecated API?
+- Actionables:
+  - Add deprecation annotation to developer tooling
+    - Pragma, e.g. `#deprecated`
+    - Comment, e.g. `@deprecated`
+    - Add support in VS Code, render as ~~addPublicKey~~
+
+## Interface default methods
+- https://github.com/onflow/cadence/pull/1076
+- Proposed for NFT metadata standard
+- Helps with preventing breakage in ecosystem
+- Avoid copy/paste and boilerplate, making code reusable
+- Does not help with FT/NFT rewrite proposal
+- Useful for utility/convenience functions, reducing boilerplate
+- Open Questions:
+  - Security implications?
+    - Related:
+      - Post-conditions with side-effects
+      - Mutability
+        - Restrict to pure functions? Potentially too restrictive, e.g. couldn’t provide
+        - Maybe allow opt out of mutating functions. But could also be used to opt-out of a fix
+    - Examples?
+    - Trust relationship problem
+      - Already have this problem with contract updates
+    - Modification of default implementation
+    - Would be useful to provide utility functions, like FT transfer
+    - It should not be possible to perform malicious code, as the same could be done in a non-default function
+- Not a breaking change, could merge and ship before Stable Cadence
+- When there is a conflict, e.g multiple interfaces provide default implementation:
+  - No "winner" is selected by default
+  - Instead, the user is required to provide an implementation
+- Follow up feature: Interface requirements
+  - Declaration site: `CI1: CI2`
+  - Use-site would still have to explicitly declare conformance for all interfaces: `C: CI1, CI2`
+
+## Storage Querying/Iteration API
+- https://github.com/onflow/cadence/issues/208
+- Finally added technical foundation a couple months ago:
+  Account storage data is now stored in [atree](https://github.com/onflow/atree) values
+  (atree is Cadence's storage layer)
+- Issue lays out proposal for adding API to iterate over account paths/values
+- Outstanding issues:
+  - Accounts may store a lot of data. Pagination iteration? Iterator, cursor, enumerator
+  - Concrete API proposal
+- First use-case: Developer tools like the Emulator and Playground
+- Need to show wallet developers how they can render information to users
+- Mutability not necessarily a problem: Run script against block.
+  - Problem: data availability
+  - Might be solved in future by "historic" data node
+- Storage Layer:
+  - Iteration over dictionary changes when mutated
+  - Keys are hashed
+- Start with MVP
+- Stages:
+  - 1. Even useful if pagination problem is not solved yet
+  - 2. Pagination for larger accounts
+  - 3. Support for handling mutation
+- Also useful for e.g. dictionary, keys. Might be too large
+  - Keys on demand
+  - E.g. useful to pick a random key
+
+## Organisational
+
+- Should we move this meeting so it is not on a Friday evening for folks in Europe?
+  - Not Friday, not Monday
+  - Tuesday is best
+  - Same time (8am PT)

runtime/compiler/compiler.go

@@ -105,7 +105,7 @@ func (compiler *Compiler) VisitVariableDeclaration(declaration *ast.VariableDecl
 	// TODO: second value
 
 	identifier := declaration.Identifier.Identifier
-	targetType := compiler.Checker.Elaboration.VariableDeclarationTargetTypes[declaration]
+	targetType := compiler.Checker.Elaboration.VariableDeclarationTypes[declaration].TargetType
 	valType := compileValueType(targetType)
 	local := compiler.declareLocal(identifier, valType)
 	exp := declaration.Value.Accept(compiler).(ir.Expr)

runtime/convertValues.go

@@ -1373,7 +1373,7 @@ func importCompositeValue(
 	fieldTypes []cadence.Field,
 	fieldValues []cadence.Value,
 ) (
-	*interpreter.CompositeValue,
+	interpreter.Value,
 	error,
 ) {
 	var fields []interpreter.CompositeField
@@ -1424,12 +1424,12 @@ func importCompositeValue(
 		case sema.HashAlgorithmType:
 			// HashAlgorithmType has a dedicated constructor
 			// (e.g. it has host functions)
-			return importHashAlgorithm(inter, fields)
+			return importHashAlgorithm(fields)
 
 		case sema.SignatureAlgorithmType:
 			// SignatureAlgorithmType has a dedicated constructor
 			// (e.g. it has host functions)
-			return importSignatureAlgorithm(inter, fields)
+			return importSignatureAlgorithm(fields)
 
 		default:
 			return nil, errors.NewDefaultUserError(
@@ -1460,7 +1460,7 @@ func importPublicKey(
 ) {
 
 	var publicKeyValue *interpreter.ArrayValue
-	var signAlgoValue *interpreter.CompositeValue
+	var signAlgoValue *interpreter.SimpleCompositeValue
 
 	ty := sema.PublicKeyType
 
@@ -1480,7 +1480,7 @@ func importPublicKey(
 			publicKeyValue = arrayValue
 
 		case sema.PublicKeySignAlgoField:
-			compositeValue, ok := field.Value.(*interpreter.CompositeValue)
+			compositeValue, ok := field.Value.(*interpreter.SimpleCompositeValue)
 			if !ok {
 				return nil, errors.NewDefaultUserError(
 					"cannot import value of type '%s'. invalid value for field '%s': %v",
@@ -1528,10 +1528,9 @@ func importPublicKey(
 }
 
 func importHashAlgorithm(
-	inter *interpreter.Interpreter,
 	fields []interpreter.CompositeField,
 ) (
-	*interpreter.CompositeValue,
+	interpreter.MemberAccessibleValue,
 	error,
 ) {
 
@@ -1570,14 +1569,14 @@ func importHashAlgorithm(
 		)
 	}
 
-	return stdlib.NewHashAlgorithmCase(inter, uint8(rawValue)), nil
+	// TODO: return existing
+	return stdlib.NewHashAlgorithmCase(rawValue), nil
 }
 
 func importSignatureAlgorithm(
-	inter *interpreter.Interpreter,
 	fields []interpreter.CompositeField,
 ) (
-	*interpreter.CompositeValue,
+	interpreter.MemberAccessibleValue,
 	error,
 ) {
 
@@ -1616,5 +1615,6 @@ func importSignatureAlgorithm(
 		)
 	}
 
-	return stdlib.NewSignatureAlgorithmCase(inter, uint8(rawValue)), nil
+	// TODO: return existing
+	return stdlib.NewSignatureAlgorithmCase(rawValue), nil
 }

runtime/convertValues_test.go

@@ -407,7 +407,7 @@ func TestExportValue(t *testing.T) {
 							return nil
 						},
 					),
-					stdlib.NewHashAlgorithmCase(inter, 1),
+					stdlib.NewHashAlgorithmCase(1),
 					interpreter.NewUnmeteredUFix64ValueWithInteger(10),
 					false,
 				)

runtime/interpreter/accountkey.go

@@ -37,7 +37,7 @@ func NewAccountKeyValue(
 	inter *Interpreter,
 	keyIndex IntValue,
 	publicKey *CompositeValue,
-	hashAlgo *CompositeValue,
+	hashAlgo Value,
 	weight UFix64Value,
 	isRevoked BoolValue,
 ) *SimpleCompositeValue {

runtime/interpreter/interpreter.go

@@ -237,7 +237,7 @@ type SignatureVerificationHandlerFunc func(
 	signature *ArrayValue,
 	signedData *ArrayValue,
 	domainSeparationTag *StringValue,
-	hashAlgorithm *CompositeValue,
+	hashAlgorithm *SimpleCompositeValue,
 	publicKey MemberAccessibleValue,
 ) BoolValue
 
@@ -1866,7 +1866,10 @@ func (interpreter *Interpreter) declareEnumConstructor(
 	intType := sema.IntType
 
 	enumCases := declaration.Members.EnumCases()
-	caseValues := make([]*CompositeValue, len(enumCases))
+	caseValues := make([]struct {
+		Value    MemberAccessibleValue
+		RawValue IntegerValue
+	}, len(enumCases))
 
 	constructorNestedVariables := map[string]*Variable{}
 
@@ -1877,7 +1880,7 @@ func (interpreter *Interpreter) declareEnumConstructor(
 			NewIntValueFromInt64(interpreter, int64(i)),
 			intType,
 			compositeType.EnumRawType,
-		)
+		).(IntegerValue)
 
 		caseValueFields := []CompositeField{
 			{
@@ -1897,7 +1900,13 @@ func (interpreter *Interpreter) declareEnumConstructor(
 			caseValueFields,
 			common.Address{},
 		)
-		caseValues[i] = caseValue
+		caseValues[i] = struct {
+			Value    MemberAccessibleValue
+			RawValue IntegerValue
+		}{
+			Value:    caseValue,
+			RawValue: rawValue,
+		}
 
 		constructorNestedVariables[enumCase.Identifier.Identifier] =
 			NewVariableWithValue(interpreter, caseValue)
@@ -1918,27 +1927,29 @@ func (interpreter *Interpreter) declareEnumConstructor(
 }
 
 func EnumConstructorFunction(
-	inter *Interpreter,
+	gauge common.MemoryGauge,
 	getLocationRange func() LocationRange,
 	enumType *sema.CompositeType,
-	caseValues []*CompositeValue,
+	cases []struct {
+		Value    MemberAccessibleValue
+		RawValue IntegerValue
+	},
 	nestedVariables map[string]*Variable,
 ) *HostFunctionValue {
 
 	// Prepare a lookup table based on the big-endian byte representation
 
-	lookupTable := make(map[string]*CompositeValue)
+	lookupTable := make(map[string]Value, len(cases))
 
-	for _, caseValue := range caseValues {
-		rawValue := caseValue.GetField(inter, getLocationRange, sema.EnumRawValueFieldName)
-		rawValueBigEndianBytes := rawValue.(IntegerValue).ToBigEndianBytes()
-		lookupTable[string(rawValueBigEndianBytes)] = caseValue
+	for _, c := range cases {
+		rawValueBigEndianBytes := c.RawValue.ToBigEndianBytes()
+		lookupTable[string(rawValueBigEndianBytes)] = c.Value
 	}
 
 	// Prepare the constructor function which performs a lookup in the lookup table
 
 	constructor := NewHostFunctionValue(
-		inter,
+		gauge,
 		func(invocation Invocation) Value {
 			rawValue, ok := invocation.Arguments[0].(IntegerValue)
 			if !ok {
@@ -1949,7 +1960,7 @@ func EnumConstructorFunction(
 
 			caseValue, ok := lookupTable[string(rawValueArgumentBigEndianBytes)]
 			if !ok {
-				return NewNilValue(inter)
+				return NewNilValue(gauge)
 			}
 
 			return NewSomeValueNonCopying(invocation.Interpreter, caseValue)

runtime/interpreter/interpreter_expression.go

@@ -91,8 +91,9 @@ func (interpreter *Interpreter) indexExpressionGetterSetter(indexExpression *ast
 
 	elaboration := interpreter.Program.Elaboration
 
-	indexedType := elaboration.IndexExpressionIndexedTypes[indexExpression]
-	indexingType := elaboration.IndexExpressionIndexingTypes[indexExpression]
+	indexExpressionTypes := elaboration.IndexExpressionTypes[indexExpression]
+	indexedType := indexExpressionTypes.IndexedType
+	indexingType := indexExpressionTypes.IndexingType
 
 	transferredIndexingValue := interpreter.transferAndConvert(
 		interpreter.evalExpression(indexExpression.IndexingExpression),
@@ -428,8 +429,9 @@ func (interpreter *Interpreter) VisitBinaryExpression(expression *ast.BinaryExpr
 
 		value := rightValue()
 
-		rightType := interpreter.Program.Elaboration.BinaryExpressionRightTypes[expression]
-		resultType := interpreter.Program.Elaboration.BinaryExpressionResultTypes[expression]
+		binaryExpressionTypes := interpreter.Program.Elaboration.BinaryExpressionTypes[expression]
+		rightType := binaryExpressionTypes.RightType
+		resultType := binaryExpressionTypes.ResultType
 
 		// NOTE: important to convert both any and optional
 		return interpreter.ConvertAndBox(getLocationRange, value, rightType, resultType)
@@ -650,8 +652,9 @@ func (interpreter *Interpreter) VisitStringExpression(expression *ast.StringExpr
 func (interpreter *Interpreter) VisitArrayExpression(expression *ast.ArrayExpression) ast.Repr {
 	values := interpreter.visitExpressionsNonCopying(expression.Values)
 
-	argumentTypes := interpreter.Program.Elaboration.ArrayExpressionArgumentTypes[expression]
-	arrayType := interpreter.Program.Elaboration.ArrayExpressionArrayType[expression]
+	arrayExpressionTypes := interpreter.Program.Elaboration.ArrayExpressionTypes[expression]
+	argumentTypes := arrayExpressionTypes.ArgumentTypes
+	arrayType := arrayExpressionTypes.ArrayType
 	elementType := arrayType.ElementType(false)
 
 	copies := make([]Value, len(values))
@@ -679,8 +682,9 @@ func (interpreter *Interpreter) VisitArrayExpression(expression *ast.ArrayExpres
 func (interpreter *Interpreter) VisitDictionaryExpression(expression *ast.DictionaryExpression) ast.Repr {
 	values := interpreter.visitEntries(expression.Entries)
 
-	entryTypes := interpreter.Program.Elaboration.DictionaryExpressionEntryTypes[expression]
-	dictionaryType := interpreter.Program.Elaboration.DictionaryExpressionType[expression]
+	dictionaryExpressionTypes := interpreter.Program.Elaboration.DictionaryExpressionTypes[expression]
+	entryTypes := dictionaryExpressionTypes.EntryTypes
+	dictionaryType := dictionaryExpressionTypes.DictionaryType
 
 	var keyValuePairs []Value
 
@@ -806,9 +810,11 @@ func (interpreter *Interpreter) VisitInvocationExpression(invocationExpression *
 
 	elaboration := interpreter.Program.Elaboration
 
-	typeParameterTypes := elaboration.InvocationExpressionTypeArguments[invocationExpression]
-	argumentTypes := elaboration.InvocationExpressionArgumentTypes[invocationExpression]
-	parameterTypes := elaboration.InvocationExpressionParameterTypes[invocationExpression]
+	invocationExpressionTypes := elaboration.InvocationExpressionTypes[invocationExpression]
+
+	typeParameterTypes := invocationExpressionTypes.TypeArguments
+	argumentTypes := invocationExpressionTypes.ArgumentTypes
+	parameterTypes := invocationExpressionTypes.TypeParameterTypes
 
 	line := invocationExpression.StartPosition().Line