feat(gnovm): handle loop variables (#2429)

# Problem Definition

The problem originates from the issue described in
[#1135](#1135). While the full
scope of the issue is broader, it fundamentally relates to the concept
of loop variable escapes block where it's defined.

e.g. 1:
```go
package main

import "fmt"

var s1 []*int

func forLoopRef() {
	defer func() {
		for i, e := range s1 {
			fmt.Printf("s1[%d] is: %d\n", i, *e)
		}
	}()

	for i := 0; i < 3; i++ {
		z := i + 1
		s1 = append(s1, &z)
	}
}

func main() {
	forLoopRef()
}
```

e.g. 2:
```go
package main

type f func()

var fs []f

func forLoopClosure() {
	defer func() {
		for _, f := range fs {
			f()
		}
	}()

	for i := 0; i < 3; i++ {
		z := i
		fs = append(fs, func() { println(z) })
	}
}

func main() {
	forLoopClosure()
}
```

e.g. 3:
```go
package main

func main() {
	c := 0
	closures := []func(){}
loop:
	i := c
	closures = append(closures, func() {
		println(i)
	})
	c += 1
	if c < 10 {
		goto loop
	}

	for _, cl := range closures {
		cl()
	}
}
```



# Solution ideas

- **identify escaped vars in preprocess**:
Detect situations where a loop variable is defined within a loop
block(including `for/range` loops or loops constructed using `goto`
statements), and escapes the block where it's defined.

- **runtime allocation**:
Allocate a new heap item for the loop variable in each iteration to
ensure each iteration operates with its unique variable instance.

- **NOTE1**: this is consistent with Go's Strategy:
"Each iteration has its own separate declared variable (or variables)
[Go 1.22]. The variable used by the first iteration is declared by the
init statement. The variable used by each subsequent iteration is
declared implicitly before executing the post statement and initialized
to the value of the previous iteration's variable at that moment."

- **NOTE2**: the `loopvar` feature of Go 1.22 is not supported in this
version, and will be supported in next version.

    not supporting capture `i` defined in for/range clause;
```go
	for i := 0; i < 3; i++ {
		s1 = append(s1, &i)
	}
```

# Implementation Details

**Preprocess Stage(Multi-Phase Preprocessor)**:

- **Phase 1: `initStaticBlocks`**: Establish a cascading scope structure
where `predefine` is conducted. In this phase Name expressions are
initially marked as `NameExprTypeDefine`, which may later be upgraded to
`NameExprTypeHeapDefine` if it is determined that they escape the loop
block. This phase also supports other processes as a
prerequisite[#2077](#2077).
   
- **Phase 2: `preprocess1`**: This represents the original preprocessing
phase(not going into details).
   
- **Phase 3: `findGotoLoopDefines`**: By traversing the AST, any name
expression defined in a loop block (for/range, goto) with the attribute
`NameExprTypeDefine` is promoted to `NameExprTypeHeapDefine`. This is
used in later phase.
   
- **Phase 4: `findLoopUses1`**: Identify the usage of
`NameExprTypeHeapDefine` name expressions. If a name expression is used
in a function literal or is referrnced(e.g. &a), and it was previously
defined as `NameExprTypeHeapDefine`, the `used` name expression is then
given the attribute `NameExprTypeHeapUse`. This step finalizes whether a
name expression will be allocated on the heap and used from heap.
`Closures` represent a particular scenario in this context. Each
closure, defined by a funcLitExpr that captures variables, is associated
with a HeapCaptures list. This list consists of NameExprs, which are
utilized at runtime to obtain the actual variable values for each
iteration. Correspondingly, within the funcLitExpr block, a list of
placeholder values are defined. These placeholders are populated during
the doOpFuncLit phase and subsequently utilized in the `doOpCall` to
ensure that each iteration uses the correct data.


- **Phase 5: `findLoopUses2`**: Convert non-loop uses of loop-defined
names to `NameExprTypeHeapUse`. Also, demote `NameExprTypeHeapDefine`
back to `NameExprTypeDefine` if no actual usage is found. Also , as the
last phase, attributes no longer needed will be cleaned up after this
phase.

**Runtime Stage**:

1. **Variable Allocation**:
- Modify the runtime so that encountering a `NameExprTypeHeapDefine`
triggers the allocation of a new `heapItemValue` for it, which will be
used by any `NameExprTypeHeapUse`.

2. **Function Literal Handling**:
- During the execution of `doOpFuncLit`, retrieve the `HeapCapture`
values (previously allocated heap item values) and fill in the
placeholder values within the `funcLitExpr` block.
- When invoking the function (`doOpCall`), the `placeHolder`
values(fv.Captures) are used to update the execution context, ensuring
accurate and consistent results across iterations.

---------

Co-authored-by: ltzMaxwell <[email protected]>
Co-authored-by: Morgan <[email protected]>

gnovm/cmd/gno/testdata/gno_test/test_with-native-fallback.txtar

@@ -4,7 +4,7 @@
 
 ! stdout .+
 stderr 'panic: unknown import path net \[recovered\]'
-stderr '	panic: gno.land/r/\w{8}/contract.gno:3:1: unknown import path net'
+stderr '	panic: gno.land/r/\w{8}/contract.gno:3:8: unknown import path net'
 
 gno test -v --with-native-fallback .
 

gnovm/cmd/gno/testdata/gno_test/unknow_lib.txtar

@@ -4,13 +4,13 @@
 
 ! stdout .+
 stderr 'panic: unknown import path foobarbaz \[recovered\]'
-stderr '	panic: gno.land/r/\w{8}/contract.gno:3:1: unknown import path foobarbaz'
+stderr '	panic: gno.land/r/\w{8}/contract.gno:3:8: unknown import path foobarbaz'
 
 ! gno test -v --with-native-fallback .
 
 ! stdout .+
 stderr 'panic: unknown import path foobarbaz \[recovered\]'
-stderr '	panic: gno.land/r/\w{8}/contract.gno:3:1: unknown import path foobarbaz'
+stderr '	panic: gno.land/r/\w{8}/contract.gno:3:8: unknown import path foobarbaz'
 
 -- contract.gno --
 package contract

gnovm/pkg/gnolang/debugger.go

@@ -257,17 +257,21 @@ func debugUpdateLocation(m *Machine) {
 	for i := nx - 1; i >= 0; i-- {
 		expr := m.Exprs[i]
 		if l := expr.GetLine(); l > 0 {
-			m.Debugger.loc.Line = l
-			m.Debugger.loc.Column = expr.GetColumn()
+			if col := expr.GetColumn(); col > 0 {
+				m.Debugger.loc.Line = l
+				m.Debugger.loc.Column = expr.GetColumn()
+			}
 			return
 		}
 	}
 
 	if len(m.Stmts) > 0 {
 		if stmt := m.PeekStmt1(); stmt != nil {
 			if l := stmt.GetLine(); l > 0 {
-				m.Debugger.loc.Line = l
-				m.Debugger.loc.Column = stmt.GetColumn()
+				if col := stmt.GetColumn(); col > 0 {
+					m.Debugger.loc.Line = l
+					m.Debugger.loc.Column = stmt.GetColumn()
+				}
 				return
 			}
 		}
@@ -648,7 +652,7 @@ func debugEvalExpr(m *Machine, node ast.Node) (tv TypedValue, err error) {
 			return tv, fmt.Errorf("invalid selector: %s", n.Sel.Name)
 		}
 		for _, vp := range tr {
-			x = x.GetPointerTo(m.Alloc, m.Store, vp).Deref()
+			x = x.GetPointerToFromTV(m.Alloc, m.Store, vp).Deref()
 		}
 		return x, nil
 	case *ast.IndexExpr:

gnovm/pkg/gnolang/debugger_test.go

@@ -131,7 +131,7 @@ func TestDebug(t *testing.T) {
 		{in: "p \"xxxx\"\n", out: `("xxxx" string)`},
 		{in: "si\n", out: "sample.gno:14"},
 		{in: "s\ns\n", out: `=>   14: var global = "test"`},
-		{in: "s\n\n", out: "=>   33: 	num := 5"},
+		{in: "s\n\n\n", out: "=>   33: 	num := 5"},
 		{in: "foo", out: "command not available: foo"},
 		{in: "\n\n", out: "dbg> "},
 		{in: "#\n", out: "dbg> "},

gnovm/pkg/gnolang/go2gno.go

@@ -754,11 +754,13 @@ func toDecls(fs *token.FileSet, gd *ast.GenDecl) (ds Decls) {
 			name := toName(s.Name)
 			tipe := toExpr(fs, s.Type)
 			alias := s.Assign != 0
-			ds = append(ds, &TypeDecl{
+			td := &TypeDecl{
 				NameExpr: NameExpr{Name: name},
 				Type:     tipe,
 				IsAlias:  alias,
-			})
+			}
+			setLoc(fs, s.Pos(), td)
+			ds = append(ds, td)
 		case *ast.ValueSpec:
 			if gd.Tok == token.CONST {
 				var names []NameExpr

gnovm/pkg/gnolang/machine.go

@@ -699,7 +699,7 @@ func (m *Machine) runFileDecls(fns ...*FileNode) []TypedValue {
 				}
 			}
 			// if dep already in loopfindr, abort.
-			if hasName(dep, loopfindr) {
+			if slices.Contains(loopfindr, dep) {
 				if _, ok := (*depdecl).(*FuncDecl); ok {
 					// recursive function dependencies
 					// are OK with func decls.
@@ -2112,15 +2112,25 @@ func (m *Machine) PushForPointer(lx Expr) {
 func (m *Machine) PopAsPointer(lx Expr) PointerValue {
 	switch lx := lx.(type) {
 	case *NameExpr:
-		lb := m.LastBlock()
-		return lb.GetPointerTo(m.Store, lx.Path)
+		switch lx.Type {
+		case NameExprTypeNormal:
+			lb := m.LastBlock()
+			return lb.GetPointerTo(m.Store, lx.Path)
+		case NameExprTypeHeapUse:
+			lb := m.LastBlock()
+			return lb.GetPointerToHeapUse(m.Store, lx.Path)
+		case NameExprTypeHeapClosure:
+			panic("should not happen")
+		default:
+			panic("unexpected NameExpr in PopAsPointer")
+		}
 	case *IndexExpr:
 		iv := m.PopValue()
 		xv := m.PopValue()
 		return xv.GetPointerAtIndex(m.Alloc, m.Store, iv)
 	case *SelectorExpr:
 		xv := m.PopValue()
-		return xv.GetPointerTo(m.Alloc, m.Store, lx.Path)
+		return xv.GetPointerToFromTV(m.Alloc, m.Store, lx.Path)
 	case *StarExpr:
 		ptr := m.PopValue().V.(PointerValue)
 		return ptr
@@ -2348,15 +2358,3 @@ func (m *Machine) ExceptionsStacktrace() string {
 
 	return builder.String()
 }
-
-//----------------------------------------
-// utility
-
-func hasName(n Name, ns []Name) bool {
-	for _, n2 := range ns {
-		if n == n2 {
-			return true
-		}
-	}
-	return false
-}

gnovm/pkg/gnolang/nodes.go

@@ -146,11 +146,26 @@ func (loc Location) IsZero() bool {
 // even after preprocessing.  Temporary attributes (e.g. those
 // for preprocessing) are stored in .data.
 
+type GnoAttribute string
+
+const (
+	ATTR_PREPROCESSED  GnoAttribute = "ATTR_PREPROCESSED"
+	ATTR_PREDEFINED    GnoAttribute = "ATTR_PREDEFINED"
+	ATTR_TYPE_VALUE    GnoAttribute = "ATTR_TYPE_VALUE"
+	ATTR_TYPEOF_VALUE  GnoAttribute = "ATTR_TYPEOF_VALUE"
+	ATTR_IOTA          GnoAttribute = "ATTR_IOTA"
+	ATTR_LOCATIONED    GnoAttribute = "ATTR_LOCATIONE"     // XXX DELETE
+	ATTR_GOTOLOOP_STMT GnoAttribute = "ATTR_GOTOLOOP_STMT" // XXX delete?
+	ATTR_LOOP_DEFINES  GnoAttribute = "ATTR_LOOP_DEFINES"  // []Name defined within loops.
+	ATTR_LOOP_USES     GnoAttribute = "ATTR_LOOP_USES"     // []Name loop defines actually used.
+	ATTR_SHIFT_RHS     GnoAttribute = "ATTR_SHIFT_RHS"
+)
+
 type Attributes struct {
 	Line   int
 	Column int
 	Label  Name
-	data   map[interface{}]interface{} // not persisted
+	data   map[GnoAttribute]interface{} // not persisted
 }
 
 func (attr *Attributes) GetLine() int {
@@ -177,22 +192,31 @@ func (attr *Attributes) SetLabel(label Name) {
 	attr.Label = label
 }
 
-func (attr *Attributes) HasAttribute(key interface{}) bool {
+func (attr *Attributes) HasAttribute(key GnoAttribute) bool {
 	_, ok := attr.data[key]
 	return ok
 }
 
-func (attr *Attributes) GetAttribute(key interface{}) interface{} {
+// GnoAttribute must not be user provided / arbitrary,
+// otherwise will create potential exploits.
+func (attr *Attributes) GetAttribute(key GnoAttribute) interface{} {
 	return attr.data[key]
 }
 
-func (attr *Attributes) SetAttribute(key interface{}, value interface{}) {
+func (attr *Attributes) SetAttribute(key GnoAttribute, value interface{}) {
 	if attr.data == nil {
-		attr.data = make(map[interface{}]interface{})
+		attr.data = make(map[GnoAttribute]interface{})
 	}
 	attr.data[key] = value
 }
 
+func (attr *Attributes) DelAttribute(key GnoAttribute) {
+	if debug && attr.data == nil {
+		panic("should not happen, attribute is expected to be non-empty.")
+	}
+	delete(attr.data, key)
+}
+
 // ----------------------------------------
 // Node
 
@@ -206,9 +230,10 @@ type Node interface {
 	SetColumn(int)
 	GetLabel() Name
 	SetLabel(Name)
-	HasAttribute(key interface{}) bool
-	GetAttribute(key interface{}) interface{}
-	SetAttribute(key interface{}, value interface{})
+	HasAttribute(key GnoAttribute) bool
+	GetAttribute(key GnoAttribute) interface{}
+	SetAttribute(key GnoAttribute, value interface{})
+	DelAttribute(key GnoAttribute)
 }
 
 // non-pointer receiver to help make immutable.
@@ -368,11 +393,22 @@ var (
 	_ Expr = &ConstExpr{}
 )
 
+type NameExprType int
+
+const (
+	NameExprTypeNormal      NameExprType = iota // default
+	NameExprTypeDefine                          // when defining normally
+	NameExprTypeHeapDefine                      // when defining escaped name in loop
+	NameExprTypeHeapUse                         // when above used in non-define lhs/rhs
+	NameExprTypeHeapClosure                     // when closure captures name
+)
+
 type NameExpr struct {
 	Attributes
 	// TODO rename .Path's to .ValuePaths.
 	Path ValuePath // set by preprocessor.
 	Name
+	Type NameExprType
 }
 
 type NameExprs []NameExpr
@@ -499,8 +535,9 @@ type KeyValueExprs []KeyValueExpr
 type FuncLitExpr struct {
 	Attributes
 	StaticBlock
-	Type FuncTypeExpr // function type
-	Body              // function body
+	Type         FuncTypeExpr // function type
+	Body                      // function body
+	HeapCaptures NameExprs    // filled in findLoopUses1
 }
 
 // The preprocessor replaces const expressions
@@ -581,11 +618,15 @@ func (ftxz FieldTypeExprs) IsNamed() bool {
 	named := false
 	for i, ftx := range ftxz {
 		if i == 0 {
-			named = ftx.Name != ""
+			if ftx.Name == "" || isHiddenResultVariable(string(ftx.Name)) {
+				named = false
+			} else {
+				named = true
+			}
 		} else {
 			if named && ftx.Name == "" {
 				panic("[]FieldTypeExpr has inconsistent namedness (starts named)")
-			} else if !named && ftx.Name != "" {
+			} else if !named && (ftx.Name != "" || !isHiddenResultVariable(string(ftx.Name))) {
 				panic("[]FieldTypeExpr has inconsistent namedness (starts unnamed)")
 			}
 		}
@@ -1489,6 +1530,7 @@ type BlockNode interface {
 	GetNumNames() uint16
 	GetParentNode(Store) BlockNode
 	GetPathForName(Store, Name) ValuePath
+	GetBlockNodeForPath(Store, ValuePath) BlockNode
 	GetIsConst(Store, Name) bool
 	GetLocalIndex(Name) (uint16, bool)
 	GetValueRef(Store, Name, bool) *TypedValue
@@ -1588,6 +1630,8 @@ func (sb *StaticBlock) GetBlockNames() (ns []Name) {
 }
 
 // Implements BlockNode.
+// NOTE: Extern names may also be local, if declared after usage as an extern
+// (thus shadowing the extern name).
 func (sb *StaticBlock) GetExternNames() (ns []Name) {
 	return sb.Externs // copy?
 }
@@ -1629,6 +1673,9 @@ func (sb *StaticBlock) GetPathForName(store Store, n Name) ValuePath {
 	}
 	// Register as extern.
 	// NOTE: uverse names are externs too.
+	// NOTE: externs may also be shadowed later in the block. Thus, usages
+	// before the declaration will have depth > 1; following it, depth == 1,
+	// matching the two different identifiers they refer to.
 	if !isFile(sb.GetSource(store)) {
 		sb.GetStaticBlock().addExternName(n)
 	}
@@ -1657,6 +1704,21 @@ func (sb *StaticBlock) GetPathForName(store Store, n Name) ValuePath {
 	panic(fmt.Sprintf("name %s not declared", n))
 }
 
+// Get the containing block node for node with path relative to this containing block.
+func (sb *StaticBlock) GetBlockNodeForPath(store Store, path ValuePath) BlockNode {
+	if path.Type != VPBlock {
+		panic("expected block type value path but got " + path.Type.String())
+	}
+
+	// NOTE: path.Depth == 1 means it's in bn.
+	bn := sb.GetSource(store)
+	for i := 1; i < int(path.Depth); i++ {
+		bn = bn.GetParentNode(store)
+	}
+
+	return bn
+}
+
 // Returns whether a name defined here in in ancestry is a const.
 // This is not the same as whether a name's static type is
 // untyped -- as in c := a == b, a name may be an untyped non-const.
@@ -1713,21 +1775,12 @@ func (sb *StaticBlock) GetStaticTypeOf(store Store, n Name) Type {
 // Implements BlockNode.
 func (sb *StaticBlock) GetStaticTypeOfAt(store Store, path ValuePath) Type {
 	if debug {
-		if path.Type != VPBlock {
-			panic("should not happen")
-		}
 		if path.Depth == 0 {
 			panic("should not happen")
 		}
 	}
-	for {
-		if path.Depth == 1 {
-			return sb.Types[path.Index]
-		} else {
-			sb = sb.GetParentNode(store).GetStaticBlock()
-			path.Depth -= 1
-		}
-	}
+	bn := sb.GetBlockNodeForPath(store, path)
+	return bn.GetStaticBlock().Types[path.Index]
 }
 
 // Implements BlockNode.
@@ -2115,18 +2168,6 @@ func (x *BasicLitExpr) GetInt() int {
 	return i
 }
 
-type GnoAttribute string
-
-const (
-	ATTR_PREPROCESSED GnoAttribute = "ATTR_PREPROCESSED"
-	ATTR_PREDEFINED   GnoAttribute = "ATTR_PREDEFINED"
-	ATTR_TYPE_VALUE   GnoAttribute = "ATTR_TYPE_VALUE"
-	ATTR_TYPEOF_VALUE GnoAttribute = "ATTR_TYPEOF_VALUE"
-	ATTR_IOTA         GnoAttribute = "ATTR_IOTA"
-	ATTR_LOCATIONED   GnoAttribute = "ATTR_LOCATIONED"
-	ATTR_SHIFT_RHS    GnoAttribute = "ATTR_SHIFT_RHS"
-)
-
 var rePkgName = regexp.MustCompile(`^[a-z][a-z0-9_]+$`)
 
 // TODO: consider length restrictions.
@@ -2136,3 +2177,12 @@ func validatePkgName(name string) {
 		panic(fmt.Sprintf("cannot create package with invalid name %q", name))
 	}
 }
+
+const hiddenResultVariable = ".res_"
+
+func isHiddenResultVariable(name string) bool {
+	if strings.HasPrefix(name, hiddenResultVariable) {
+		return true
+	}
+	return false
+}