Add token limit

[turbolent]

Description

Limit the amount of tokens that may be emitted during a lex.

---

• Targeted PR against master branch
• Linked to Github issue with discussion and accepted design OR link to spec that describes this work
• Code follows the standards mentioned here
• Updated relevant documentation
• Re-reviewed Files changed in the Github PR explorer
• Added appropriate labels

[codecov]

Codecov Report

Merging #1668 (b8e71ce) into master (66eac2a) will decrease coverage by 0.01%.
The diff coverage is 66.66%.

@@            Coverage Diff             @@
##           master    #1668      +/-   ##
==========================================
- Coverage   76.50%   76.49%   -0.02%     
==========================================
  Files         291      291              
  Lines       60995    60982      -13     
==========================================
- Hits        46666    46648      -18     
- Misses      12719    12722       +3     
- Partials     1610     1612       +2     

Flag	Coverage Δ
unittests	76.49% <66.66%> (-0.02%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
runtime/parser2/lexer/lexer.go	97.97% <66.66%> (-0.65%)	⬇️
runtime/common/metering.go	91.30% <0.00%> (-0.55%)	⬇️
runtime/sema/checker.go	90.33% <0.00%> (-0.12%)	⬇️
runtime/sema/type.go	88.94% <0.00%> (-0.08%)	⬇️
values.go	65.82% <0.00%> (-0.08%)	⬇️
runtime/interpreter/statictype.go	81.88% <0.00%> (-0.05%)	⬇️
runtime/parser2/declaration.go	90.40% <0.00%> (-0.02%)	⬇️
runtime/interpreter/interpreter.go	89.51% <0.00%> (-0.01%)	⬇️
runtime/common/memorykind_string.go	40.00% <0.00%> (ø)
... and 5 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 66eac2a...b8e71ce. Read the comment docs.

[github-actions]

Cadence Benchstat comparison

This branch with compared with the base branch onflow:master commit 37aea92
The command for i in {1..N}; do go test ./... -run=XXX -bench=. -benchmem -shuffle=on; done was used.
Bench tests were run a total of 7 times on each branch.

Results

	old.txt	new.txt
	time/op		delta
CheckContractInterfaceFungibleTokenConformance-2	138µs ± 5%	135µs ± 2%	~	(p=0.073 n=7+7)
ContractInterfaceFungibleToken-2	40.8µs ± 3%	40.7µs ± 3%	~	(p=0.945 n=7+6)
InterpretRecursionFib-2	2.89ms ± 1%	2.95ms ± 6%	~	(p=0.234 n=6+7)
NewInterpreter/new_interpreter-2	1.14µs ± 7%	1.11µs ± 1%	~	(p=0.178 n=7+7)
NewInterpreter/new_sub-interpreter-2	2.33µs ± 8%	2.27µs ± 1%	~	(p=0.094 n=7+6)
ParseArray-2	8.41ms ± 1%	8.38ms ± 1%	~	(p=0.383 n=7+7)
ParseDeploy/byte_array-2	12.7ms ± 1%	12.6ms ± 1%	−1.00%	(p=0.035 n=7+6)
ParseDeploy/decode_hex-2	1.17ms ± 1%	1.17ms ± 1%	~	(p=1.000 n=6+7)
ParseFungibleToken/With_memory_metering-2	218µs ± 2%	209µs ± 1%	−4.14%	(p=0.001 n=7+6)
ParseFungibleToken/Without_memory_metering-2	162µs ± 1%	163µs ± 5%	~	(p=0.534 n=6+7)
ParseInfix-2	7.64µs ± 5%	7.53µs ± 1%	~	(p=0.534 n=7+6)
QualifiedIdentifierCreation/One_level-2	2.68ns ± 0%	2.68ns ± 0%	~	(p=0.645 n=7+6)
QualifiedIdentifierCreation/Three_levels-2	141ns ± 1%	141ns ± 1%	~	(p=0.765 n=6+7)
RuntimeFungibleTokenTransfer-2	1.18ms ±31%	1.23ms ±28%	~	(p=0.318 n=7+7)
RuntimeResourceDictionaryValues-2	6.77ms ± 1%	6.77ms ± 2%	~	(p=0.731 n=6+7)
Transfer-2	87.8ns ± 1%	88.4ns ± 1%	~	(p=0.138 n=7+6)
	alloc/op		delta
CheckContractInterfaceFungibleTokenConformance-2	66.3kB ± 0%	66.3kB ± 0%	~	(p=0.245 n=6+7)
ContractInterfaceFungibleToken-2	26.7kB ± 0%	26.7kB ± 0%	~	(p=1.000 n=7+7)
InterpretRecursionFib-2	1.25MB ± 0%	1.25MB ± 0%	~	(p=1.000 n=7+7)
NewInterpreter/new_interpreter-2	872B ± 0%	872B ± 0%	~	(all equal)
NewInterpreter/new_sub-interpreter-2	1.38kB ± 0%	1.38kB ± 0%	~	(all equal)
ParseArray-2	2.96MB ± 2%	2.94MB ± 2%	~	(p=0.318 n=7+7)
ParseDeploy/byte_array-2	4.33MB ± 3%	4.25MB ± 3%	~	(p=0.221 n=7+7)
ParseDeploy/decode_hex-2	213kB ± 0%	213kB ± 0%	~	(p=0.551 n=7+7)
ParseFungibleToken/With_memory_metering-2	36.3kB ± 0%	36.3kB ± 0%	~	(p=0.409 n=7+7)
ParseFungibleToken/Without_memory_metering-2	36.3kB ± 0%	36.3kB ± 0%	~	(p=1.000 n=7+7)
ParseInfix-2	2.14kB ± 0%	2.14kB ± 0%	~	(p=0.694 n=7+7)
QualifiedIdentifierCreation/One_level-2	0.00B	0.00B	~	(all equal)
QualifiedIdentifierCreation/Three_levels-2	64.0B ± 0%	64.0B ± 0%	~	(all equal)
RuntimeFungibleTokenTransfer-2	237kB ± 1%	237kB ± 0%	~	(p=0.874 n=7+7)
RuntimeResourceDictionaryValues-2	2.24MB ± 0%	2.24MB ± 0%	~	(p=1.000 n=7+5)
Transfer-2	48.0B ± 0%	48.0B ± 0%	~	(all equal)
	allocs/op		delta
CheckContractInterfaceFungibleTokenConformance-2	1.07k ± 0%	1.07k ± 0%	~	(all equal)
ContractInterfaceFungibleToken-2	460 ± 0%	460 ± 0%	~	(all equal)
InterpretRecursionFib-2	23.8k ± 0%	23.8k ± 0%	~	(all equal)
NewInterpreter/new_interpreter-2	12.0 ± 0%	12.0 ± 0%	~	(all equal)
NewInterpreter/new_sub-interpreter-2	40.0 ± 0%	40.0 ± 0%	~	(all equal)
ParseArray-2	70.0k ± 0%	70.0k ± 0%	~	(p=0.592 n=7+7)
ParseDeploy/byte_array-2	105k ± 0%	105k ± 0%	~	(p=0.592 n=7+7)
ParseDeploy/decode_hex-2	79.0 ± 0%	79.0 ± 0%	~	(all equal)
ParseFungibleToken/With_memory_metering-2	1.06k ± 0%	1.06k ± 0%	~	(all equal)
ParseFungibleToken/Without_memory_metering-2	1.06k ± 0%	1.06k ± 0%	~	(all equal)
ParseInfix-2	66.0 ± 0%	66.0 ± 0%	~	(all equal)
QualifiedIdentifierCreation/One_level-2	0.00	0.00	~	(all equal)
QualifiedIdentifierCreation/Three_levels-2	2.00 ± 0%	2.00 ± 0%	~	(all equal)
RuntimeFungibleTokenTransfer-2	4.58k ± 0%	4.58k ± 0%	~	(p=0.927 n=7+7)
RuntimeResourceDictionaryValues-2	37.6k ± 0%	37.6k ± 0%	~	(p=0.567 n=7+7)
Transfer-2	1.00 ± 0%	1.00 ± 0%	~	(all equal)