Refactor configs #154

m0ar · 2019-07-19T14:07:36Z

This is still a bit unpolished, but I think it's in good enough order for someone to take to the finish line while I'm away 🏁 Sorry for leaving stuff unfinished, but the alternative was chillin' the last two days instead 😅

Does away with all TransformerConfig structs, the shiny new GetEventTransformerConfig polls all values automagically instead 💯
Does away with configuration "getter constants", except ABI's since I haven't been able to figure out how we can handle the signatures : (
Removes the checked.go set of constants, and renamed the labels.go entries to use snake_case we can utilise everywhere instead

One step closer to automagic generation of transformers from config 🙃

m0ar · 2019-07-19T14:14:54Z

db/schema.sql

+    vat_fork integer DEFAULT 0 NOT NULL,
+    jug_init integer DEFAULT 0 NOT NULL,
+    yank integer DEFAULT 0 NOT NULL,
+    flip_tick integer DEFAULT 0 NOT NULL
 );


Dropping the _checked ending allows us to use the transformer labels as columns here.

Heads up that this requires a small change in vDB core, where it assumes all of these has _checked appended. This is probably why the integration test suite crashes, probably on the plugin test.

Alternatively we can append _checked to the strings sent from mcd_transformers into MarkHeaderChecked et al, but I don't love the _checked bit anyway. Your choice!

m0ar · 2019-07-19T15:00:06Z

environments/mcdTransformers.toml

+    [contract.COL1_FLIP]
+        address = "0x911c1bb935ce7fe5acb8069acd52d50d3c896353"
+        abi      = '[{"constant":false,"inputs":[{"name":"id","type":"uint256"}],"name":"yank","outputs":[],"payable":false,"stateMutability":"nonpayable","type":"function"},{"constant":false,"inputs":[{"name":"what","type":"bytes32"},{"name":"data","type":"uint256"}],"name":"file","outputs":[],"payable":false,"stateMutability":"nonpayable","type":"function"},{"constant":false,"inputs":[{"name":"usr","type":"address"},{"name":"gal","type":"address"},{"name":"tab","type":"uint256"},{"name":"lot","type":"uint256"},{"name":"bid","type":"uint256"}],"name":"kick","outputs":[{"name":"id","type":"uint256"}],"payable":false,"stateMutability":"nonpayable","type":"function"},{"constant":true,"inputs":[],"name":"vat","outputs":[{"name":"","type":"address"}],"payable":false,"stateMutability":"view","type":"function"},{"constant":true,"inputs":[{"name":"","type":"uint256"}],"name":"bids","outputs":[{"name":"bid","type":"uint256"},{"name":"lot","type":"uint256"},{"name":"guy","type":"address"},{"name":"tic","type":"uint48"},{"name":"end","type":"uint48"},{"name":"usr","type":"address"},{"name":"gal","type":"address"},{"name":"tab","type":"uint256"}],"payable":false,"stateMutability":"view","type":"function"},{"constant":false,"inputs":[{"name":"id","type":"uint256"},{"name":"lot","type":"uint256"},{"name":"bid","type":"uint256"}],"name":"tend","outputs":[],"payable":false,"stateMutability":"nonpayable","type":"function"},{"constant":true,"inputs":[],"name":"ttl","outputs":[{"name":"","type":"uint48"}],"payable":false,"stateMutability":"view","type":"function"},{"constant":false,"inputs":[{"name":"id","type":"uint256"},{"name":"lot","type":"uint256"},{"name":"bid","type":"uint256"}],"name":"dent","outputs":[],"payable":false,"stateMutability":"nonpayable","type":"function"},{"constant":false,"inputs":[{"name":"usr","type":"address"}],"name":"rely","outputs":[],"payable":false,"stateMutability":"nonpayable","type":"function"},{"constant":true,"inputs":[],"name":"beg","outputs":[{"name":"","type":"uint256"}],"payable":false,"stateMutability":"view","type":"function"},{"constant":false,"inputs":[{"name":"usr","type":"address"}],"name":"deny","outputs":[],"payable":false,"stateMutability":"nonpayable","type":"function"},{"constant":true,"inputs":[{"name":"","type":"address"}],"name":"wards","outputs":[{"name":"","type":"uint256"}],"payable":false,"stateMutability":"view","type":"function"},{"constant":true,"inputs":[],"name":"ilk","outputs":[{"name":"","type":"bytes32"}],"payable":false,"stateMutability":"view","type":"function"},{"constant":false,"inputs":[{"name":"id","type":"uint256"}],"name":"deal","outputs":[],"payable":false,"stateMutability":"nonpayable","type":"function"},{"constant":true,"inputs":[],"name":"tau","outputs":[{"name":"","type":"uint48"}],"payable":false,"stateMutability":"view","type":"function"},{"constant":true,"inputs":[],"name":"kicks","outputs":[{"name":"","type":"uint256"}],"payable":false,"stateMutability":"view","type":"function"},{"constant":false,"inputs":[{"name":"id","type":"uint256"}],"name":"tick","outputs":[],"payable":false,"stateMutability":"nonpayable","type":"function"},{"inputs":[{"name":"vat_","type":"address"},{"name":"ilk_","type":"bytes32"}],"payable":false,"stateMutability":"nonpayable","type":"constructor"},{"anonymous":false,"inputs":[{"indexed":false,"name":"id","type":"uint256"},{"indexed":false,"name":"lot","type":"uint256"},{"indexed":false,"name":"bid","type":"uint256"},{"indexed":false,"name":"tab","type":"uint256"},{"indexed":true,"name":"usr","type":"address"},{"indexed":true,"name":"gal","type":"address"}],"name":"Kick","type":"event"},{"anonymous":true,"inputs":[{"indexed":true,"name":"sig","type":"bytes4"},{"indexed":true,"name":"usr","type":"address"},{"indexed":true,"name":"arg1","type":"bytes32"},{"indexed":true,"name":"arg2","type":"bytes32"},{"indexed":false,"name":"data","type":"bytes"}],"name":"LogNote","type":"event"}]'
+        deployed = 11579891 # deployment block of ETH_FLIP_A contract-- on update, ensure that it's still the first one deployed


This file is just composeAndExecute with a bit different structure (and a better name 😁 ).

The structural changes makes lookup of contract metadata a bit more self-explanatory, is in line with the formatting in the exporter section, and it'll be easier to see which contracts changes refer to.

Oh, and all these FLIPs have the same deployment block, but GetMinDeploymentBlock (this dude) will take the lowest one.

It's a necessity that all entries have the abi field, even though it's identical for the FLIP family. This way we can blindly lookup ABIs for the contracts referenced in the exporter section:

[exporter.tend] path = "transformers/events/tend/initializer" type = "eth_event" repository = "github.com/vulcanize/mcd_transformers" migrations = "db/migrations" contracts = [ "ETH_FLIP_A", "ETH_FLIP_B", "ETH_FLIP_C", "COL1_FLIP", "COL2_FLIP", "COL3_FLIP", "COL4_FLIP", "COL5_FLIP", ] rank = "0"

The code eating this contract cross-reference field and looking up the ABIs is here, and actually checks that they are identical when used in the same transformer 💪

m0ar · 2019-07-19T15:12:55Z

transformers/events/bite/config.go

-		StartingBlockNumber: constants.CatDeploymentBlock(),
-		EndingBlockNumber:   -1,
-	}
-}


All of these dudes disappear since we can generate them freely given just the transformer label (vow_fess) and the corresponding signature 💯

func GetEventTransformerConfig(transformerLabel, signature string) transformer.EventTransformerConfig { contractNames := constants.GetTransformerContractNames(transformerLabel) return transformer.EventTransformerConfig{ TransformerName: transformerLabel, ContractAddresses: constants.GetContractAddresses(contractNames), ContractAbi: constants.GetContractsABI(contractNames), Topic: signature, StartingBlockNumber: constants.GetMinDeploymentBlock(contractNames), EndingBlockNumber: -1, } }

m0ar · 2019-07-19T15:34:43Z

transformers/shared/constants/external.go

-func Col4FlipContractAddress() string   { return getEnvironmentString("contract.address.COL4_FLIP") }
-func Col5FlipContractAddress() string   { return getEnvironmentString("contract.address.COL5_FLIP") }
-func FlipperContractAddresses() []string {
-	return []string{EthFlipContractAddressA(), EthFlipContractAddressB(), EthFlipContractAddressC(), Col1FlipContractAddress(), Col2FlipContractAddress(), Col3FlipContractAddress(), Col4FlipContractAddress(), Col5FlipContractAddress()}


m0ar · 2019-07-19T15:35:34Z

transformers/shared/constants/external.go

+			log.WithField("contracts", contractNames).Fatalf("ABIs not consistent between contracts")
+		}
+	}
+	return abi


When a transformer has several contracts listed, it checks that they all have the same ABI

transformers/integration_tests/bite.go

transformers/integration_tests/deal.go

transformers/integration_tests/plugin.go

rmulhol

LGTM! The only things that I think are blockers are (1) passing the correct SpotPokeLabel to MarkHeaderChecked on the SpotPokeRepository (rather than BiteLabel 🙈) and (2) passing the contract address to the mappings + repository in the flap storage initializer

transformers/events/spot_poke/repository_test.go

transformers/events/spot_poke/repository.go

transformers/integration_tests/bite.go

transformers/integration_tests/cat_file.go

transformers/integration_tests/deal.go

transformers/storage/flap/initializer/initializer.go

transformers/storage/flap/storage_keys_lookup_test.go

transformers/test_data/tend.go

transformers/test_data/config_values.go

transformers/test_data/vat_file.go

elizabethengelman

Looking good! Just a few small comments/questions.

transformers/events/bite/initializer/initializer.go

transformers/events/cat_file/flip/initializer/initializer.go

transformers/integration_tests/bite.go

transformers/integration_tests/cat_file.go

transformers/shared/config.go

transformers/storage/flap/initializer/initializer.go

transformers/test_data/flap_kick.go

Note this breaks one integration test because vDB core is not updated

…nitializers after rebase

m0ar requested review from yaoandrew, elizabethengelman, rmulhol, aaizuss and Gslaughl July 19, 2019 14:07

m0ar commented Jul 19, 2019

View reviewed changes

yaoandrew mentioned this pull request Jul 29, 2019

Removes _checked from column name query vulcanize/vulcanizedb#124

Merged

yaoandrew force-pushed the refactor-configs branch 5 times, most recently from 2f2129f to 6bb3596 Compare July 31, 2019 05:29

aaizuss reviewed Jul 31, 2019

View reviewed changes

transformers/integration_tests/bite.go Outdated Show resolved Hide resolved

aaizuss reviewed Jul 31, 2019

View reviewed changes

transformers/integration_tests/deal.go Outdated Show resolved Hide resolved

aaizuss reviewed Jul 31, 2019

View reviewed changes

transformers/integration_tests/plugin.go Outdated Show resolved Hide resolved

yaoandrew force-pushed the refactor-configs branch 2 times, most recently from 248183f to fcdc246 Compare July 31, 2019 20:20

rmulhol reviewed Jul 31, 2019

View reviewed changes

elizabethengelman approved these changes Jul 31, 2019

View reviewed changes

yaoandrew force-pushed the refactor-configs branch from fcdc246 to 8f82aab Compare July 31, 2019 23:35

aaizuss approved these changes Aug 1, 2019

View reviewed changes

yaoandrew force-pushed the refactor-configs branch from 8f82aab to 5d8f287 Compare August 1, 2019 18:04

m0ar added 4 commits August 1, 2019 11:21

Remove _checked constants, use consistens transformer names

b56834a

Note this breaks one integration test because vDB core is not updated

Add general config getter, remove config constants (WIP)

6479f65

Move event transformers to new config generator

54b4bd8

Update tests for new config structure (phew)

3b0e1f3

m0ar and others added 6 commits August 1, 2019 11:21

remove old config format

cb44646

Updates to vdb 0.0.4 and fixes config files

c00c15a

Updates address, starting block numbers in testing config and fixes i…

dd34822

…nitializers after rebase

Updates storage initializers to use new config lookup

d28f1e0

Fixes Flip addresses in toml and initializers

8f48b3d

Addresses PR feedback - import order, address lookup in tests, etc

bd2362e

yaoandrew force-pushed the refactor-configs branch from 5d8f287 to bd2362e Compare August 1, 2019 18:33

yaoandrew merged commit c7fe3b1 into staging Aug 1, 2019

yaoandrew deleted the refactor-configs branch August 1, 2019 19:10

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Refactor configs #154

Refactor configs #154

m0ar commented Jul 19, 2019

m0ar Jul 19, 2019 •

edited

Loading

m0ar Jul 19, 2019

m0ar Jul 19, 2019

m0ar Jul 19, 2019

m0ar Jul 19, 2019

m0ar Jul 19, 2019

m0ar Jul 19, 2019

rmulhol left a comment •

edited

Loading

elizabethengelman left a comment

Refactor configs #154

Refactor configs #154

Conversation

m0ar commented Jul 19, 2019

m0ar Jul 19, 2019 • edited Loading

Choose a reason for hiding this comment

m0ar Jul 19, 2019

Choose a reason for hiding this comment

m0ar Jul 19, 2019

Choose a reason for hiding this comment

m0ar Jul 19, 2019

Choose a reason for hiding this comment

m0ar Jul 19, 2019

Choose a reason for hiding this comment

m0ar Jul 19, 2019

Choose a reason for hiding this comment

m0ar Jul 19, 2019

Choose a reason for hiding this comment

rmulhol left a comment • edited Loading

Choose a reason for hiding this comment

elizabethengelman left a comment

Choose a reason for hiding this comment

m0ar Jul 19, 2019 •

edited

Loading

rmulhol left a comment •

edited

Loading