Add Graphql support to scribe's data #165

CryptoMaxPlanck · 2022-09-05T02:56:38Z

Description
The Scribe will go through the history of chains and backfill data into a database. However, for this data to be useful, an API around it must be build to easily retrieve data. This PR uses GraphQL to build an API around the scribe's data.

Todo:

Deprecate database retrieval functions in favor of one with filter
fix context issues
fix generate
fix closing of request and listenandserve
add resolver tests

CryptoMaxPlanck · 2022-09-05T02:58:04Z

services/scribe/graphql/server/contrib/main.go

+	"github.com/integralist/go-findroot/find"
+)
+
+func main() {


the generate from this needs to be fixed. is popping up files in the wrong dir.

CryptoMaxPlanck · 2022-09-05T02:58:38Z

services/scribe/graphql/server/gin.go

+
+	fmt.Printf("started graphiql server on port: http://localhost:%d/graphiql\n", port)
+
+	// TODO: respect context cancellation


need to figure out how to do this

thsi is a bit of a pain. Might make sense for you to add this to core as a server functionality an dimport from there.

var lc net.ListenConfig host := fmt.Sprintf("0.0.0.0:%d", s.config.RPCPort) s.listener, err = lc.Listen(ctx, "tcp", host) if err != nil { return fmt.Errorf("could not listen on %s: %w", host, err) } go func() { err := s.server.Serve(s.listener) if err != nil { logger.Errorf(fmt.Sprintf("rpc server got error: %v", err)) } }() select { case <-ctx.Done(): return }

CryptoMaxPlanck · 2022-09-05T02:59:23Z

services/scribe/graphql/server/gin.go

+func initDB(database string, path string) (db.EventDB, error) {
+	switch {
+	case database == "sqlite":
+		sqliteStore, err := sqlite.NewSqliteStore(context.TODO(), path)


what context am i supposed to use here if start doesn't have one? do I just make a new one with a timeout and pass that in to make sure the function doesn't take more than x seconds (like 5 seconds)

should be the one from start

CryptoMaxPlanck · 2022-09-05T03:00:15Z

services/scribe/cmd/commands.go

+var portFlag = &cli.UintFlag{
+	Name:  "port",
+	Usage: "--port 5121",
+	//nolint:staticcheck


is this okay to use even though it is depracated? the alternative is using net.listen with port of :0 and then getting that port. but then would need to add a function with panics in case of errors, so less clean.

We should probably handle errors in an init here. Something like

func init() { ports := freeport.get(1) if len(port) > 0 { port = uint16(ports[0]) } } var port uint16

CryptoMaxPlanck · 2022-09-05T03:01:14Z

services/scribe/db/event.go

+	// RetrieveEthTxsInRange retrieves eth transactions in a range.
+	RetrieveEthTxsInRange(ctx context.Context, ethTxFilter EthTxFilter, startBlock, endBlock uint64) ([]types.Transaction, error)
+
+	// TODO: Deprecate these methods in favor of the above methods.


will progressively go thru and deprecate these throughout scribe. just want to get ur review on it first.

codecov · 2022-09-05T03:02:33Z

Codecov Report

Merging #165 (d45447d) into master (8d51934) will decrease coverage by 0.10890%.
The diff coverage is 0.80645%.

@@                 Coverage Diff                 @@
##              master        #165         +/-   ##
===================================================
- Coverage   52.41622%   52.30732%   -0.10891%     
===================================================
  Files            185         185                 
  Lines           8236        8278         +42     
  Branches          83          83                 
===================================================
+ Hits            4317        4330         +13     
- Misses          3512        3541         +29     
  Partials         407         407

Impacted Files	Coverage Δ
services/scribe/db/datastore/sql/base/log.go	`0.00000% <0.00000%> (ø)`
services/scribe/db/datastore/sql/base/receipt.go	`0.00000% <0.00000%> (ø)`
...rvices/scribe/db/datastore/sql/base/transaction.go	`0.00000% <0.00000%> (ø)`
services/scribe/backfill/contract.go	`75.00000% <100.00000%> (+1.28204%)`	⬆️
ethergo/chain/block.go	`68.54839% <0.00000%> (+2.41935%)`	⬆️
ethergo/signer/signer/kmssigner/signing.go	`42.39130% <0.00000%> (+3.26087%)`	⬆️
services/scribe/backfill/filter.go	`74.69880% <0.00000%> (+6.02410%)`	⬆️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

CryptoMaxPlanck · 2022-09-05T03:06:14Z

services/scribe/graphql/integration_test/integration_test.go

+	i.Eventually(func() bool {
+		request, err := http.NewRequestWithContext(i.GetTestContext(), http.MethodGet, fmt.Sprintf("%s%s", baseURL, server.GraphiqlEndpoint), nil)
+		Nil(i.T(), err)
+		_, err = i.gqlClient.Client.Client.Do(request)


need to look into how to close this without messing up the tests. (or is ignoring safe??)

res, err = i.gqlClient.Client.Client.Do(request) defer res.Close()

CryptoMaxPlanck · 2022-09-05T15:24:10Z

services/scribe/graphql/server/graph/resolver.go

+//
+// It serves as dependency injection for your app, add any dependencies you require here.
+
+// go:generate go run github.com/synapsecns/sanguine/services/scribe/server/contrib


again, fix the generate stuff

CryptoMaxPlanck · 2022-09-05T15:24:55Z

services/scribe/graphql/server/graph/utils.go

+	}
+}
+
+func (r Resolver) buildLogFilter(contractAddress *string, blockNumber *int, txHash *string, txIndex *int, blockHash *string, index *int) db.LogFilter {


this is not a super clean way to do this, but not sure how else. maybe utilize optionals in the future?

yeah, I like that. If we're going to do it, let's throw it in core and use it cross-lib from go-ethereum.

This will become much easier when ethereum fully adopts go 1.18 ethereum/go-ethereum#24549

services/scribe/db/filters.go

trajan0x · 2022-09-05T17:53:12Z

services/scribe/cmd/commands.go

+var portFlag = &cli.UintFlag{
+	Name:  "port",
+	Usage: "--port 5121",
+	//nolint:staticcheck


We should probably handle errors in an init here. Something like

func init() { ports := freeport.get(1) if len(port) > 0 { port = uint16(ports[0]) } } var port uint16

trajan0x · 2022-09-05T17:53:49Z

services/scribe/cmd/commands.go

+	Description: "starts a graphql server",
+	Flags:       []cli.Flag{portFlag, dbFlag, pathFlag},
+	Action: func(c *cli.Context) error {
+		err := server.Start(uint16(c.Uint(portFlag.Name)), c.String(dbFlag.Name), c.String(pathFlag.Name))


This should 100% have a context passed in

services/scribe/db/datastore/sql/base/log.go

trajan0x · 2022-09-05T17:55:23Z

services/scribe/db/datastore/sql/base/log.go

-// RetrieveLogs retrieves all logs that match a tx hash and chain id.
-func (s Store) RetrieveLogs(ctx context.Context, txHash common.Hash, chainID uint32) (logs []*types.Log, err error) {
+func logFilterToQuery(logFilter db.LogFilter) Log {
+	return Log{


Does this work if certain fields are nil?

yup. will be done in tests when RetrieveLogsByTxHash is deprecated cuz ill be using a filter w only two fields then.

got it, that going to be in this pr?

ya, the logs_test file fully uses this now

trajan0x · 2022-09-05T17:56:00Z

services/scribe/db/datastore/sql/base/log.go

+func (s Store) RetrieveLogsInRange(ctx context.Context, logFilter db.LogFilter, startBlock, endBlock uint64) (logs []*types.Log, err error) {
+	dbLogs := []Log{}
+	queryFilter := logFilterToQuery(logFilter)
+	rangeQuery := BlockNumberFieldName + " BETWEEN ? AND ?"


fmt.Sprintf("%s BETWEEN ? AND ?", BlockNumberFieldName)

trajan0x · 2022-09-05T18:02:44Z

services/scribe/graphql/server/gin.go

+func initDB(database string, path string) (db.EventDB, error) {
+	switch {
+	case database == "sqlite":
+		sqliteStore, err := sqlite.NewSqliteStore(context.TODO(), path)


should be the one from start

trajan0x · 2022-09-05T18:03:11Z

services/scribe/graphql/server/gin.go

+}
+
+// Start starts the server and initializes the database.
+func Start(port uint16, database string, path string) error {


should take a context, c *cli.Context is a context (var _ context.Context = &cli.Context) so that can be passed in here

trajan0x · 2022-09-05T18:05:09Z

services/scribe/graphql/server/gin.go

+
+	fmt.Printf("started graphiql server on port: http://localhost:%d/graphiql\n", port)
+
+	// TODO: respect context cancellation


thsi is a bit of a pain. Might make sense for you to add this to core as a server functionality an dimport from there.

var lc net.ListenConfig host := fmt.Sprintf("0.0.0.0:%d", s.config.RPCPort) s.listener, err = lc.Listen(ctx, "tcp", host) if err != nil { return fmt.Errorf("could not listen on %s: %w", host, err) } go func() { err := s.server.Serve(s.listener) if err != nil { logger.Errorf(fmt.Sprintf("rpc server got error: %v", err)) } }() select { case <-ctx.Done(): return }

trajan0x · 2022-09-05T18:07:54Z

services/scribe/graphql/server/graph/utils.go

+	}
+}
+
+func (r Resolver) buildLogFilter(contractAddress *string, blockNumber *int, txHash *string, txIndex *int, blockHash *string, index *int) db.LogFilter {


yeah, I like that. If we're going to do it, let's throw it in core and use it cross-lib from go-ethereum.

This will become much easier when ethereum fully adopts go 1.18 ethereum/go-ethereum#24549

trajan0x · 2022-09-05T18:08:14Z

services/scribe/graphql/server/graph/utils.go

+func (r Resolver) buildReceiptFilter(txHash *string, contractAddress *string, blockHash *string, blockNumber *int, transactionIndex *int) db.ReceiptFilter {
+	receiptFilter := db.ReceiptFilter{}
+	if txHash != nil {
+		receiptFilter.TxHash = *txHash


not sure how I feel about the potential nil dereference here

is there a scenario where the ref is not nil, but the de-ref is?

trajan0x · 2022-09-06T23:05:55Z

services/scribe/db/datastore/sql/base/log.go

-// RetrieveLogs retrieves all logs that match a tx hash and chain id.
-func (s Store) RetrieveLogs(ctx context.Context, txHash common.Hash, chainID uint32) (logs []*types.Log, err error) {
+func logFilterToQuery(logFilter db.LogFilter) Log {
+	return Log{


got it, that going to be in this pr?

trajan0x · 2022-09-06T23:06:20Z

services/scribe/db/datastore/sql/base/log.go

+}
+
+// RetrieveLogsWithFilter retrieves all logs that match a filter.
+func (s Store) RetrieveLogsWithFilter(ctx context.Context, logFilter db.LogFilter) (logs []*types.Log, err error) {


Let's handle in a future pr

trajan0x · 2022-09-06T23:09:38Z

services/scribe/graphql/server/graph/queries.resolvers.go

+	"fmt"
+
+	"github.com/synapsecns/sanguine/services/scribe/graphql/server/graph/model"
+	resolvers "github.com/synapsecns/sanguine/services/scribe/graphql/server/graph/resolver"


see ci build issue

This reverts commit a6fc4c3.

This reverts commit b0a691e.

This reverts commit 06bfec3.

trajan0x and others added 7 commits September 1, 2022 15:39

add graphql implementation

3d0d972

save progress in case of hardware failure

43c3210

pushing in case this regen breaks something

70722f3

working tests

1132579

restructure graphql stuff

5836556

remove missed folder

2d1e0a4

v1

bda60bd

github-actions bot added the go Pull requests that update Go code label Sep 5, 2022

CryptoMaxPlanck commented Sep 5, 2022

View reviewed changes

remove comment

419ae4e

CryptoMaxPlanck commented Sep 5, 2022

View reviewed changes

CryptoMaxPlanck marked this pull request as draft September 5, 2022 03:31

CryptoMaxPlanck commented Sep 5, 2022

View reviewed changes

services/scribe/db/filters.go Show resolved Hide resolved

trajan0x requested changes Sep 5, 2022

View reviewed changes

CryptoMaxPlanck added 8 commits September 5, 2022 16:04

some changes and deprecating eventdbreader funcs

b9c970b

'Merge' branch 'master' into feat/graphql

d78faca

more go mod stuff

09a7b04

port and server fixes

ac959cf

fix testing problems

d28f1c6

removed unneeded file

be684b2

small

79ed64b

add require

7c9a2af

github-actions bot added the needs-go-generate label Sep 6, 2022

CryptoMaxPlanck added 4 commits September 5, 2022 23:06

change dir

18cfc9e

readme dir

67826e3

go mod verison for gqlgen

12cb02f

yaml lint

8eb352c

github-actions bot added needs-go-generate-ethergo needs-go-generate-services/scribe needs-go-generate-agents and removed needs-go-generate-core needs-go-generate-services/scribe needs-go-generate-agents labels Sep 6, 2022

trajan0x added 7 commits September 6, 2022 17:41

gi

ab68892

ethergo fix

0533f04

use contains

75f1fda

syntax fix

07dc5e7

fix contains

8279afc

recontain

8c5bd4b

contains

9682ff7

github-actions bot added needs-go-generate-services/scribe and removed needs-go-generate-services/scribe needs-go-generate-ethergo labels Sep 6, 2022

trajan0x requested changes Sep 6, 2022

View reviewed changes

manually readd stuff

389d58b

github-actions bot removed the needs-go-generate-services/scribe label Sep 6, 2022

trajan0x and others added 8 commits September 6, 2022 19:35

srevice

596388e

pupterm

a6fc4c3

[ci skip]

7d82b3e

Revert "pupterm"

8cdfe51

This reverts commit a6fc4c3.

hi

b0a691e

Revert "hi"

402bde2

This reverts commit b0a691e.

test

06bfec3

Revert "test"

d45447d

This reverts commit 06bfec3.

trajan0x merged commit 7d59e36 into master Sep 7, 2022

trajan0x deleted the feat/graphql branch September 7, 2022 00:43

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Graphql support to scribe's data #165

Add Graphql support to scribe's data #165

CryptoMaxPlanck commented Sep 5, 2022 •

edited

Loading

CryptoMaxPlanck Sep 5, 2022

CryptoMaxPlanck Sep 5, 2022

trajan0x Sep 5, 2022

CryptoMaxPlanck Sep 5, 2022

trajan0x Sep 5, 2022

CryptoMaxPlanck Sep 5, 2022

trajan0x Sep 5, 2022

CryptoMaxPlanck Sep 5, 2022

trajan0x Sep 5, 2022

codecov bot commented Sep 5, 2022 •

edited

Loading

CryptoMaxPlanck Sep 5, 2022

trajan0x Sep 5, 2022

CryptoMaxPlanck Sep 5, 2022

CryptoMaxPlanck Sep 5, 2022

trajan0x Sep 5, 2022

trajan0x Sep 5, 2022

trajan0x Sep 5, 2022

trajan0x Sep 5, 2022

CryptoMaxPlanck Sep 5, 2022

trajan0x Sep 6, 2022

CryptoMaxPlanck Sep 6, 2022

trajan0x Sep 5, 2022

trajan0x Sep 5, 2022

trajan0x Sep 5, 2022

trajan0x Sep 5, 2022

trajan0x Sep 5, 2022

trajan0x Sep 5, 2022

CryptoMaxPlanck Sep 5, 2022

trajan0x Sep 6, 2022

trajan0x Sep 6, 2022

trajan0x Sep 6, 2022


		fmt.Printf("started graphiql server on port: http://localhost:%d/graphiql\n", port)

		// TODO: respect context cancellation

Add Graphql support to scribe's data #165

Add Graphql support to scribe's data #165

Conversation

CryptoMaxPlanck commented Sep 5, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Sep 5, 2022 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

CryptoMaxPlanck commented Sep 5, 2022 •

edited

Loading

codecov bot commented Sep 5, 2022 •

edited

Loading