indexer binary #168

miiu96 · 2022-09-01T11:55:40Z

Make indexer a stand-alone binary that will receive data over web-sockets
Also the old functionality is there (can be used as a go module)

cmd/elasticindexer/config/config.toml

cmd/elasticindexer/main.go

factory/wsIndexerFactory.go

.github/workflows/pr-build.yml

process/wsindexer/indexer.go

process/wsindexer/interface.go

process/wsindexer/unmarsall.go

cmd/elasticindexer/config/config.toml

factory/wsIndexerFactory.go

bogdan-rosianu · 2022-09-29T08:32:27Z

go.mod

-	github.com/ElrondNetwork/elrond-go-core v1.1.19-0.20220829095131-68a85ab9fe00
-	github.com/ElrondNetwork/elrond-go-logger v1.0.7
-	github.com/ElrondNetwork/elrond-vm-common v1.3.7
+	github.com/ElrondNetwork/elrond-go-core v1.1.21-0.20220927113554-34ccab6be8c3


proper tags before merging

will do after reviews and testing

cmd/elasticindexer/main.go

process/factory/indexerFactory.go

process/wsclient/client.go

process/wsindexer/indexer.go

bogdan-rosianu · 2022-10-03T10:27:38Z

cmd/elasticindexer/config/config.toml

-        url = "http://localhost:9200"
-        username = ""
-        password = ""
-        bulk-request-max-size-in-bytes = 4194304 # 4MB


enabled-indices and web-socket are still "prefs".. might move them as well

bogdan-rosianu · 2022-10-03T10:28:43Z

cmd/elasticindexer/main.go

-	log.Info("closing indexer")
-	if !check.IfNil(fileLogging) {
+	<-interrupt
+	wsClient.Close()


might add a log before wsClient.Close() closing app at user's signal or something similar

bogdan-rosianu · 2022-10-03T10:29:28Z

cmd/elasticindexer/main.go

@@ -131,5 +147,15 @@ func initializeLogger(ctx *cli.Context, cfg *config.Config) (file.FileLoggingHan
 		return nil, err
 	}

+	err = logger.RemoveLogObserver(os.Stdout)


might extract the new code in a function removeANSICollorsForLogger

bogdan-rosianu · 2022-10-03T10:31:23Z

integrationtests/accountsBalanceNftTransfer_test.go

@@ -68,7 +68,7 @@ func TestAccountBalanceNFTTransfer(t *testing.T) {
 		},
 	}

-	err = esProc.SaveTransactions(body, header, pool, coreAlteredAccounts, false, 3)
+	err = esProc.SaveTransactions(body, header, pool, coreAlteredAccounts, false, testsNumOfShards)


testsNumOfShards -> testNumOfShards

bogdan-rosianu · 2022-10-03T10:34:24Z

process/wsclient/client.go

 }

-func NewWebSocketClient(urlReceive string, actions map[data.OperationType]func(marshalledData []byte) error) (*client, error) {
-	urlReceiveData := url.URL{Scheme: "ws", Host: fmt.Sprintf(urlReceive), Path: "/operations"}
+// New will create a new instance of web-sockets client


bogdan-rosianu · 2022-10-03T10:35:04Z

process/wsclient/client.go

+			if strings.Contains(err.Error(), closedConnection) {
+				return true
+			}
+			log.Warn("websocket error, retrying in ...", "error", err.Error())


bogdan-rosianu · 2022-10-03T10:35:46Z

process/wsindexer/indexer.go

 	return &indexer{
 		marshaller: marshaller,
 		di:         dataIndexer,
 	}, nil
 }

-func (i *indexer) GetFunctionsMap() map[data.OperationType]func(d []byte) error {
+func (i *indexer) GetOperationsMap() map[data.OperationType]func(d []byte) error {


add comment

bogdan-rosianu · 2022-10-03T10:36:03Z

process/wsindexer/interface.go

@@ -5,11 +5,13 @@ import (
 	"github.com/ElrondNetwork/elrond-go-core/data/outport"
 )

+// WSClient defines what a websockets client should do


iulianpascalau · 2022-10-05T14:20:50Z

cmd/elasticindexer/config/prefs.toml

@@ -0,0 +1,16 @@
+[config]
+    enabled-indices =  [


as agreed, let's move back in the config this list, we will call it available-indices and in this prefs.toml we will have the disabled-indices

iulianpascalau · 2022-10-05T14:23:08Z

cmd/elasticindexer/main.go

 	return fileLogging, nil
 }
+
+func removeANSIColorsForLogger() error {


this should be conditioned by a flag. Why do we want to remove the colors?

added a flag for this.

iulianpascalau · 2022-10-05T14:24:56Z

.github/workflows/golangci-lint.yml

+on:
+  push:
+    branches:
+      - master


this will need to get changed to branches: [ master, development, feat/*, rc/* ] as fixing a linter issue on a PR will not start the re-check process. Where did you get this file? So we can also apply the patch there.

iulianpascalau · 2022-10-05T14:26:47Z

integrationtests/consts.go

+package integrationtests
+
+const (
+	//nolint


why nolint call here?

gogang-ci linter detects that these constants are not used, which is false

iulianpascalau · 2022-10-05T14:27:43Z

integrationtests/consts.go

+
+const (
+	//nolint
+	testNumOfShards = 3


👍 for defining consts

iulianpascalau · 2022-10-05T14:28:27Z

integrationtests/utils.go

 func setLogLevelDebug() {
 	_ = logger.SetLogLevel("process:DEBUG")
 }

+//nolint


why nolint?

gogang-ci linter detects that function is not used, which is false

iulianpascalau · 2022-10-05T14:36:34Z

process/wsclient/client.go

-			case <-time.After(time.Second):
-			}
+		closed := c.listeningOnWebSocket()
+		if closed {


so this process will immediately close if the node process closes but when it is first started, it will wait until the node process boots up. Is this correct?

If it is first started will close immediately, and it will wait on the close when writing something in the Elasticsearch database

iulianpascalau · 2022-10-05T14:40:51Z

process/wsindexer/types.go

+}
+
+type argsSaveBlock struct {
+	TransactionsPool *poolStruct


Exported field in an unexported struct of an unexported type. "cessary" exported TransactionsPool param

Yes. This structure is used to can unmarshal the received data

bogdan-rosianu · 2022-10-06T12:23:29Z

factory/wsIndexerFactory.go

+
+	for _, availableIndex := range availableIndices {
+		_, found := mapDisabledIndices[availableIndex]
+		if !found {


rename found to shouldSkip and refactor: if shouldSkip { continue } indices = ...

miiu96 added 26 commits September 1, 2022 14:54

indexer binary

1e04687

remove shard coordinator from dataIndexer.go

e5faae5

change folder structure

7356166

fix build

8d2adcc

ws client and indexer

a1f1dc9

move factory

a192b52

small fix

0672c96

remove is in import db flag

02a08b1

refactoring and fix unit tests

fc9a3d4

fix integration tests

7d9abbd

remove shard coordinator

15ebfe7

update vm common and fixes

ee5ab58

fix unit tests

3f2f38b

fix integration tests 1

e112162

fix integration tests 2

2b74722

move factory

4b904e1

create ws client and indexer

3ab9c0d

go mod tidy

961c00b

refactor client in order to retry to connect

f965585

unmarshall data

ec81d2a

latest elrond go core

20f3e23

fix unmarshal

bab1d99

remove comment code and extra checks

a435a28

change log

7cebfc3

change log

2d410cf

ignore peer blocks

b5d8871

iulianpascalau self-requested a review September 29, 2022 07:15

bogdan-rosianu self-requested a review September 29, 2022 08:24

miiu96 marked this pull request as ready for review September 29, 2022 08:26

iulianpascalau reviewed Sep 29, 2022

View reviewed changes

bogdan-rosianu requested changes Sep 29, 2022

View reviewed changes

miiu96 added 4 commits September 30, 2022 11:32

fixes after review 1

6de5af7

implement close

9b52471

fix linter issues

f0b4137

latest elrond go logger

d8a3f18

bogdan-rosianu requested changes Oct 3, 2022

View reviewed changes

fixes after re review

14ddedc

bogdan-rosianu previously approved these changes Oct 5, 2022

View reviewed changes

iulianpascalau reviewed Oct 5, 2022

View reviewed changes

fixes

cd568fa

miiu96 dismissed bogdan-rosianu’s stale review via cd568fa October 6, 2022 08:12

miiu96 added 2 commits October 6, 2022 11:55

available indices and disabled

d9de7ca

latest elrond-go core

7d83c40

bogdan-rosianu reviewed Oct 6, 2022

View reviewed changes

small fix

fd94fbb

bogdan-rosianu previously approved these changes Oct 6, 2022

View reviewed changes

change action

104ae39

miiu96 dismissed bogdan-rosianu’s stale review via 104ae39 October 6, 2022 12:41

iulianpascalau approved these changes Oct 6, 2022

View reviewed changes

bogdan-rosianu approved these changes Oct 6, 2022

View reviewed changes

miiu96 merged commit a09a9b9 into feat/altered-accounts Oct 6, 2022

miiu96 deleted the indexer-binary branch October 6, 2022 12:43

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

indexer binary #168

indexer binary #168

miiu96 commented Sep 1, 2022 •

edited

Loading

bogdan-rosianu Sep 29, 2022

miiu96 Sep 29, 2022

bogdan-rosianu Oct 3, 2022

miiu96 Oct 3, 2022

bogdan-rosianu Oct 3, 2022

miiu96 Oct 3, 2022

bogdan-rosianu Oct 3, 2022

miiu96 Oct 3, 2022

bogdan-rosianu Oct 3, 2022

miiu96 Oct 3, 2022

bogdan-rosianu Oct 3, 2022

miiu96 Oct 3, 2022

bogdan-rosianu Oct 3, 2022

miiu96 Oct 3, 2022

bogdan-rosianu Oct 3, 2022

miiu96 Oct 3, 2022

bogdan-rosianu Oct 3, 2022

miiu96 Oct 3, 2022

iulianpascalau Oct 5, 2022

iulianpascalau Oct 5, 2022

miiu96 Oct 6, 2022

iulianpascalau Oct 5, 2022

miiu96 Oct 6, 2022

iulianpascalau Oct 5, 2022

miiu96 Oct 6, 2022

iulianpascalau Oct 5, 2022

iulianpascalau Oct 5, 2022

miiu96 Oct 6, 2022

iulianpascalau Oct 5, 2022

miiu96 Oct 6, 2022

iulianpascalau Oct 5, 2022

miiu96 Oct 6, 2022

bogdan-rosianu Oct 6, 2022

miiu96 Oct 6, 2022

indexer binary #168

indexer binary #168

Conversation

miiu96 commented Sep 1, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

miiu96 commented Sep 1, 2022 •

edited

Loading