[alpha] move all the devel changes to alpha (too many to cherry-pick) (…

…#3975) * Change version to alpha (#3926) Co-authored-by: Alexey Sharp <[email protected]> Co-authored-by: Alex Sharp <[email protected]> * docs: update libmdbx links (#3929) * Makefile: refactor build flags and fix 1.17 (#3930) * Fix some cli flag descriptions (#3933) * Fix some cli flag descriptions * add node about verbosity * min requirement to go 1.18 (#3934) * save * save * save * Added Ethstats service (#3931) * somewhat there but not yet * lol * more efficient ethstats * lint * not die on no wifi * Update bor mumbai config (#3937) * Update ci.yml (#3936) * Use heimdall url in integration bor consensus (#3940) * Downloader: re-use flags defaults (#3941) * torrent: print peers amount in logs (#3942) * Observer - P2P network crawler (#3928) Observer crawls the Ethereum network and collects information about the nodes. * Torrent conns print (#3943) * save * save * [erigon2] Fuzz tests for commitment (#3939) * [erigon2] Fuzz tests for commitment * Cleanup * Update to erigon-lib main Co-authored-by: Alexey Sharp <[email protected]> * Introduce unlimited download rate (#3945) * Introduce unlimited download rate * More generous burst Co-authored-by: Alexey Sharp <[email protected]> * Replace ioutil with io and os (#3946) * Sentry GRPC: rename Peers to PeerEvents (#3944) * Sentry GRPC: rename Peers to PeerEvents see erigontech/interfaces#101 * Update to erigon-lib main Co-authored-by: Alexey Sharp <[email protected]> * cleaned up forkchoices db insertions #3949 * fixed ethstats (#3951) * bsc: disable snap sync (#3955) * bsc: disable snap sync (#3956) * Snapshots: support empty buf case (#3957) * Snapshots: rare nil pointer at fresh start (#3958) * got rid of the automatic usage of net api (#3952) * got rid of the automatic usage of net api * less confusing comment * ops * ops2 * important * ops * RPC: admin.peers() (#3960) * RPC: admin.peers() This RPC method returns information about the connected remote nodes. https://geth.ethereum.org/docs/rpc/ns-admin#admin_peers The peers are collected from all configured sentries. See: erigontech/interfaces#102 Test with: curl -X POST -H "Content-Type: application/json" --data '{"jsonrpc": "2.0", "method": "admin_peers", "params": [], "id":1}' localhost:8545 * save * liner fix Co-authored-by: alex.sharov <[email protected]> * sentry: refactor flags, add maxpeers. (#3961) * Experiment files 1 by 1 (#3959) * Experiment files 1 by 1 * Remove check * sort preverified snapshots * docs: docker permissions * sort preverified snapshots * sort preverified snapshots * sort preverified snapshots * sort preverified snapshots * sort preverified snapshots * sort preverified snapshots * save * Fix speed log, remove file name * Move timer out of the loop * Calculate total size of downloaded files * Fixes * Fix * Fix * Fix * Move downloadData * Fix * Revert "Fix" This reverts commit 038e02b. * Revert "Move downloadData" This reverts commit 8130a4d. * Revert "Fix" This reverts commit 1dca25b. * Revert "Fix" This reverts commit ee5a1e8. * Revert "Fix" This reverts commit 8af7be7. * Revert "Fixes" This reverts commit 50509af. * Revert "Calculate total size of downloaded files" This reverts commit 64a26df. * Remove progress * Remove progress Co-authored-by: Alexey Sharp <[email protected]> Co-authored-by: alex.sharov <[email protected]> * Update stage_headers.go (#3966) * Snapshots: open bittorrent udp port in docker (#3969) * Snapshots: open torrent udp in docker-compose.yml * Snapshots: open torrent udp in docker-compose.yml * Delete blocks in [from, to) range (#3970) * Snapshots: allow stage_headers --unwind behind available snapshots (#3971) * save * save * save * Integration: allow headers --reset (#3972) * Bsc: enable syncmode=snap by default #3973 Co-authored-by: Alexey Sharp <[email protected]> Co-authored-by: Alex Sharp <[email protected]> Co-authored-by: battlmonstr <[email protected]> Co-authored-by: Chase Wright <[email protected]> Co-authored-by: Alex Sharov <[email protected]> Co-authored-by: Giulio rebuffo <[email protected]> Co-authored-by: Krishna Upadhyaya <[email protected]> Co-authored-by: Håvard Anda Estensen <[email protected]> Co-authored-by: Enrique Jose Avila Asapche <[email protected]>
erigontech · Apr 26, 2022 · 3c37b39 · 3c37b39
1 parent 9ccc76a
commit 3c37b39
Show file tree

Hide file tree

Showing 100 changed files with 4,811 additions and 472 deletions.
diff --git a/Dockerfile b/Dockerfile
@@ -21,7 +21,7 @@ RUN chown -R erigon:erigon /home/erigon
 
 USER erigon
 
-EXPOSE 8545 8550 8551 8546 30303 30303/udp 30304 30304/udp 8080 9090 6060
+EXPOSE 8545 8550 8551 8546 30303 30303/udp 42069 42069/udp 8080 9090 6060
 
 # https://github.com/opencontainers/image-spec/blob/main/annotations.md
 ARG BUILD_DATE

diff --git a/Makefile b/Makefile
@@ -61,6 +61,7 @@ COMMANDS += downloader
 COMMANDS += evm
 COMMANDS += hack
 COMMANDS += integration
+COMMANDS += observer
 COMMANDS += pics
 COMMANDS += rpcdaemon
 COMMANDS += rpctest

diff --git a/README.md b/README.md
@@ -362,6 +362,12 @@ Reserved for future use: **gRPC ports**: `9092` consensus engine, `9093` snapsho
 ### How to run local devnet?
 <code> 🔬 Detailed explanation is [here](/DEV_CHAIN.md).</code>
 
+### Docker permissions error
+
+Docker uses user erigon with UID/GID 1000 (for security reasons). You can see this user being created in the Dockerfile.
+Can fix by giving a host's user ownership of the folder, where the host's user UID/GID is the same as the docker's user UID/GID (1000).
+More details in [post](https://www.fullstaq.com/knowledge-hub/blogs/docker-and-the-host-filesystem-owner-matching-problem)
+
 Getting in touch
 ================
 

diff --git a/accounts/abi/bind/bind_test.go b/accounts/abi/bind/bind_test.go
diff --git a/cmd/abigen/main.go b/cmd/abigen/main.go
@@ -19,7 +19,7 @@ package main
 import (
 	"encoding/json"
 	"fmt"
-	"io/ioutil"
+	"io"
 	"os"
 	"path/filepath"
 	"regexp"
@@ -151,9 +151,9 @@ func abigen(c *cli.Context) error {
 		)
 		input := c.GlobalString(abiFlag.Name)
 		if input == "-" {
-			abi, err = ioutil.ReadAll(os.Stdin)
+			abi, err = io.ReadAll(os.Stdin)
 		} else {
-			abi, err = ioutil.ReadFile(input)
+			abi, err = os.ReadFile(input)
 		}
 		if err != nil {
 			utils.Fatalf("Failed to read input ABI: %v", err)
@@ -162,7 +162,7 @@ func abigen(c *cli.Context) error {
 
 		var bin []byte
 		if binFile := c.GlobalString(binFlag.Name); binFile != "" {
-			if bin, err = ioutil.ReadFile(binFile); err != nil {
+			if bin, err = os.ReadFile(binFile); err != nil {
 				utils.Fatalf("Failed to read input bytecode: %v", err)
 			}
 			if strings.Contains(string(bin), "//") {
@@ -209,7 +209,7 @@ func abigen(c *cli.Context) error {
 			}
 
 		case c.GlobalIsSet(jsonFlag.Name):
-			jsonOutput, err := ioutil.ReadFile(c.GlobalString(jsonFlag.Name))
+			jsonOutput, err := os.ReadFile(c.GlobalString(jsonFlag.Name))
 			if err != nil {
 				utils.Fatalf("Failed to read combined-json from compiler: %v", err)
 			}
@@ -259,7 +259,7 @@ func abigen(c *cli.Context) error {
 		fmt.Printf("%s\n", code)
 		return nil
 	}
-	if err := ioutil.WriteFile(c.GlobalString(outFlag.Name), []byte(code), 0600); err != nil {
+	if err := os.WriteFile(c.GlobalString(outFlag.Name), []byte(code), 0600); err != nil {
 		utils.Fatalf("Failed to write ABI binding: %v", err)
 	}
 	return nil

diff --git a/cmd/devnettest/requests/request_generator.go b/cmd/devnettest/requests/request_generator.go
@@ -3,7 +3,7 @@ package requests
 import (
 	"errors"
 	"fmt"
-	"io/ioutil"
+	"io"
 	"net/http"
 	"time"
 
@@ -56,7 +56,7 @@ func (req *RequestGenerator) Get() rpctest.CallResult {
 		return res
 	}
 
-	body, err := ioutil.ReadAll(resp.Body)
+	body, err := io.ReadAll(resp.Body)
 	if err != nil {
 		res.Took = time.Since(start)
 		res.Err = err

diff --git a/cmd/downloader/downloader/downloader.go b/cmd/downloader/downloader/downloader.go
@@ -102,8 +102,6 @@ func LoggingLoop(ctx context.Context, torrentClient *torrent.Client) {
 				case <-t.GotInfo(): // all good
 					gotInfo++
 				default:
-					t.AllowDataUpload()
-					t.AllowDataDownload()
 				}
 				allComplete = allComplete && t.Complete.Bool()
 			}
@@ -118,8 +116,8 @@ func LoggingLoop(ctx context.Context, torrentClient *torrent.Client) {
 				log.Info("[torrent] Seeding",
 					"download", common2.ByteCount(uint64(stats.readBytesPerSec))+"/s",
 					"upload", common2.ByteCount(uint64(stats.writeBytesPerSec))+"/s",
-					"peers", stats.peersCount,
-					"torrents", stats.torrentsCount,
+					"unique_peers", stats.peersCount,
+					"files", stats.torrentsCount,
 					"alloc", common2.ByteCount(m.Alloc), "sys", common2.ByteCount(m.Sys))
 				continue
 			}
@@ -128,8 +126,8 @@ func LoggingLoop(ctx context.Context, torrentClient *torrent.Client) {
 				"Progress", fmt.Sprintf("%.2f%%", stats.Progress),
 				"download", common2.ByteCount(uint64(stats.readBytesPerSec))+"/s",
 				"upload", common2.ByteCount(uint64(stats.writeBytesPerSec))+"/s",
-				"peers", stats.peersCount,
-				"torrents", stats.torrentsCount,
+				"unique_peers", stats.peersCount,
+				"files", stats.torrentsCount,
 				"alloc", common2.ByteCount(m.Alloc), "sys", common2.ByteCount(m.Sys))
 			if stats.peersCount == 0 {
 				ips := torrentClient.BadPeerIPs()
@@ -170,27 +168,15 @@ func CalcStats(prevStats AggStats, interval time.Duration, client *torrent.Clien
 	//var aggCompletedPieces, aggNumPieces, aggPartialPieces int
 	peers := map[torrent.PeerID]*torrent.PeerConn{}
 	torrents := client.Torrents()
+	connStats := client.ConnStats()
+
+	result.bytesRead += connStats.BytesReadUsefulIntendedData.Int64()
+	result.bytesWritten += connStats.BytesWrittenData.Int64()
+
 	for _, t := range torrents {
-		stats := t.Stats()
-		/*
-			var completedPieces, partialPieces int
-			psrs := t.PieceStateRuns()
-			for _, r := range psrs {
-				if r.Complete {
-					completedPieces += r.Length
-				}
-				if r.Partial {
-					partialPieces += r.Length
-				}
-			}
-			aggCompletedPieces += completedPieces
-			aggPartialPieces += partialPieces
-			aggNumPieces = t.NumPieces()
-		*/
-		result.bytesRead += stats.BytesRead.Int64() + stats.BytesReadData.Int64()
-		result.bytesWritten += stats.BytesWritten.Int64() + stats.BytesWrittenData.Int64()
 		aggBytesCompleted += t.BytesCompleted()
 		aggLen += t.Length()
+
 		for _, peer := range t.PeerConns() {
 			peers[peer.PeerID] = peer
 		}
@@ -275,6 +261,9 @@ func ResolveAbsentTorrents(ctx context.Context, torrentClient *torrent.Client, p
 		case <-ctx.Done():
 			return ctx.Err()
 		case <-t.GotInfo():
+			if !t.Complete.Bool() {
+				t.DownloadAll()
+			}
 			mi := t.Metainfo()
 			if err := CreateTorrentFileIfNotExists(snapshotDir, t.Info(), &mi); err != nil {
 				return err

diff --git a/cmd/downloader/downloader/grpc_server.go b/cmd/downloader/downloader/grpc_server.go
@@ -41,9 +41,11 @@ func CreateTorrentFilesAndAdd(ctx context.Context, snapshotDir *dir.Rw, torrentC
 		return err
 	}
 	for _, t := range torrentClient.Torrents() {
-		t.AllowDataDownload()
 		t.AllowDataUpload()
-		t.DownloadAll()
+		if !t.Complete.Bool() {
+			t.AllowDataDownload()
+			t.DownloadAll()
+		}
 	}
 	return nil
 }
@@ -78,7 +80,9 @@ func (s *GrpcServer) Download(ctx context.Context, request *proto_downloader.Dow
 	for _, t := range s.t.TorrentClient.Torrents() {
 		t.AllowDataDownload()
 		t.AllowDataUpload()
-		t.DownloadAll()
+		if !t.Complete.Bool() {
+			t.DownloadAll()
+		}
 	}
 	return &emptypb.Empty{}, nil
 }
@@ -97,6 +101,8 @@ func (s *GrpcServer) Stats(ctx context.Context, request *proto_downloader.StatsR
 			reply.BytesCompleted += uint64(t.BytesCompleted())
 			reply.BytesTotal += uint64(t.Info().TotalLength())
 			reply.Completed = reply.Completed && t.Complete.Bool()
+			reply.Connections += uint64(len(t.PeerConns()))
+
 			for _, peer := range t.PeerConns() {
 				peers[peer.PeerID] = struct{}{}
 			}

diff --git a/cmd/downloader/downloader/torrentcfg/torrentcfg.go b/cmd/downloader/downloader/torrentcfg/torrentcfg.go
@@ -37,9 +37,9 @@ func Default() *torrent.ClientConfig {
 	//torrentConfig.DisableWebseeds = true
 
 	// Increase default timeouts, because we often run on commodity networks
-	torrentConfig.MinDialTimeout = 6 * time.Second      // default: 3sec
-	torrentConfig.NominalDialTimeout = 20 * time.Second // default: 20sec
-	torrentConfig.HandshakesTimeout = 8 * time.Second   // default: 4sec
+	torrentConfig.MinDialTimeout = 1 * time.Second      // default: 3sec
+	torrentConfig.NominalDialTimeout = 10 * time.Second // default: 20sec
+	torrentConfig.HandshakesTimeout = 1 * time.Second   // default: 4sec
 
 	return torrentConfig
 }
@@ -49,9 +49,9 @@ func New(snapshotsDir *dir.Rw, verbosity lg.Level, natif nat.Interface, download
 	// We would-like to reduce amount of goroutines in Erigon, so reducing next params
 	torrentConfig.EstablishedConnsPerTorrent = connsPerFile // default: 50
 	torrentConfig.TorrentPeersHighWater = maxPeers          // default: 500
-	torrentConfig.TorrentPeersLowWater = 5                  // default: 50
-	torrentConfig.HalfOpenConnsPerTorrent = 5               // default: 25
-	torrentConfig.TotalHalfOpenConns = 100                  // default: 100
+	torrentConfig.TorrentPeersLowWater = 50                 // default: 50
+	torrentConfig.HalfOpenConnsPerTorrent = 25              // default: 25
+	torrentConfig.TotalHalfOpenConns = 50                   // default: 100
 
 	torrentConfig.ListenPort = port
 	torrentConfig.Seed = true
@@ -78,8 +78,14 @@ func New(snapshotsDir *dir.Rw, verbosity lg.Level, natif nat.Interface, download
 		}
 	}
 	// rates are divided by 2 - I don't know why it works, maybe bug inside torrent lib accounting
-	torrentConfig.UploadRateLimiter = rate.NewLimiter(rate.Limit(uploadRate.Bytes()), 2*DefaultPieceSize)     // default: unlimited
-	torrentConfig.DownloadRateLimiter = rate.NewLimiter(rate.Limit(downloadRate.Bytes()), 2*DefaultPieceSize) // default: unlimited
+	torrentConfig.UploadRateLimiter = rate.NewLimiter(rate.Limit(uploadRate.Bytes()), 2*DefaultPieceSize) // default: unlimited
+	if downloadRate.Bytes() < 500_000_000 {
+		b := int(2 * DefaultPieceSize)
+		if downloadRate.Bytes() > DefaultPieceSize {
+			b = int(2 * downloadRate.Bytes())
+		}
+		torrentConfig.DownloadRateLimiter = rate.NewLimiter(rate.Limit(downloadRate.Bytes()), b) // default: unlimited
+	}
 
 	// debug
 	if lg.Debug == verbosity {

diff --git a/cmd/downloader/downloader/util.go b/cmd/downloader/downloader/util.go
@@ -7,7 +7,6 @@ import (
 	"errors"
 	"fmt"
 	"io"
-	"io/ioutil"
 	"os"
 	"path/filepath"
 	"sync"
@@ -46,7 +45,7 @@ func AllTorrentPaths(dir string) ([]string, error) {
 }
 
 func AllTorrentFiles(dir string) ([]string, error) {
-	files, err := ioutil.ReadDir(dir)
+	files, err := os.ReadDir(dir)
 	if err != nil {
 		return nil, err
 	}
@@ -55,7 +54,11 @@ func AllTorrentFiles(dir string) ([]string, error) {
 		if !snapshotsync.IsCorrectFileName(f.Name()) {
 			continue
 		}
-		if f.Size() == 0 {
+		fileInfo, err := f.Info()
+		if err != nil {
+			return nil, err
+		}
+		if fileInfo.Size() == 0 {
 			continue
 		}
 		if filepath.Ext(f.Name()) != ".torrent" { // filter out only compressed files
@@ -66,7 +69,7 @@ func AllTorrentFiles(dir string) ([]string, error) {
 	return res, nil
 }
 func allSegmentFiles(dir string) ([]string, error) {
-	files, err := ioutil.ReadDir(dir)
+	files, err := os.ReadDir(dir)
 	if err != nil {
 		return nil, err
 	}
@@ -75,7 +78,11 @@ func allSegmentFiles(dir string) ([]string, error) {
 		if !snapshotsync.IsCorrectFileName(f.Name()) {
 			continue
 		}
-		if f.Size() == 0 {
+		fileInfo, err := f.Info()
+		if err != nil {
+			return nil, err
+		}
+		if fileInfo.Size() == 0 {
 			continue
 		}
 		if filepath.Ext(f.Name()) != ".seg" { // filter out only compressed files

diff --git a/cmd/downloader/main.go b/cmd/downloader/main.go
@@ -3,13 +3,11 @@ package main
 import (
 	"context"
 	"fmt"
-	"io/ioutil"
 	"net"
 	"os"
 	"path/filepath"
 	"time"
 
-	lg "github.com/anacrolix/log"
 	"github.com/anacrolix/torrent/metainfo"
 	"github.com/c2h5oh/datasize"
 	grpc_middleware "github.com/grpc-ecosystem/go-grpc-middleware"
@@ -57,14 +55,14 @@ func init() {
 
 	withDataDir(rootCmd)
 
-	rootCmd.Flags().StringVar(&natSetting, "nat", "", utils.NATFlag.Usage)
+	rootCmd.Flags().StringVar(&natSetting, "nat", utils.NATFlag.Value, utils.NATFlag.Usage)
 	rootCmd.Flags().StringVar(&downloaderApiAddr, "downloader.api.addr", "127.0.0.1:9093", "external downloader api network address, for example: 127.0.0.1:9093 serves remote downloader interface")
-	rootCmd.Flags().StringVar(&torrentVerbosity, "torrent.verbosity", lg.Warning.LogString(), "DEBUG | INFO | WARN | ERROR")
-	rootCmd.Flags().StringVar(&downloadRateStr, "torrent.download.rate", "8mb", "bytes per second, example: 32mb")
-	rootCmd.Flags().StringVar(&uploadRateStr, "torrent.upload.rate", "4mb", "bytes per second, example: 32mb")
-	rootCmd.Flags().IntVar(&torrentPort, "torrent.port", 42069, "port to listen and serve BitTorrent protocol")
-	rootCmd.Flags().IntVar(&torrentMaxPeers, "torrent.maxpeers", 10, "")
-	rootCmd.Flags().IntVar(&torrentConnsPerFile, "torrent.conns.perfile", 5, "connections per file")
+	rootCmd.Flags().StringVar(&torrentVerbosity, "torrent.verbosity", utils.TorrentVerbosityFlag.Value, utils.TorrentVerbosityFlag.Usage)
+	rootCmd.Flags().StringVar(&downloadRateStr, "torrent.download.rate", utils.TorrentDownloadRateFlag.Value, utils.TorrentDownloadRateFlag.Usage)
+	rootCmd.Flags().StringVar(&uploadRateStr, "torrent.upload.rate", utils.TorrentUploadRateFlag.Value, utils.TorrentUploadRateFlag.Usage)
+	rootCmd.Flags().IntVar(&torrentPort, "torrent.port", utils.TorrentPortFlag.Value, utils.TorrentPortFlag.Usage)
+	rootCmd.Flags().IntVar(&torrentMaxPeers, "torrent.maxpeers", utils.TorrentMaxPeersFlag.Value, utils.TorrentMaxPeersFlag.Usage)
+	rootCmd.Flags().IntVar(&torrentConnsPerFile, "torrent.conns.perfile", utils.TorrentConnsPerFileFlag.Value, utils.TorrentConnsPerFileFlag.Usage)
 
 	withDataDir(printTorrentHashes)
 	printTorrentHashes.PersistentFlags().BoolVar(&forceRebuild, "rebuild", false, "Force re-create .torrent files")
@@ -239,7 +237,7 @@ var printTorrentHashes = &cobra.Command{
 			return nil
 		}
 
-		oldContent, err := ioutil.ReadFile(targetFile)
+		oldContent, err := os.ReadFile(targetFile)
 		if err != nil {
 			return err
 		}
@@ -251,7 +249,7 @@ var printTorrentHashes = &cobra.Command{
 			log.Info("amount of lines in target file is equal or greater than amount of lines in snapshot dir", "old", len(oldLines), "new", len(res))
 			return nil
 		}
-		if err := ioutil.WriteFile(targetFile, serialized, 0644); err != nil {
+		if err := os.WriteFile(targetFile, serialized, 0644); err != nil {
 			return err
 		}
 		return nil

diff --git a/cmd/evm/compiler.go b/cmd/evm/compiler.go
@@ -19,7 +19,7 @@ package main
 import (
 	"errors"
 	"fmt"
-	"io/ioutil"
+	"os"
 
 	"github.com/ledgerwatch/erigon/cmd/evm/internal/compiler"
 
@@ -41,7 +41,7 @@ func compileCmd(ctx *cli.Context) error {
 	}
 
 	fn := ctx.Args().First()
-	src, err := ioutil.ReadFile(fn)
+	src, err := os.ReadFile(fn)
 	if err != nil {
 		return err
 	}

diff --git a/cmd/evm/disasm.go b/cmd/evm/disasm.go
@@ -19,7 +19,7 @@ package main
 import (
 	"errors"
 	"fmt"
-	"io/ioutil"
+	"os"
 	"strings"
 
 	"github.com/ledgerwatch/erigon/core/asm"
@@ -38,7 +38,7 @@ func disasmCmd(ctx *cli.Context) error {
 	switch {
 	case len(ctx.Args().First()) > 0:
 		fn := ctx.Args().First()
-		input, err := ioutil.ReadFile(fn)
+		input, err := os.ReadFile(fn)
 		if err != nil {
 			return err
 		}