Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

panic: invalid page type: 0: 4 #2543

Closed
rfikki opened this issue May 8, 2019 · 13 comments
Closed

panic: invalid page type: 0: 4 #2543

rfikki opened this issue May 8, 2019 · 13 comments
Labels
Bug Something isn't working

Comments

@rfikki
Copy link

rfikki commented May 8, 2019

https://gist.github.com/rfikki/63acd4067c18ff909fdc5e2392366be5
@SjonHortensius
Copy link
Contributor

SjonHortensius commented May 10, 2019 •
edited
Loading

I think you should have posted the complete logs here directly - however these invalid page type errors seem to happen so deep in the boltdb code that I actually suspect some form of (hardware) corruption - I take it you're not using ECC? Your first message seems to imply there isn't even a data-directory, can you confirm this?

Doesn't clearing the data-dir resolve this issue?

@rfikki
Copy link
Author

rfikki commented May 10, 2019

So, it was mentioned that this could be a permissions issue. So, I added "sudo" to the command line and seem to be getting some what further, but error persists: see below:

rfikki@LAPTOP-ECKKGQ2B:~/prysm$ sudo bazel run //beacon-chain -- --clear-db --datadir=/home/rfikki/prysm/prysm-data
[sudo] password for rfikki:
Extracting Bazel installation...
Starting local Bazel server and connecting to it...
DEBUG: Rule 'com_google_protobuf' indicated that a canonical reproducible form can be obtained by modifying arguments shallow_since = "1551387314 -0800"
INFO: Analyzed target //beacon-chain:beacon-chain (536 packages loaded, 15980 targets configured).
INFO: Found 1 target...
INFO: From Generating Descriptor Set proto_library @com_google_protobuf//:wrappers_proto:
external/com_google_protobuf: warning: directory does not exist.
INFO: From Generating Descriptor Set proto_library @com_google_protobuf//:descriptor_proto:
external/com_google_protobuf: warning: directory does not exist.
INFO: From Generating Descriptor Set proto_library @com_google_protobuf//:struct_proto:
external/com_google_protobuf: warning: directory does not exist.
INFO: From Generating Descriptor Set proto_library @com_google_protobuf//:compiler_plugin_proto:
external/com_google_protobuf: warning: directory does not exist.
INFO: From Generating Descriptor Set proto_library @com_google_protobuf//:field_mask_proto:
external/com_google_protobuf: warning: directory does not exist.
INFO: From Generating Descriptor Set proto_library @com_google_protobuf//:source_context_proto:
external/com_google_protobuf: warning: directory does not exist.
INFO: From Generating Descriptor Set proto_library @com_google_protobuf//:any_proto:
external/com_google_protobuf: warning: directory does not exist.
INFO: From Generating Descriptor Set proto_library @com_google_protobuf//:timestamp_proto:
external/com_google_protobuf: warning: directory does not exist.
INFO: From Generating Descriptor Set proto_library @com_google_protobuf//:empty_proto:
external/com_google_protobuf: warning: directory does not exist.
INFO: From Generating Descriptor Set proto_library @com_google_protobuf//:duration_proto:
external/com_google_protobuf: warning: directory does not exist.
INFO: From Generating Descriptor Set proto_library //proto/beacon/p2p/v1:v1_proto:
external/com_google_protobuf: warning: directory does not exist.
INFO: From Generating Descriptor Set proto_library @com_github_libp2p_go_libp2p_autonat//pb:autonat_pb_proto:
[libprotobuf WARNING external/com_google_protobuf/src/google/protobuf/compiler/parser.cc:564] No syntax specified for the proto file: pb/autonat.proto. Please use 'syntax = "proto2";' or 'syntax = "proto3";' to specify a syntax version. (Defaulted to proto2 syntax.)
INFO: From Generating Descriptor Set proto_library @com_google_protobuf//:type_proto:
external/com_google_protobuf: warning: directory does not exist.
INFO: From Generating Descriptor Set proto_library //proto/sharding/p2p/v1:v1_proto:
external/com_google_protobuf: warning: directory does not exist.
INFO: From Generating Descriptor Set proto_library @go_googleapis//google/rpc:status_proto:
external/com_google_protobuf: warning: directory does not exist.
INFO: From Generating Descriptor Set proto_library @com_google_protobuf//:api_proto:
external/com_google_protobuf: warning: directory does not exist.
INFO: From Generating Descriptor Set proto_library //proto/beacon/rpc/v1:v1_proto:
external/com_google_protobuf: warning: directory does not exist.
Target //beacon-chain:beacon-chain up-to-date:
bazel-bin/beacon-chain/linux_amd64_stripped/beacon-chain
INFO: Elapsed time: 874.733s, Critical Path: 111.27s
INFO: 1961 processes: 1961 processwrapper-sandbox.
INFO: Build completed successfully, 1992 total actions
INFO: Build completed successfully, 1992 total actions
[2019-05-10 15:20:06] INFO node: Using custom parameter configuration
[2019-05-10 15:20:06] INFO node: Checking db path=/home/rfikki/prysm/prysm-data/beaconchaindata
[2019-05-10 15:20:06] INFO node: Fetching testnet cluster address endpoint=https://beta.prylabs.net/contract
panic: invalid page type: 0: 4

goroutine 1 [running]:
github.com/boltdb/bolt.(*Cursor).search(0xc000371670, 0x20121d8, 0xa, 0xa, 0x4)
external/com_github_boltdb_bolt/cursor.go:256 +0x354
github.com/boltdb/bolt.(*Cursor).seek(0xc000371670, 0x20121d8, 0xa, 0xa, 0x211de40, 0xc0002e5e50, 0x2, 0x2, 0x2, 0x2, ...)
external/com_github_boltdb_bolt/cursor.go:159 +0x7e
github.com/boltdb/bolt.(*Bucket).Bucket(0xc000342558, 0x20121d8, 0xa, 0xa, 0x1487fa0)
external/com_github_boltdb_bolt/bucket.go:112 +0xd5
github.com/boltdb/bolt.(*Tx).Bucket(...)
external/com_github_boltdb_bolt/tx.go:101
github.com/prysmaticlabs/prysm/beacon-chain/db.(*BeaconDB).VerifyContractAddress.func1(0xc000342540, 0x1484520, 0xc000342540)
beacon-chain/db/verify_contract.go:24 +0x73
github.com/boltdb/bolt.(*DB).Update(0xc0002223c0, 0xc0001d7a80, 0x0, 0x0)
external/com_github_boltdb_bolt/db.go:598 +0x90
github.com/prysmaticlabs/prysm/beacon-chain/db.(*BeaconDB).update(...)
beacon-chain/db/db.go:52
github.com/prysmaticlabs/prysm/beacon-chain/db.(*BeaconDB).VerifyContractAddress(0xc0001c23c0, 0x160bae0, 0xc000042100, 0xf8c541a3f3d806ab, 0xae7a47a38dd9d5a8, 0x742dfcab, 0x0, 0x0)
beacon-chain/db/verify_contract.go:23 +0xd9
github.com/prysmaticlabs/prysm/beacon-chain/node.(*BeaconNode).registerPOWChainService(0xc0005af280, 0xc000278000, 0x0, 0x0)
beacon-chain/node/node.go:289 +0x5f4
github.com/prysmaticlabs/prysm/beacon-chain/node.NewBeaconNode(0xc000278000, 0x4, 0x4, 0x0)
beacon-chain/node/node.go:88 +0x278
main.startNode(0xc000278000, 0x0, 0x0)
beacon-chain/main.go:27 +0x8c
github.com/urfave/cli.HandleAction(0x11165c0, 0x14875f0, 0xc000278000, 0xc0001b28a0, 0x0)
external/com_github_urfave_cli/app.go:490 +0xc8
github.com/urfave/cli.(*App).Run(0xc0001929c0, 0xc00003c090, 0x3, 0x3, 0x0, 0x0)
external/com_github_urfave_cli/app.go:264 +0x590
main.main()
beacon-chain/main.go:83 +0xa6d

@prestonvanloon
Copy link
Member

Maybe related:
microsoft/WSL#3162
https://github.com/odeke-em/drive/wiki/Boltdb-breaks-drive-with-(panic:-invalid-page-type:)-or-(panic:--above-high-water-mark)

@prestonvanloon
Copy link
Member

Just to be totally sure can you try this?

bazel build //beacon-chain 
bazel-bin/beacon-chain/linux_amd64_stripped/beacon-chain --clear-db --datadir=/tmp/prysm-data

If that doesn't work, try running the second command with sudo. I'm not entirely sure how this error occurs so consistently. :(

@rfikki
Copy link
Author

rfikki commented May 11, 2019 •
edited
Loading

rfikki@LAPTOP-ECKKGQ2B:~/prysm$ bazel build //beacon-chain
Starting local Bazel server and connecting to it...
INFO: Analyzed target //beacon-chain:beacon-chain (536 packages loaded, 15979 targets configured).
INFO: Found 1 target...
Target //beacon-chain:beacon-chain up-to-date:
bazel-bin/beacon-chain/linux_amd64_stripped/beacon-chain
INFO: Elapsed time: 31.613s, Critical Path: 1.59s
INFO: 0 processes.
INFO: Build completed successfully, 1 total action

then second command resulted in:

rfikki@LAPTOP-ECKKGQ2B:~/prysm$ bazel-bin/beacon-chain/linux_amd64_stripped/beacon-chain --clear-db --datadir=/tmp/prysm-data
[2019-05-11 14:02:53] INFO node: Using custom parameter configuration
[2019-05-11 14:02:53] INFO node: Checking db path=/tmp/prysm-data/beaconchaindata
[2019-05-11 14:02:53] INFO node: Fetching testnet cluster address endpoint=https://beta.prylabs.net/contract
panic: invalid page type: 0: 4

goroutine 1 [running]:
github.com/boltdb/bolt.(*Cursor).search(0xc000775670, 0x20121d8, 0xa, 0xa, 0x4)
external/com_github_boltdb_bolt/cursor.go:256 +0x354
github.com/boltdb/bolt.(*Cursor).seek(0xc000775670, 0x20121d8, 0xa, 0xa, 0x211de40, 0xc00035e620, 0x2, 0x2, 0x2, 0x2, ...)
external/com_github_boltdb_bolt/cursor.go:159 +0x7e
github.com/boltdb/bolt.(*Bucket).Bucket(0xc000337278, 0x20121d8, 0xa, 0xa, 0x1487fa0)
external/com_github_boltdb_bolt/bucket.go:112 +0xd5
github.com/boltdb/bolt.(*Tx).Bucket(...)
external/com_github_boltdb_bolt/tx.go:101
github.com/prysmaticlabs/prysm/beacon-chain/db.(*BeaconDB).VerifyContractAddress.func1(0xc000337260, 0x1484520, 0xc000337260)
beacon-chain/db/verify_contract.go:24 +0x73
github.com/boltdb/bolt.(*DB).Update(0xc0002183c0, 0xc0005c2440, 0x0, 0x0)
external/com_github_boltdb_bolt/db.go:598 +0x90
github.com/prysmaticlabs/prysm/beacon-chain/db.(*BeaconDB).update(...)
beacon-chain/db/db.go:52
github.com/prysmaticlabs/prysm/beacon-chain/db.(*BeaconDB).VerifyContractAddress(0xc0001bc3c0, 0x160bae0, 0xc0000440f8, 0xf8c541a3f3d806ab, 0xae7a47a38dd9d5a8, 0x742dfcab, 0x0, 0x0)
beacon-chain/db/verify_contract.go:23 +0xd9
github.com/prysmaticlabs/prysm/beacon-chain/node.(*BeaconNode).registerPOWChainService(0xc0004e10c0, 0xc000198000, 0x0, 0x0)
beacon-chain/node/node.go:289 +0x5f4
github.com/prysmaticlabs/prysm/beacon-chain/node.NewBeaconNode(0xc000198000, 0x4, 0x4, 0x0)
beacon-chain/node/node.go:88 +0x278
main.startNode(0xc000198000, 0x0, 0x0)
beacon-chain/main.go:27 +0x8c
github.com/urfave/cli.HandleAction(0x11165c0, 0x14875f0, 0xc000198000, 0xc00057e060, 0x0)
external/com_github_urfave_cli/app.go:490 +0xc8
github.com/urfave/cli.(*App).Run(0xc00017c000, 0xc00003c090, 0x3, 0x3, 0x0, 0x0)
external/com_github_urfave_cli/app.go:264 +0x590
main.main()
beacon-chain/main.go:83 +0xa6d

Using the sudo command gave the same error................

rfikki@LAPTOP-ECKKGQ2B:~/prysm$ sudo bazel-bin/beacon-chain/linux_amd64_stripped/beacon-chain --clear-db --datadir=/tmp/prysm-data
[sudo] password for rfikki:
[2019-05-11 14:03:12] INFO node: Using custom parameter configuration
[2019-05-11 14:03:13] INFO node: Checking db path=/tmp/prysm-data/beaconchaindata
[2019-05-11 14:03:13] INFO node: Fetching testnet cluster address endpoint=https://beta.prylabs.net/contract
panic: invalid page type: 0: 4

goroutine 1 [running]:
github.com/boltdb/bolt.(*Cursor).search(0xc000757670, 0x20121d8, 0xa, 0xa, 0x4)
external/com_github_boltdb_bolt/cursor.go:256 +0x354
github.com/boltdb/bolt.(*Cursor).seek(0xc000757670, 0x20121d8, 0xa, 0xa, 0x211de40, 0xc000599230, 0x2, 0x2, 0x2, 0x2, ...)
external/com_github_boltdb_bolt/cursor.go:159 +0x7e
github.com/boltdb/bolt.(*Bucket).Bucket(0xc00003e1d8, 0x20121d8, 0xa, 0xa, 0x1487fa0)
external/com_github_boltdb_bolt/bucket.go:112 +0xd5
github.com/boltdb/bolt.(*Tx).Bucket(...)
external/com_github_boltdb_bolt/tx.go:101
github.com/prysmaticlabs/prysm/beacon-chain/db.(*BeaconDB).VerifyContractAddress.func1(0xc00003e1c0, 0x1484520, 0xc00003e1c0)
beacon-chain/db/verify_contract.go:24 +0x73
github.com/boltdb/bolt.(*DB).Update(0xc00000c1e0, 0xc0002797e0, 0x0, 0x0)
external/com_github_boltdb_bolt/db.go:598 +0x90
github.com/prysmaticlabs/prysm/beacon-chain/db.(*BeaconDB).update(...)
beacon-chain/db/db.go:52
github.com/prysmaticlabs/prysm/beacon-chain/db.(*BeaconDB).VerifyContractAddress(0xc0001148c0, 0x160bae0, 0xc0000c2068, 0xf8c541a3f3d806ab, 0xae7a47a38dd9d5a8, 0x742dfcab, 0x0, 0x0)
beacon-chain/db/verify_contract.go:23 +0xd9
github.com/prysmaticlabs/prysm/beacon-chain/node.(*BeaconNode).registerPOWChainService(0xc0000cba00, 0xc000398000, 0x0, 0x0)
beacon-chain/node/node.go:289 +0x5f4
github.com/prysmaticlabs/prysm/beacon-chain/node.NewBeaconNode(0xc000398000, 0x4, 0x4, 0x0)
beacon-chain/node/node.go:88 +0x278
main.startNode(0xc000398000, 0x0, 0x0)
beacon-chain/main.go:27 +0x8c
github.com/urfave/cli.HandleAction(0x11165c0, 0x14875f0, 0xc000398000, 0xc0000c89c0, 0x0)
external/com_github_urfave_cli/app.go:490 +0xc8
github.com/urfave/cli.(*App).Run(0xc00022a340, 0xc0000ba060, 0x3, 0x3, 0x0, 0x0)
external/com_github_urfave_cli/app.go:264 +0x590
main.main()
beacon-chain/main.go:83 +0xa6d

@SjonHortensius
Copy link
Contributor

@rfikki can you clarify which OS and FS you use, is it also zfs (or maybe even WSL)? I've been trying to reproduce this but a zfs mount on a recent Archlinux machine does not show this - and boltdbs tests also run fine on it.

Given the related reports I'm pretty sure this is a go-specific issue with regards to filesystem interaction that might be worked around - but is not caused by anything prysm specific

@prestonvanloon prestonvanloon added the Bug Something isn't working label May 12, 2019
@rfikki
Copy link
Author

rfikki commented May 13, 2019

Windows Subsystem for Linux (WSL) is my OS setup.

@rfikki
Copy link
Author

rfikki commented May 14, 2019

I wonder if this is somehow related to these issues:
#2598

@SjonHortensius
Copy link
Contributor

SjonHortensius commented May 16, 2019 •
edited
Loading

I have reproduced the testcase in microsoft/WSL#3162 and apparently this was a WSL specific issue that has been fixed in microsoft/WSL#3939 and also listed under https://docs.microsoft.com/en-us/windows/wsl/release-notes#build-18890 as

  • Extending mmap’d files does not work properly [GH 3939]

I think @rfikki should retry this with a newer (at least 18890) WSL build, checkout the "Windows Insider Program" settings to download bleeding edge releases.

Also - I think in the future mentioning up-front that you're using WSL is pretty significant and useful for people attempting to reproduce your issue :)

@rfikki
Copy link
Author

rfikki commented May 18, 2019

I agree, that it would have been helpful to mention WSL. However it might be easier to know what is pertinent to list/provide when submitting an issue. See the template provided by the Geth team here when submitting an issue. https://github.com/ethereum/go-ethereum/issues/new

@rfikki
Copy link
Author

rfikki commented May 18, 2019

Thanks for all the assistance provided. I will look into the newer builds after 18890 which have not yet been made part of an official release.

@rfikki
Copy link
Author

rfikki commented May 19, 2019

Can confirm that the new build works for me. Updated to 18898.

@SjonHortensius
Copy link
Contributor

@rfikki can you close this issue as it is resolved - and wasn't caused by prysm? This issue has no use remaining open

@rfikki rfikki closed this as completed May 22, 2019
@nisdas nisdas mentioned this issue Aug 6, 2020
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@rfikki @SjonHortensius @prestonvanloon