Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

IPFS 0.4.5 datastore migration 4 -> 5 failure. #3681

Closed
NeoTeo opened this issue Feb 12, 2017 · 28 comments
Closed

IPFS 0.4.5 datastore migration 4 -> 5 failure. #3681

NeoTeo opened this issue Feb 12, 2017 · 28 comments

Comments

@NeoTeo
Copy link

NeoTeo commented Feb 12, 2017

Version information:

go-ipfs version: 0.4.5-
Repo version: 5
System version: amd64/darwin
Golang version: go1.7.4

Type:

Bug

Priority:

P0

Description:

Running ipfs daemon automatically initiates the datastore migration.
The migration fails with the following log output:

teo@muthr-2 ~/D/go-ipfs> ipfs daemon
Initializing daemon...
Adjusting current ulimit to 1024...
Successfully raised file descriptor limit to 1024.
Found outdated fs-repo, migrations need to be run.
Run migrations now? [y/N] y 
  => Looking for suitable fs-repo-migrations binary.
  => None found, downloading.
  => Running: /var/folders/qx/gcnhmzf529gdkndv2wbz57n00000gn/T/go-ipfs-migrate108616551/fs-repo-migrations -to 5 -y
Found fs-repo version 4 at /Users/teo/.ipfs
===> Running migration 4 to 5...
applying 4-to-5 repo migration
locking repo at "/Users/teo/.ipfs"
  - verifying version is '4'
> Upgrading datastore format to have sharding specification file
> creating a new flatfs datastore with new format
> converting current flatfs datastore to new format
Moving Keys...
1580 keys so farfailed to decode entry in flatfs
4400 keys so farfailed to decode entry in flatfs
5320 keys so farfailed to decode entry in flatfs
71030 keys so far
Cleaning Up...
found unexpected file in datastore directory: ".DS_Store", moving anywayERROR: failed to convert flatfs datastore: remove /Users/teo/.ipfs/blocks-v4/CIQAG: directory not empty
attempting to revert...
Moving Keys...
71030 keys so far
Cleaning Up...
found unexpected file in datastore directory: ".DS_Store", moving anywayAll Done.
ipfs migration:  migration 4 to 5 failed: remove /Users/teo/.ipfs/blocks-v4/CIQAG: directory not empty
  => Failed: /var/folders/qx/gcnhmzf529gdkndv2wbz57n00000gn/T/go-ipfs-migrate108616551/fs-repo-migrations -to 5 -y
The migrations of fs-repo failed:
  migration failed: exit status 1
If you think this is a bug, please file an issue and include this whole log output.
  https://github.com/ipfs/fs-repo-migrations
Error: migration failed: exit status 1
@Kubuxu
Copy link
Member

Kubuxu commented Feb 12, 2017

Please remove the .DS_Store from the directory, we probably can add an exception for Mac in this case.

@NeoTeo
Copy link
Author

NeoTeo commented Feb 12, 2017

Thanks for the quick answer @Kubuxu but the issue remains:

teo@muthr-2 ~/D/go-ipfs> ipfs daemon
Initializing daemon...
Adjusting current ulimit to 1024...
Successfully raised file descriptor limit to 1024.
Found outdated fs-repo, migrations need to be run.
Run migrations now? [y/N] y
  => Looking for suitable fs-repo-migrations binary.
  => None found, downloading.
  => Running: /var/folders/qx/gcnhmzf529gdkndv2wbz57n00000gn/T/go-ipfs-migrate288610175/fs-repo-migrations -to 5 -y
Found fs-repo version 4 at /Users/teo/.ipfs
===> Running migration 4 to 5...
applying 4-to-5 repo migration
locking repo at "/Users/teo/.ipfs"
  - verifying version is '4'
> Upgrading datastore format to have sharding specification file
> creating a new flatfs datastore with new format
> converting current flatfs datastore to new format
Moving Keys...
1580 keys so farfailed to decode entry in flatfs
4400 keys so farfailed to decode entry in flatfs
5320 keys so farfailed to decode entry in flatfs
71030 keys so far
Cleaning Up...
ERROR: failed to convert flatfs datastore: remove /Users/teo/.ipfs/blocks-v4/CIQAG: directory not empty
attempting to revert...
Moving Keys...
71030 keys so far
Cleaning Up...
All Done.
ipfs migration:  migration 4 to 5 failed: remove /Users/teo/.ipfs/blocks-v4/CIQAG: directory not empty
  => Failed: /var/folders/qx/gcnhmzf529gdkndv2wbz57n00000gn/T/go-ipfs-migrate288610175/fs-repo-migrations -to 5 -y
The migrations of fs-repo failed:
  migration failed: exit status 1
If you think this is a bug, please file an issue and include this whole log output.
  https://github.com/ipfs/fs-repo-migrations
Error: migration failed: exit status 1

@Kubuxu
Copy link
Member

Kubuxu commented Feb 12, 2017

@kevina can you take a look at it?

@NeoTeo
Copy link
Author

NeoTeo commented Feb 12, 2017

The contents of the directory it complains about seem reasonable:

https://gist.github.com/NeoTeo/138bccbddb02b9cc2f12583ab9b9cf32

@Kubuxu
Copy link
Member

Kubuxu commented Feb 12, 2017

It probably complains about: put-675353937, which is left over from atomic put operation that didn't go through.

@NeoTeo
Copy link
Author

NeoTeo commented Feb 12, 2017

Is there any way of telling what it belongs to and whether it's safe to remove?

@Kubuxu
Copy link
Member

Kubuxu commented Feb 12, 2017

Files with long string of capital letters and extension .data are good. Rest in blocks directory is not important.

ipfs repo fsck should be improved to remove those leftovers

@NeoTeo
Copy link
Author

NeoTeo commented Feb 12, 2017

Ok, that did it. There were three of those put- files, as hinted at by this not-as-useful-as-could-be message:

Moving Keys...
1580 keys so farfailed to decode entry in flatfs
4400 keys so farfailed to decode entry in flatfs
5320 keys so farfailed to decode entry in flatfs

Moving those and rerunning the daemon & migration worked.
:) Thanks for your help @Kubuxu

@whyrusleeping
Copy link
Member

Yeah, ipfs repo fsck should be improved to clean these up. The migrations code should be able to handle these as well.

@kevina
Copy link
Contributor

kevina commented Feb 12, 2017

found unexpected file in datastore directory: ".DS_Store", moving anyway

Note, this is just a warning. The file was moved. There is a formatting problem with the error message though.

@Kubuxu
Copy link
Member

Kubuxu commented Feb 12, 2017

It is mostly connected with this: ipfs/go-ds-flatfs#16

@whyrusleeping
Copy link
Member

Hey all, mind verifying that the new binaries for fs-repo-migrations 1.2.1 here fix the issue? https://ipfs.io/ipfs/QmZMzC8ihgybDLzstWm9iYTEbrhVEeBtevWE6vLjdci3Ay/#fs-repo-migrations

@NeoTeo
Copy link
Author

NeoTeo commented Feb 17, 2017

My original repo is now converted so I can't check on that. I have an old repo on a separate machine that doesn't have the offending files in it so it'd probably convert just fine. Would making any bogus file in there cause the issue and would a test with that be useful?

@Kubuxu
Copy link
Member

Kubuxu commented Feb 17, 2017

You can try to make go-ipfs create file like that by force killing it (double Ctrl+C) while adding some files.

@NeoTeo
Copy link
Author

NeoTeo commented Feb 17, 2017

Nope, that doesn't do it.

@NeoTeo
Copy link
Author

NeoTeo commented Feb 17, 2017

Ok,Managed it by killing the daemon whilst adding. Will test.

@NeoTeo
Copy link
Author

NeoTeo commented Feb 17, 2017

Failed. The output:

teo@Vger-6 ~/D/fs-repo-migrations> ./fs-repo-migrations 
Found fs-repo version 4 at /Users/teo/.ipfs
Do you want to upgrade this to version 5? [y/n] y
===> Running migration 4 to 5...
applying 4-to-5 repo migration
locking repo at "/Users/teo/.ipfs"
  - verifying version is '4'
> Upgrading datastore format to have sharding specification file
> creating a new flatfs datastore with new format
> converting current flatfs datastore to new format
Moving Keys...
630 keys so far2017/02/17 13:52:56 failed to decode entry in flatfs:  put-936201373
1370 keys so far
Cleaning Up...
ERROR: failed to convert flatfs datastore: open /Users/teo/.ipfs/blocks-v4/CIQIF: too many open files
attempting to revert...
Moving Keys...
1370 keys so far
Cleaning Up...
ERROR: reverting flatfs conversion failed: open /Users/teo/.ipfs/blocks-v5/EB: too many open files
ERROR: Please file a bug report at https://github.com/ipfs/fs-repo-migrations
ipfs migration:  migration 4 to 5 failed: open /Users/teo/.ipfs/blocks-v5/EB: too many open files

@Kubuxu
Copy link
Member

Kubuxu commented Feb 17, 2017

Now it looks like something else (too many open files).

@Kubuxu
Copy link
Member

Kubuxu commented Feb 17, 2017

Ahh, I see. We are using deffer calls to Close, but those functions run for a long time which will cause a failure. This should be quite easy to fix by doing manual close calls.

@kevina will you have time to fix it?

@kevina
Copy link
Contributor

kevina commented Feb 17, 2017

@Kubuxu yes, sorry I missed the message, part of of been my fought for misunderstanding how defer works in go (I assumed it would execute at the end of the scope (like C++ would) not the end of the function).

Expect something in a few hours.

@kevina
Copy link
Contributor

kevina commented Feb 17, 2017

@Kubuxu okay, I just created a p.r. that should help. ipfs/go-ds-flatfs#18.

This will need to be bubbled up once merged.

@NeoTeo
Copy link
Author

NeoTeo commented Feb 18, 2017

Getting a lot of these. Related?

teo@muthr-2 ~> ipfs daemon
Initializing daemon...
Adjusting current ulimit to 1024...
Successfully raised file descriptor limit to 1024.
Swarm listening on /ip4/100.100.160.222/tcp/31622
Swarm listening on /ip4/127.0.0.1/tcp/4001
Swarm listening on /ip4/192.168.5.14/tcp/4001
Swarm listening on /ip4/192.168.5.8/tcp/4001
Swarm listening on /ip4/89.23.224.58/tcp/40556
Swarm listening on /ip6/::1/tcp/4001
Swarm listening on /ip6/fd00:e95b:cea7:bd1e:291e:a867:7813:4598/tcp/4001
API server listening on /ip4/127.0.0.1/tcp/5001
Gateway (readonly) server listening on /ip4/127.0.0.1/tcp/8080
Daemon is ready
15:41:16.538 ERROR  providers: deleting provider set: open /Users/teo/.ipfs/datastore/020302.log: too many open files providers.go:209
15:41:16.540 ERROR  providers: deleting provider set: open /Users/teo/.ipfs/datastore/020302.log: too many open files providers.go:209
15:41:16.540 ERROR  providers: deleting provider set: open /Users/teo/.ipfs/datastore/020302.log: too many open files providers.go:209
15:41:16.540 ERROR  providers: deleting provider set: open /Users/teo/.ipfs/datastore/020302.log: too many open files providers.go:209
15:41:16.540 ERROR  providers: deleting provider set: open /Users/teo/.ipfs/datastore/020302.log: too many open files providers.go:209
15:41:16.541 ERROR  providers: deleting provider set: open /Users/teo/.ipfs/datastore/020302.log: too many open files providers.go:209
15:41:16.541 ERROR  providers: deleting provider set: open /Users/teo/.ipfs/datastore/020302.log: too many open files providers.go:209
15:41:16.542 ERROR  providers: deleting provider set: open /Users/teo/.ipfs/datastore/020302.log: too many open files providers.go:209
15:41:16.542 ERROR  providers: deleting provider set: open /Users/teo/.ipfs/datastore/020302.log: too many open files providers.go:209
15:41:16.542 ERROR  providers: deleting provider set: open /Users/teo/.ipfs/datastore/020302.log: too many open files providers.go:209
15:41:16.543 ERROR  providers: deleting provider set: open /Users/teo/.ipfs/datastore/020302.log: too many open files providers.go:209
15:41:16.543 ERROR  providers: deleting provider set: open /Users/teo/.ipfs/datastore/020302.log: too many open files providers.go:209
15:41:16.544 ERROR  providers: deleting provider set: open /Users/teo/.ipfs/datastore/020302.log: too many open files providers.go:209
15:41:16.544 ERROR  providers: deleting provider set: open /Users/teo/.ipfs/datastore/020302.log: too many open files providers.go:209
15:41:16.544 ERROR  providers: deleting provider set: open /Users/teo/.ipfs/datastore/020302.log: too many open files providers.go:209
15:41:16.544 ERROR  providers: deleting provider set: open /Users/teo/.ipfs/datastore/020302.log: too many open files providers.go:209
15:41:20.913 ERROR       mdns: mdns lookup error: failed to bind to any multicast udp port mdns.go:135

@Kubuxu
Copy link
Member

Kubuxu commented Feb 18, 2017 •
edited
Loading

No, but it is interesting either way. Can you open another issue.

@NeoTeo
Copy link
Author

NeoTeo commented Feb 18, 2017

Opened as issue #3706

@whyrusleeping
Copy link
Member

This should be resolved now (as well as #3706 )

@Kubuxu
Copy link
Member

Kubuxu commented Feb 21, 2017

@lgierth can you test migrations on our gateways?

@Kubuxu
Copy link
Member

Kubuxu commented Mar 15, 2017

I think it was fixed.

@Kubuxu Kubuxu closed this as completed Mar 15, 2017
@d10r
Copy link

d10r commented Mar 21, 2017

I can confirm the fix. Had the same issue as OP with 0.4.5 and was fixed after updating to 0.4.7.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@NeoTeo @whyrusleeping @kevina @Kubuxu @d10r