Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

pinning is broken #1051

Closed
2 tasks
jbenet opened this issue Apr 11, 2015 · 4 comments
Closed
2 tasks

pinning is broken #1051

jbenet opened this issue Apr 11, 2015 · 4 comments
Labels
kind/bug A bug in existing code (including security flaws) topic/commands Topic commands

Comments

@jbenet
Copy link
Member

jbenet commented Apr 11, 2015 

# new repo
> ipfs init
initializing ipfs node at /Users/jbenet/.go-ipfs
generating 4096-bit RSA keypair...done
peer identity: QmeGkJ23fbmyHopembDxFJVZ7gSem7982MVkkV5igjgEPt
to get started, enter:

  ipfs cat /ipfs/Qmcqtw8FfrVSBaRmbWwHxt3AuySBhJLcvmFYi3Lbc4xnwj/readme

> tree dir
dir
├── afile
└── bfile

0 directories, 2 files

# add dir. this should pin the dir recursively.
> ipfs add -r dir
added QmPyJ2nJMQFo7Ee6kk6R7MHvuNEdq1QXfQfv6P5TRPzSco dir/afile
added QmRYTiRs8PZURrGcmvnn4Drfp3WGs5UsankDtVe1K834Ts dir/bfile
added QmeuBj5MygivbTHbCfCrrcCkTGcnQ51N5QANYhXrJrhxXd dir

# no direct. good.
> ipfs pin ls --type=direct

# dir should be in recursive. ... but it isn't?
> ipfs pin ls --type=recursive | grep QmeuBj5MygivbTHbCfCrrcCkTGcnQ51N5QANYhXrJrhxXd
# what's in recursive, then?
> ipfs pin ls --type=recursive
QmTumTjvcYCAvRRwQ8sDRxh8ezmrcr88YFU7iYNroGGTBZ
QmeEqpsKrvdhuuuVsguHaVdJcPnnUHHZ5qEWjCHavYbNqU
QmPyJ2nJMQFo7Ee6kk6R7MHvuNEdq1QXfQfv6P5TRPzSco <---- wat
QmfE3nUohq2nEYwieF7YFnJF1VfiL4i3wDxkMq8aGUg8Mt  
QmRYTiRs8PZURrGcmvnn4Drfp3WGs5UsankDtVe1K834Ts <---- wat
QmUNLLsPACCz1vLxQVkXqqLX5R1X345qqfHbsf67hvA3Nn
QmYCvbfNbCwFR45HiNP45rwJgvatpiW38D961L5qAhUM5Y
Qmcqtw8FfrVSBaRmbWwHxt3AuySBhJLcvmFYi3Lbc4xnwj
QmY5heUM5qgRubMDD1og9fhCPA6QdkMp3QCwd4s7gJsyE7
QmUFtMrBHqdjTtbebsL6YGebvjShh3Jud1insUv12fEVdA
> ipfs pin ls --type=indirect
QmTumTjvcYCAvRRwQ8sDRxh8ezmrcr88YFU7iYNroGGTBZ
QmUFtMrBHqdjTtbebsL6YGebvjShh3Jud1insUv12fEVdA
QmY5heUM5qgRubMDD1og9fhCPA6QdkMp3QCwd4s7gJsyE7
QmYCvbfNbCwFR45HiNP45rwJgvatpiW38D961L5qAhUM5Y
QmeEqpsKrvdhuuuVsguHaVdJcPnnUHHZ5qEWjCHavYbNqU
QmfE3nUohq2nEYwieF7YFnJF1VfiL4i3wDxkMq8aGUg8Mt
> echo "okay. fine, we'll pin it ourselves"
okay. fine, we'll pin it ourselves
# let's clear the current, _incorrect_ pins.
> ipfs pin rm -r QmPyJ2nJMQFo7Ee6kk6R7MHvuNEdq1QXfQfv6P5TRPzSco
unpinned QmPyJ2nJMQFo7Ee6kk6R7MHvuNEdq1QXfQfv6P5TRPzSco
> ipfs pin rm -r QmRYTiRs8PZURrGcmvnn4Drfp3WGs5UsankDtVe1K834Ts
unpinned QmRYTiRs8PZURrGcmvnn4Drfp3WGs5UsankDtVe1K834Ts
# let's add a recursive pin
> ipfs pin add -r QmeuBj5MygivbTHbCfCrrcCkTGcnQ51N5QANYhXrJrhxXd
pinned QmeuBj5MygivbTHbCfCrrcCkTGcnQ51N5QANYhXrJrhxXd recursively
# check it's there.
> ipfs pin ls --type=recursive | grep QmeuBj5MygivbTHbCfCrrcCkTGcnQ51N5QANYhXrJrhxXd
QmeuBj5MygivbTHbCfCrrcCkTGcnQ51N5QANYhXrJrhxXd
# check the children are not in recursive
> ipfs pin ls --type=recursive
QmfE3nUohq2nEYwieF7YFnJF1VfiL4i3wDxkMq8aGUg8Mt
QmTumTjvcYCAvRRwQ8sDRxh8ezmrcr88YFU7iYNroGGTBZ
QmeEqpsKrvdhuuuVsguHaVdJcPnnUHHZ5qEWjCHavYbNqU
QmeuBj5MygivbTHbCfCrrcCkTGcnQ51N5QANYhXrJrhxXd
QmYCvbfNbCwFR45HiNP45rwJgvatpiW38D961L5qAhUM5Y
QmUNLLsPACCz1vLxQVkXqqLX5R1X345qqfHbsf67hvA3Nn
QmY5heUM5qgRubMDD1og9fhCPA6QdkMp3QCwd4s7gJsyE7
QmUFtMrBHqdjTtbebsL6YGebvjShh3Jud1insUv12fEVdA
Qmcqtw8FfrVSBaRmbWwHxt3AuySBhJLcvmFYi3Lbc4xnwj
# check the children are in indirect
> ipfs pin ls --type=indirect
QmeEqpsKrvdhuuuVsguHaVdJcPnnUHHZ5qEWjCHavYbNqU
QmfE3nUohq2nEYwieF7YFnJF1VfiL4i3wDxkMq8aGUg8Mt
QmPyJ2nJMQFo7Ee6kk6R7MHvuNEdq1QXfQfv6P5TRPzSco <---- yep
QmRYTiRs8PZURrGcmvnn4Drfp3WGs5UsankDtVe1K834Ts <---- yep
QmTumTjvcYCAvRRwQ8sDRxh8ezmrcr88YFU7iYNroGGTBZ
QmUFtMrBHqdjTtbebsL6YGebvjShh3Jud1insUv12fEVdA
QmY5heUM5qgRubMDD1og9fhCPA6QdkMp3QCwd4s7gJsyE7
QmYCvbfNbCwFR45HiNP45rwJgvatpiW38D961L5qAhUM5Y
> echo "ok so far so good"
ok so far so good
# remove the recursive pin
> ipfs pin rm -r QmeuBj5MygivbTHbCfCrrcCkTGcnQ51N5QANYhXrJrhxXd
unpinned QmeuBj5MygivbTHbCfCrrcCkTGcnQ51N5QANYhXrJrhxXd
# check it's no longer there.
> ipfs pin ls --type=recursive
QmeEqpsKrvdhuuuVsguHaVdJcPnnUHHZ5qEWjCHavYbNqU
QmfE3nUohq2nEYwieF7YFnJF1VfiL4i3wDxkMq8aGUg8Mt
QmYCvbfNbCwFR45HiNP45rwJgvatpiW38D961L5qAhUM5Y
QmUNLLsPACCz1vLxQVkXqqLX5R1X345qqfHbsf67hvA3Nn
QmUFtMrBHqdjTtbebsL6YGebvjShh3Jud1insUv12fEVdA
QmY5heUM5qgRubMDD1og9fhCPA6QdkMp3QCwd4s7gJsyE7
Qmcqtw8FfrVSBaRmbWwHxt3AuySBhJLcvmFYi3Lbc4xnwj
QmTumTjvcYCAvRRwQ8sDRxh8ezmrcr88YFU7iYNroGGTBZ
# check the children are no longer there...
> ipfs pin ls --type=indirect
QmUFtMrBHqdjTtbebsL6YGebvjShh3Jud1insUv12fEVdA
QmY5heUM5qgRubMDD1og9fhCPA6QdkMp3QCwd4s7gJsyE7
QmYCvbfNbCwFR45HiNP45rwJgvatpiW38D961L5qAhUM5Y
QmeEqpsKrvdhuuuVsguHaVdJcPnnUHHZ5qEWjCHavYbNqU
QmfE3nUohq2nEYwieF7YFnJF1VfiL4i3wDxkMq8aGUg8Mt
QmPyJ2nJMQFo7Ee6kk6R7MHvuNEdq1QXfQfv6P5TRPzSco <---- wait wat!?
QmRYTiRs8PZURrGcmvnn4Drfp3WGs5UsankDtVe1K834Ts <---- wait wat!?
QmTumTjvcYCAvRRwQ8sDRxh8ezmrcr88YFU7iYNroGGTBZ
> ipfs pin ls --type=direct
> ipfs pin add QmeuBj5MygivbTHbCfCrrcCkTGcnQ51N5QANYhXrJrhxXd
pinned QmeuBj5MygivbTHbCfCrrcCkTGcnQ51N5QANYhXrJrhxXd directly
> ipfs pin add QmPyJ2nJMQFo7Ee6kk6R7MHvuNEdq1QXfQfv6P5TRPzSco
pinned QmPyJ2nJMQFo7Ee6kk6R7MHvuNEdq1QXfQfv6P5TRPzSco directly
> ipfs pin ls --type=direct
QmeuBj5MygivbTHbCfCrrcCkTGcnQ51N5QANYhXrJrhxXd
QmPyJ2nJMQFo7Ee6kk6R7MHvuNEdq1QXfQfv6P5TRPzSco
> ipfs pin ls --type=indirect
QmfE3nUohq2nEYwieF7YFnJF1VfiL4i3wDxkMq8aGUg8Mt
QmPyJ2nJMQFo7Ee6kk6R7MHvuNEdq1QXfQfv6P5TRPzSco <---- still wat.
QmRYTiRs8PZURrGcmvnn4Drfp3WGs5UsankDtVe1K834Ts <---- still wat.
QmTumTjvcYCAvRRwQ8sDRxh8ezmrcr88YFU7iYNroGGTBZ
QmUFtMrBHqdjTtbebsL6YGebvjShh3Jud1insUv12fEVdA
QmY5heUM5qgRubMDD1og9fhCPA6QdkMp3QCwd4s7gJsyE7
QmYCvbfNbCwFR45HiNP45rwJgvatpiW38D961L5qAhUM5Y
QmeEqpsKrvdhuuuVsguHaVdJcPnnUHHZ5qEWjCHavYbNqU
> ipfs repo gc
> ipfs pin ls --type=indirect
QmTumTjvcYCAvRRwQ8sDRxh8ezmrcr88YFU7iYNroGGTBZ
QmUFtMrBHqdjTtbebsL6YGebvjShh3Jud1insUv12fEVdA
QmY5heUM5qgRubMDD1og9fhCPA6QdkMp3QCwd4s7gJsyE7
QmYCvbfNbCwFR45HiNP45rwJgvatpiW38D961L5qAhUM5Y
QmeEqpsKrvdhuuuVsguHaVdJcPnnUHHZ5qEWjCHavYbNqU
QmfE3nUohq2nEYwieF7YFnJF1VfiL4i3wDxkMq8aGUg8Mt
QmPyJ2nJMQFo7Ee6kk6R7MHvuNEdq1QXfQfv6P5TRPzSco
QmRYTiRs8PZURrGcmvnn4Drfp3WGs5UsankDtVe1K834Ts
# this file should be gone, but it's magically pinned indirectly
> ipfs cat QmRYTiRs8PZURrGcmvnn4Drfp3WGs5UsankDtVe1K834Ts
some text b  
> ipfs pin rm QmRYTiRs8PZURrGcmvnn4Drfp3WGs5UsankDtVe1K834Ts
Error: QmRYTiRs8PZURrGcmvnn4Drfp3WGs5UsankDtVe1K834Ts is pinned indirectly. indirect pins cannot be removed directly

at least:

  • pinning on add -r does not pin the root directory. it pins the files...
  • unpinning recursively does not unpin indirect pins correctly.
jbenet added a commit that referenced this issue Apr 11, 2015
@jbenet
test/sharness: added more pinning tests
b8c7f26 
Pinning is currently broken. See issue #1051. This commit introduces
a few more pinning tests. These are by no means exhaustive, but
definitely surface the present problems going on. I believe these
tests are correct, but not sure. Pushing them as failing so that
pinning is fixed in this PR.
jbenet added a commit that referenced this issue Apr 11, 2015
@jbenet @whyrusleeping
test/sharness: added more pinning tests
d26a0a2 
Pinning is currently broken. See issue #1051. This commit introduces
a few more pinning tests. These are by no means exhaustive, but
definitely surface the present problems going on. I believe these
tests are correct, but not sure. Pushing them as failing so that
pinning is fixed in this PR.
@whyrusleeping whyrusleeping added kind/bug A bug in existing code (including security flaws) topic/commands Topic commands labels Apr 12, 2015
@jbenet jbenet mentioned this issue Apr 13, 2015
Open
whyrusleeping added a commit that referenced this issue Apr 19, 2015
@whyrusleeping
fix for #1008 and other pinning fixes
3cf18e8 
This commit adds a new set of sharness tests for pinning, and addresses
bugs that were pointed out by said tests.

test/sharness: added more pinning tests

Pinning is currently broken. See issue #1051. This commit introduces
a few more pinning tests. These are by no means exhaustive, but
definitely surface the present problems going on. I believe these
tests are correct, but not sure. Pushing them as failing so that
pinning is fixed in this PR.

make pinning and merkledag.Get take contexts

improve 'add' commands usage of pinning

FIXUP: fix 'pin lists look good'

ipfs-pin-stat simple script to help check pinning

This is a simple shell script to help check pinning.

We ought to strive towards making adding commands this easy.
The http api is great and powerful, but our setup right now
gets in the way. Perhaps we can clean up that area.

updated t0081-repo-pinning

- fixed a couple bugs with the tests
- made it a bit clearer (still a lot going on)
- the remaining tests are correct and highlight a problem with
  pinning. Namely, that recursive pinning is buggy. At least:
  towards the end of the test, $HASH_DIR4 and $HASH_FILE4 should
  be pinned indirectly, but they're not. And thus get gc-ed out.
  There may be other problems too.

cc @whyrusleeping

fix grep params for context deadline check

fix bugs in pin and pin tests

check for block local before checking recursive pin
whyrusleeping added a commit that referenced this issue Apr 20, 2015
@whyrusleeping
fix for #1008 and other pinning fixes
0a6b880 
This commit adds a new set of sharness tests for pinning, and addresses
bugs that were pointed out by said tests.

test/sharness: added more pinning tests

Pinning is currently broken. See issue #1051. This commit introduces
a few more pinning tests. These are by no means exhaustive, but
definitely surface the present problems going on. I believe these
tests are correct, but not sure. Pushing them as failing so that
pinning is fixed in this PR.

make pinning and merkledag.Get take contexts

improve 'add' commands usage of pinning

FIXUP: fix 'pin lists look good'

ipfs-pin-stat simple script to help check pinning

This is a simple shell script to help check pinning.

We ought to strive towards making adding commands this easy.
The http api is great and powerful, but our setup right now
gets in the way. Perhaps we can clean up that area.

updated t0081-repo-pinning

- fixed a couple bugs with the tests
- made it a bit clearer (still a lot going on)
- the remaining tests are correct and highlight a problem with
  pinning. Namely, that recursive pinning is buggy. At least:
  towards the end of the test, $HASH_DIR4 and $HASH_FILE4 should
  be pinned indirectly, but they're not. And thus get gc-ed out.
  There may be other problems too.

cc @whyrusleeping

fix grep params for context deadline check

fix bugs in pin and pin tests

check for block local before checking recursive pin
@whyrusleeping
Copy link
Member

this is a result of the path.Clean being called on datastore keys.

@kevina
Copy link
Contributor

kevina commented Jun 30, 2016

This relates to #2601.

@kevina
Copy link
Contributor

kevina commented Jul 2, 2016

@whyrusleeping is this fixed now?

@whyrusleeping
Copy link
Member

Yes, this is fixed now

ariescodescream pushed a commit to ariescodescream/go-ipfs that referenced this issue Apr 7, 2022
@whyrusleeping
fix for ipfs#1008 and other pinning fixes
99bdf19 
This commit adds a new set of sharness tests for pinning, and addresses
bugs that were pointed out by said tests.

test/sharness: added more pinning tests

Pinning is currently broken. See issue ipfs#1051. This commit introduces
a few more pinning tests. These are by no means exhaustive, but
definitely surface the present problems going on. I believe these
tests are correct, but not sure. Pushing them as failing so that
pinning is fixed in this PR.

make pinning and merkledag.Get take contexts

improve 'add' commands usage of pinning

FIXUP: fix 'pin lists look good'

ipfs-pin-stat simple script to help check pinning

This is a simple shell script to help check pinning.

We ought to strive towards making adding commands this easy.
The http api is great and powerful, but our setup right now
gets in the way. Perhaps we can clean up that area.

updated t0081-repo-pinning

- fixed a couple bugs with the tests
- made it a bit clearer (still a lot going on)
- the remaining tests are correct and highlight a problem with
  pinning. Namely, that recursive pinning is buggy. At least:
  towards the end of the test, $HASH_DIR4 and $HASH_FILE4 should
  be pinned indirectly, but they're not. And thus get gc-ed out.
  There may be other problems too.

cc @whyrusleeping

fix grep params for context deadline check

fix bugs in pin and pin tests

check for block local before checking recursive pin
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind/bug A bug in existing code (including security flaws) topic/commands Topic commands
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@jbenet @whyrusleeping @kevina