Pinning progress

[achingbrain]

Version information:

$ ipfs --version --all
go-ipfs version: 0.8.0
Repo version: 11
System version: amd64/darwin
Golang version: go1.15.8

Description:

I'm looking at exposing the progress arg to ipfs.pin.addAll for ipfs-http-client and am trying to nail down the behaviour of go-IPFS.

If I add a directory recursively I see progress:

$ ipfs add -r --pin=false .
added QmdAtxgJwAaK41woVuM4tq9qr6NPw8cC212TajNAot7gqV ipfs/CHANGELOG.md
added QmdaGY5JWEVPDR2WoMYETnhqJvbCgCZTvocjF1YqXnJMHt ipfs/CODE_OF_CONDUCT.md
added QmVFQ52asX4zgY6Sbp7tmDe8Vs6AemgiLfj6nkRuRrJkuC ipfs/CONTRIBUTING.md
added QmfNPdCxwKXQxo1fKjhCggaPLA9rfG4uJSZuuPWgrSSnnb ipfs/COPYRIGHT
added QmdcxcVZw8Pojj1rhTCSp4uoNTV2fY8JwpBb3Bj5XNDMpd ipfs/LICENSE-APACHE
added QmRcNnUwgBVjBeJLiBV2uSi8CmFdSvoHj62kWrCZRBA749 ipfs/LICENSE-MIT
... more files here
added QmbgNwtUFsPkrnz9QYwwLWXKP1u1SrmMqUqpats3hvtHkn ipfs/node_modules
added QmYdcvCbpKGq2bT7Y35cnhGDAmFdULQ99bcQRre7C3iFDL ipfs/src
added QmX74v4CMyrGWEjA8tKeQKFJDXowEJ5Zfzjbdo2dnMAXot ipfs/test/utils
added Qma4PYzbxzYkUq36g5eZi8QdNVC7Rf6dP9MxgtgFNRMQcx ipfs/test
added QmdLWTWW9cRQJuiyeAXDw9ZMgcaSdu9SRPEw2M43wMpReU ipfs
 175.16 KiB / 175.16 KiB [=======================================================================] 100.00%

If I then pin that directory with the progress CLI arg, I see no progress:

$ ipfs pin add -r --progress=true QmdLWTWW9cRQJuiyeAXDw9ZMgcaSdu9SRPEw2M43wMpReU
pinned QmdLWTWW9cRQJuiyeAXDw9ZMgcaSdu9SRPEw2M43wMpReU recursively

If I do the same operation via the HTTP API, I get a progress indicator, but only one and it's only emitted once the whole DAG has been pinned:

$ curl -X POST http://localhost:5001/api/v0/pin/add?arg=QmdLWTWW9cRQJuiyeAXDw9ZMgcaSdu9SRPEw2M43wMpReU\&progress=true
{"Pins":null,"Progress":32}
{"Pins":["QmdLWTWW9cRQJuiyeAXDw9ZMgcaSdu9SRPEw2M43wMpReU"]}

I'm guessing the value 32 is the total number of blocks in the DAG that was pinned?

Is this functioning as intended? It doesn't look like this can be used to indicate progress, in that it only arrives once the operation is complete.

[luavixen]

Popping out of the woodwork to say that there has never been a way to get any kind of progress via the HTTP API, since HTTP can only return a result once (which is done once the operation in question has completed). It would be super beneficial to have a way to get the progress of long-running operations via the HTTP API but HTTP does not provide any way to actually communicate the progress information (without doing weird things or using WebSockets).

[achingbrain]

there has never been a way to get any kind of progress via the HTTP API, since HTTP can only return a result once (which is done once the operation in question has completed)

This isn't quite right - progress works for other IPFS HTTP API methods by holding the response stream open and writing progress events into it as newline delimited JSON as they become available. This is the mechanism by which you see progress during the ipfs add operation as shown above.

To answer my own question from above - 'Is this functioning as intended?' no, it's not - ipfs pin add -r $CID once reported progress as the DAG being pinned was traversed and individual blocks were added to the blockstore but now it does not, which is just a regression and can be fixed.

[luavixen]

Wow, that's interesting! It looks like I have a bug of my own to track down, thanks for helping me figure out that it isn't the IPFS daemon :P
Sorry for adding noise to this issue!

[Stebalien]

Ok, after testing this, it appears to be working as intended.

The "Pins": null part should jus be omitted... that's a bug. Otherwise, Progress is the number of blocks pinned so far and appears to be working correctly (it gets printed every half second).

Unfortunately, we can't print out filenames because pinning doesn't know about unixfs.

[achingbrain]

it gets printed every half second

Aha, I think that's what I was missing, the thing that I pinned took less than half a second so I only saw one progress notification and thought there would only be one at the end of the operation.

[kallisti5]

Just pinging here in agreement :-)

Pinning ~20GiB on my node, and this is the progress I get:

$ ipfs pin add --progress /ipns/hpkg.haiku-os.org
Fetched/Processed 8353 nodes
Fetched/Processed 8540 nodes
.
.
Fetched/Processed 95036 nodes
etc.

It would be nice to have a little more information. Even some cleanup of the language would be helpful to be consistent. (is this node like a peer? Is it node like a chunk?)

Examples:

• Discovering Chunks: 8353 nodes (rolling update)

Then...

• Discovered Chunks across 9000 nodes
• Doing other stuff... (rolling update)

[Stebalien]

@kallisti5 could you open a new issue for this? This one was a bug report so I'm going to close it.