ipfs add --chunker options documentation

[thisconnect]

Version information:

go-ipfs version: 0.4.11-
Repo version: 6
System version: amd64/darwin
Golang version: go1.9

Type:

Documentation, cli help

Severity:

Low

Description:

ipfs add --help mentions the chunker option, but has no explanation for what values it accepts.

ipfs add --help
...
 -s,          --chunker             string - Chunking algorithm to use.

According to @JustinDrake

The default is to use a block size of DefaultBlockSize int64 = 1024 * 256. Another block size can be specified using size-. There's also the rabin- chunker which uses rabin-[min]-[avg]-[max].

ipfs-inactive/faq#214 (comment)

is also missing in https://ipfs.io/docs/api/#api-v0-add

Is this correct and should it be added to ipfs add --help?

---

other findings

• https://github.com/ipfs/js-ipfs#files (links to interface-ipfs-core)
  • https://github.com/ipfs/interface-ipfs-core/tree/master/API/files#add (mentions options and points to js-ipfs-unixfs-engine)
    • https://github.com/ipfs/js-ipfs-unixfs-engine#importer-api this one has "a" chunker option:

      chunker (string, defaults to "fixed"): the chunking strategy. Now only supports "fixed"
      chunkerOptions (object, optional): the options for the chunker. Defaults to an object with the following properties:
      maxChunkSize (positive integer, defaults to 262144): the maximum chunk size for the fixed chunker.

[Stebalien]

Is this correct and should it be added to

Yes. Are you volunteering 😸 ?

FYI, the variables in size-[size] and rabin-[min]-[avg]-[max] is in bytes, not KiB (as I just learned...).

[thisconnect]

Are you volunteering 😸 ?

Let me try, but I am slow bear with me. 2 questions

1. What exactly is the default? Guessing something like size-256
2. I change the text from Chunking algorithm to use. to Chunking algorithm to use. Another block size can be specified using size-. There's also the rabin- chunker which uses rabin-[min]-[avg]-[max] + default

[Stebalien]

1. It's size-262144 (1024*256).
2. Sounds good.

[olizilla]

There were 2 PRs to improve this #4306 and #4347 from @thisconnect back in 2017 so I'm gonna go ahead and close this. Thanks @thisconnect !

Also of note, @alanshaw and I made a tool to visualise how these options alter the dag that is created

• https://github.com/ipfs-shipyard/ipfs-dag-builder-vis