Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Lindz/ipld1.3 #444

Merged
merged 4 commits into from
Dec 19, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion config/_default/goals.json
Original file line number Diff line number Diff line change
Expand Up @@ -245,7 +245,7 @@
},
{
"id": "1.33",
"description": "Identify what kind of data types can you have in IPLD"
"description": "Identify what kind of data types can you use with IPLD"
},
{
"id": "1.34",
Expand Down
17 changes: 10 additions & 7 deletions content/en/curriculum/ipld/ipld-and-ipfs/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ At its core, IPFS is best viewed as a suite of standards for storing, sharing, n

#### How IPFS Deals With Files | IPFS Camp 2019 Workshop – Alan Shaw

In the following talk, Alan Shaw goes over theory and live examples of what happens with files added to IPFS. He highlights different variations to the underlying data structure created with a file added to IPFS and how it affects the CID.
In the following talk, Alan Shaw goes over theory and live examples of what happens with files added to IPFS. He highlights different variations to the underlying data structure created with a file added to IPFS and how it affects the CID.

{{< youtube SP1qMZA3UG8 >}}

Expand All @@ -44,11 +44,13 @@ At a high level, the talk covered the following concepts:

You can access and experiment with the same DAG builder from the video at [https://dag.ipfs.tech/](https://dag.ipfs.tech/)

## A Brief DAG-PB Primer
## The DAG-PB Format

The majority of data that passes through kubo (go-ipfs), js-ipfs, the IPFS gateways and various other IPFS nodes in existence, will be file data. This data uses the [**DAG-PB**](https://ipld.io/specs/codecs/dag-pb/) codec and **raw** block encoding to form graphs of IPLD blocks to represent the file data.
The majority of data that passes through kubo (go-ipfs), js-ipfs, the IPFS gateways and various other IPFS nodes in existence, will be file data. This data uses the [**DAG-PB**](https://ipld.io/specs/codecs/dag-pb/) encoding to transform binary file data into human-readable structures (see below) that helps IPLD form Merkle DAGs.

DAG-PB is a protobuf based format that can only contain two things: a Byte array, and a list of named and sized links to other blocks. Those links are almost always to other DAG-PB blocks or raw leaf blocks.
The majority of data that passes through kubo (go-ipfs), js-ipfs, the IPFS gateways and various other IPFS nodes in existence uses the [**DAG-PB**](https://ipld.io/specs/codecs/dag-pb/) codec and **raw** block encoding to form graphs of IPLD blocks to represent the file data.

DAG-PB is a [protobuf based format](https://developers.google.com/protocol-buffers/docs/overview) that can only contain two things: a Byte array, and a list of named and sized links to other blocks. Those links are almost always to other DAG-PB blocks or raw leaf blocks.

A typical DAG-PB block, represented as JSON, might look something like this:

Expand Down Expand Up @@ -77,7 +79,8 @@ Read more about the DAG-PB codec in the [**specification**](https://ipld.io/spec

## UnixFS

[UnixFS](https://docs.ipfs.tech/concepts/file-systems/#unix-file-system-unixfs) is a data format for creating directory & file hierarchies and Merkle DAGs. UnixFS does this by adding an encoding layer _above_ the DAG-PB codec that is applied to serialize filesystem metadata. Metadata can include file timestamps, file permissions, mime types and whether an object is a file, directory or even a symlink. This means that a single block may incur a double Protobuf decode to retrieve complete information about the filesystem.
[UnixFS](https://docs.ipfs.tech/concepts/file-systems/#unix-file-system-unixfs) is a data format for creating directory & file hierarchies and Merkle DAGs. UnixFS does this by adding an encoding layer _above_ the DAG-PB codec that takes the file metadata, such as timestamps, permissions, and labels for types of data (mime types), and converts it into bytes that make it easier to transfer data to other files, databases, or memory storage.
After the byte data is sent over the wire, a UnixFS block may incur a double Protobuf decode to retrieve complete information about the filesystem.

![Unixfs_addon](unixfs_addon.png)

Expand All @@ -91,7 +94,7 @@ The IPLD Data Model is how we reason about data moving through the various state

### Limitations of File Data

Files are a great abstraction, but can have limitations. Often, users store structured file data with IPFS, such as JSON files. These file types don't scale well and it also prevents the use of IPLD's sophisticated graph tooling from working on the data itself. They also rely on smart file chunking algorithms to de-duplicate files as they are modified and re-stored over time, which can be a hit-or-miss process.
Files are a great abstraction, but can have limitations. Often, users store structured file data with IPFS, such as JSON files. These file types don't scale well and it also prevents the use of IPLD's sophisticated graph tooling from working on the data itself. They also rely on smart file chunking algorithms to de-duplicate files as they are modified and re-stored over time, which can be a hit-or-miss process.

### Intro to Codecs
IPLD is ambitious in its aims to be able to represent many, varied types of content addressed data. IPLD-native codecs that support the full IPLD Data Model, in particular [**DAG-CBOR**](https://ipld.io/specs/codecs/dag-cbor/), _allow for the storage of structured data within IPLD blocks_ in a way that the _data itself forms nodes within the full graph._ Commonly used codecs are Protobuf, JSON, and JOSE. They have different use cases because of the way they organize their data. But IPLD will build on those codecs to include the Data Model's Bytes, Links, and data Kinds, thus converting it to an IPLD-native Codec.
Expand All @@ -107,4 +110,4 @@ Schemas are an important tool for extending IPLD’s scope into the application
## Further Reading
* [**IPLD Data Model**](https://ipld.io/docs/data-model/)
* [**IPLD Codecs**](https://ipld.io/docs/codecs/)
* [**IPLD Schemas**](https://ipld.io/docs/schemas/)
* [**IPLD Schemas**](https://ipld.io/docs/schemas/)