diff --git a/config/_default/goals.json b/config/_default/goals.json index 3b0f702a..7f9eb1e7 100644 --- a/config/_default/goals.json +++ b/config/_default/goals.json @@ -245,7 +245,7 @@ }, { "id": "1.33", - "description": "Identify what kind of data types can you have in IPLD" + "description": "Identify what kind of data types can you use with IPLD" }, { "id": "1.34", diff --git a/content/en/curriculum/ipld/ipld-and-ipfs/index.md b/content/en/curriculum/ipld/ipld-and-ipfs/index.md index aa353126..e6f3d717 100644 --- a/content/en/curriculum/ipld/ipld-and-ipfs/index.md +++ b/content/en/curriculum/ipld/ipld-and-ipfs/index.md @@ -31,7 +31,7 @@ At its core, IPFS is best viewed as a suite of standards for storing, sharing, n #### How IPFS Deals With Files | IPFS Camp 2019 Workshop – Alan Shaw -In the following talk, Alan Shaw goes over theory and live examples of what happens with files added to IPFS. He highlights different variations to the underlying data structure created with a file added to IPFS and how it affects the CID. +In the following talk, Alan Shaw goes over theory and live examples of what happens with files added to IPFS. He highlights different variations to the underlying data structure created with a file added to IPFS and how it affects the CID. {{< youtube SP1qMZA3UG8 >}} @@ -44,11 +44,13 @@ At a high level, the talk covered the following concepts: You can access and experiment with the same DAG builder from the video at [https://dag.ipfs.tech/](https://dag.ipfs.tech/) -## A Brief DAG-PB Primer +## The DAG-PB Format -The majority of data that passes through kubo (go-ipfs), js-ipfs, the IPFS gateways and various other IPFS nodes in existence, will be file data. This data uses the [**DAG-PB**](https://ipld.io/specs/codecs/dag-pb/) codec and **raw** block encoding to form graphs of IPLD blocks to represent the file data. +The majority of data that passes through kubo (go-ipfs), js-ipfs, the IPFS gateways and various other IPFS nodes in existence, will be file data. This data uses the [**DAG-PB**](https://ipld.io/specs/codecs/dag-pb/) encoding to transform binary file data into human-readable structures (see below) that helps IPLD form Merkle DAGs. -DAG-PB is a protobuf based format that can only contain two things: a Byte array, and a list of named and sized links to other blocks. Those links are almost always to other DAG-PB blocks or raw leaf blocks. +The majority of data that passes through kubo (go-ipfs), js-ipfs, the IPFS gateways and various other IPFS nodes in existence uses the [**DAG-PB**](https://ipld.io/specs/codecs/dag-pb/) codec and **raw** block encoding to form graphs of IPLD blocks to represent the file data. + +DAG-PB is a [protobuf based format](https://developers.google.com/protocol-buffers/docs/overview) that can only contain two things: a Byte array, and a list of named and sized links to other blocks. Those links are almost always to other DAG-PB blocks or raw leaf blocks. A typical DAG-PB block, represented as JSON, might look something like this: @@ -77,7 +79,8 @@ Read more about the DAG-PB codec in the [**specification**](https://ipld.io/spec ## UnixFS -[UnixFS](https://docs.ipfs.tech/concepts/file-systems/#unix-file-system-unixfs) is a data format for creating directory & file hierarchies and Merkle DAGs. UnixFS does this by adding an encoding layer _above_ the DAG-PB codec that is applied to serialize filesystem metadata. Metadata can include file timestamps, file permissions, mime types and whether an object is a file, directory or even a symlink. This means that a single block may incur a double Protobuf decode to retrieve complete information about the filesystem. +[UnixFS](https://docs.ipfs.tech/concepts/file-systems/#unix-file-system-unixfs) is a data format for creating directory & file hierarchies and Merkle DAGs. UnixFS does this by adding an encoding layer _above_ the DAG-PB codec that takes the file metadata, such as timestamps, permissions, and labels for types of data (mime types), and converts it into bytes that make it easier to transfer data to other files, databases, or memory storage. +After the byte data is sent over the wire, a UnixFS block may incur a double Protobuf decode to retrieve complete information about the filesystem. ![Unixfs_addon](unixfs_addon.png) @@ -91,7 +94,7 @@ The IPLD Data Model is how we reason about data moving through the various state ### Limitations of File Data -Files are a great abstraction, but can have limitations. Often, users store structured file data with IPFS, such as JSON files. These file types don't scale well and it also prevents the use of IPLD's sophisticated graph tooling from working on the data itself. They also rely on smart file chunking algorithms to de-duplicate files as they are modified and re-stored over time, which can be a hit-or-miss process. +Files are a great abstraction, but can have limitations. Often, users store structured file data with IPFS, such as JSON files. These file types don't scale well and it also prevents the use of IPLD's sophisticated graph tooling from working on the data itself. They also rely on smart file chunking algorithms to de-duplicate files as they are modified and re-stored over time, which can be a hit-or-miss process. ### Intro to Codecs IPLD is ambitious in its aims to be able to represent many, varied types of content addressed data. IPLD-native codecs that support the full IPLD Data Model, in particular [**DAG-CBOR**](https://ipld.io/specs/codecs/dag-cbor/), _allow for the storage of structured data within IPLD blocks_ in a way that the _data itself forms nodes within the full graph._ Commonly used codecs are Protobuf, JSON, and JOSE. They have different use cases because of the way they organize their data. But IPLD will build on those codecs to include the Data Model's Bytes, Links, and data Kinds, thus converting it to an IPLD-native Codec. @@ -107,4 +110,4 @@ Schemas are an important tool for extending IPLD’s scope into the application ## Further Reading * [**IPLD Data Model**](https://ipld.io/docs/data-model/) * [**IPLD Codecs**](https://ipld.io/docs/codecs/) -* [**IPLD Schemas**](https://ipld.io/docs/schemas/) \ No newline at end of file +* [**IPLD Schemas**](https://ipld.io/docs/schemas/)