From 1193d757df71b6badb5c2ffd24780b5d46d039b1 Mon Sep 17 00:00:00 2001 From: Lindsay jo Walker Date: Wed, 14 Dec 2022 14:02:19 -0600 Subject: [PATCH 1/4] minor re-word of goal, edits to DAG-PB article to make it simpler --- config/_default/goals.json | 2 +- content/en/curriculum/ipld/ipld-and-ipfs/index.md | 14 ++++++++------ 2 files changed, 9 insertions(+), 7 deletions(-) diff --git a/config/_default/goals.json b/config/_default/goals.json index 3b0f702a..7f9eb1e7 100644 --- a/config/_default/goals.json +++ b/config/_default/goals.json @@ -245,7 +245,7 @@ }, { "id": "1.33", - "description": "Identify what kind of data types can you have in IPLD" + "description": "Identify what kind of data types can you use with IPLD" }, { "id": "1.34", diff --git a/content/en/curriculum/ipld/ipld-and-ipfs/index.md b/content/en/curriculum/ipld/ipld-and-ipfs/index.md index aa353126..95f9646d 100644 --- a/content/en/curriculum/ipld/ipld-and-ipfs/index.md +++ b/content/en/curriculum/ipld/ipld-and-ipfs/index.md @@ -31,7 +31,7 @@ At its core, IPFS is best viewed as a suite of standards for storing, sharing, n #### How IPFS Deals With Files | IPFS Camp 2019 Workshop – Alan Shaw -In the following talk, Alan Shaw goes over theory and live examples of what happens with files added to IPFS. He highlights different variations to the underlying data structure created with a file added to IPFS and how it affects the CID. +In the following talk, Alan Shaw goes over theory and live examples of what happens with files added to IPFS. He highlights different variations to the underlying data structure created with a file added to IPFS and how it affects the CID. {{< youtube SP1qMZA3UG8 >}} @@ -44,11 +44,13 @@ At a high level, the talk covered the following concepts: You can access and experiment with the same DAG builder from the video at [https://dag.ipfs.tech/](https://dag.ipfs.tech/) -## A Brief DAG-PB Primer +## The DAG-PB Format -The majority of data that passes through kubo (go-ipfs), js-ipfs, the IPFS gateways and various other IPFS nodes in existence, will be file data. This data uses the [**DAG-PB**](https://ipld.io/specs/codecs/dag-pb/) codec and **raw** block encoding to form graphs of IPLD blocks to represent the file data. +DAG-PB is a codec that describes the way that file data (binary) can be structured, similar to a JSON file, in a way that is built to be compatible with all data formats. -DAG-PB is a protobuf based format that can only contain two things: a Byte array, and a list of named and sized links to other blocks. Those links are almost always to other DAG-PB blocks or raw leaf blocks. +The majority of data that passes through kubo (go-ipfs), js-ipfs, the IPFS gateways and various other IPFS nodes in existence uses the [**DAG-PB**](https://ipld.io/specs/codecs/dag-pb/) codec and **raw** block encoding to form graphs of IPLD blocks to represent the file data. + +DAG-PB is a [protobuf based format](https://developers.google.com/protocol-buffers/docs/overview) that can only contain two things: a Byte array, and a list of named and sized links to other blocks. Those links are almost always to other DAG-PB blocks or raw leaf blocks. A typical DAG-PB block, represented as JSON, might look something like this: @@ -91,7 +93,7 @@ The IPLD Data Model is how we reason about data moving through the various state ### Limitations of File Data -Files are a great abstraction, but can have limitations. Often, users store structured file data with IPFS, such as JSON files. These file types don't scale well and it also prevents the use of IPLD's sophisticated graph tooling from working on the data itself. They also rely on smart file chunking algorithms to de-duplicate files as they are modified and re-stored over time, which can be a hit-or-miss process. +Files are a great abstraction, but can have limitations. Often, users store structured file data with IPFS, such as JSON files. These file types don't scale well and it also prevents the use of IPLD's sophisticated graph tooling from working on the data itself. They also rely on smart file chunking algorithms to de-duplicate files as they are modified and re-stored over time, which can be a hit-or-miss process. ### Intro to Codecs IPLD is ambitious in its aims to be able to represent many, varied types of content addressed data. IPLD-native codecs that support the full IPLD Data Model, in particular [**DAG-CBOR**](https://ipld.io/specs/codecs/dag-cbor/), _allow for the storage of structured data within IPLD blocks_ in a way that the _data itself forms nodes within the full graph._ Commonly used codecs are Protobuf, JSON, and JOSE. They have different use cases because of the way they organize their data. But IPLD will build on those codecs to include the Data Model's Bytes, Links, and data Kinds, thus converting it to an IPLD-native Codec. @@ -107,4 +109,4 @@ Schemas are an important tool for extending IPLD’s scope into the application ## Further Reading * [**IPLD Data Model**](https://ipld.io/docs/data-model/) * [**IPLD Codecs**](https://ipld.io/docs/codecs/) -* [**IPLD Schemas**](https://ipld.io/docs/schemas/) \ No newline at end of file +* [**IPLD Schemas**](https://ipld.io/docs/schemas/) From 5750ef15f1cb1f5bf65a537fcbd46a89cb4035de Mon Sep 17 00:00:00 2001 From: Lindsay jo Walker Date: Wed, 14 Dec 2022 14:02:28 -0600 Subject: [PATCH 2/4] minor re-word of goal, edits to DAG-PB article to make it simpler --- content/en/curriculum/ipld/ipld-and-ipfs/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/curriculum/ipld/ipld-and-ipfs/index.md b/content/en/curriculum/ipld/ipld-and-ipfs/index.md index 95f9646d..918b4a3b 100644 --- a/content/en/curriculum/ipld/ipld-and-ipfs/index.md +++ b/content/en/curriculum/ipld/ipld-and-ipfs/index.md @@ -46,7 +46,7 @@ You can access and experiment with the same DAG builder from the video at [https ## The DAG-PB Format -DAG-PB is a codec that describes the way that file data (binary) can be structured, similar to a JSON file, in a way that is built to be compatible with all data formats. +DAG-PB is a codec that describes the way that file data (binary) can be structured, similar to a JSON file, in a way that is compatible with all data formats. The majority of data that passes through kubo (go-ipfs), js-ipfs, the IPFS gateways and various other IPFS nodes in existence uses the [**DAG-PB**](https://ipld.io/specs/codecs/dag-pb/) codec and **raw** block encoding to form graphs of IPLD blocks to represent the file data. From 85f8fd9512d5158842c12d08572bff1981981fe0 Mon Sep 17 00:00:00 2001 From: Lindsay jo Walker Date: Wed, 14 Dec 2022 14:14:28 -0600 Subject: [PATCH 3/4] rewriting UnixFS in ipld-and-ipfs section --- content/en/curriculum/ipld/ipld-and-ipfs/index.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/content/en/curriculum/ipld/ipld-and-ipfs/index.md b/content/en/curriculum/ipld/ipld-and-ipfs/index.md index 918b4a3b..1b25e830 100644 --- a/content/en/curriculum/ipld/ipld-and-ipfs/index.md +++ b/content/en/curriculum/ipld/ipld-and-ipfs/index.md @@ -79,7 +79,8 @@ Read more about the DAG-PB codec in the [**specification**](https://ipld.io/spec ## UnixFS -[UnixFS](https://docs.ipfs.tech/concepts/file-systems/#unix-file-system-unixfs) is a data format for creating directory & file hierarchies and Merkle DAGs. UnixFS does this by adding an encoding layer _above_ the DAG-PB codec that is applied to serialize filesystem metadata. Metadata can include file timestamps, file permissions, mime types and whether an object is a file, directory or even a symlink. This means that a single block may incur a double Protobuf decode to retrieve complete information about the filesystem. +[UnixFS](https://docs.ipfs.tech/concepts/file-systems/#unix-file-system-unixfs) is a data format for creating directory & file hierarchies and Merkle DAGs. UnixFS does this by adding an encoding layer _above_ the DAG-PB codec that takes the file metadata, such as timestamps, permissions, and labels for types of data (mime types), and converts it into bytes that make it easier to transfer data to other files, databases, or memory storage. +After the byte data is sent over the wire, a UnixFS block may incur a double Protobuf decode to retrieve complete information about the filesystem. ![Unixfs_addon](unixfs_addon.png) From b85ae9f30ad61dafaa8a32253ab82185f00a56fa Mon Sep 17 00:00:00 2001 From: Marco Rodriguez-Salinas Date: Mon, 19 Dec 2022 15:23:22 -0800 Subject: [PATCH 4/4] Update content/en/curriculum/ipld/ipld-and-ipfs/index.md --- content/en/curriculum/ipld/ipld-and-ipfs/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/curriculum/ipld/ipld-and-ipfs/index.md b/content/en/curriculum/ipld/ipld-and-ipfs/index.md index 1b25e830..e6f3d717 100644 --- a/content/en/curriculum/ipld/ipld-and-ipfs/index.md +++ b/content/en/curriculum/ipld/ipld-and-ipfs/index.md @@ -46,7 +46,7 @@ You can access and experiment with the same DAG builder from the video at [https ## The DAG-PB Format -DAG-PB is a codec that describes the way that file data (binary) can be structured, similar to a JSON file, in a way that is compatible with all data formats. +The majority of data that passes through kubo (go-ipfs), js-ipfs, the IPFS gateways and various other IPFS nodes in existence, will be file data. This data uses the [**DAG-PB**](https://ipld.io/specs/codecs/dag-pb/) encoding to transform binary file data into human-readable structures (see below) that helps IPLD form Merkle DAGs. The majority of data that passes through kubo (go-ipfs), js-ipfs, the IPFS gateways and various other IPFS nodes in existence uses the [**DAG-PB**](https://ipld.io/specs/codecs/dag-pb/) codec and **raw** block encoding to form graphs of IPLD blocks to represent the file data.