Skip to content

Latest commit

 

History

History
150 lines (114 loc) · 3.76 KB

w3-index.md

File metadata and controls

150 lines (114 loc) · 3.76 KB

W3 Index

wip

Introduction

Abstract

W3 Indexing protocol allows authorized agents to submit verifiable claims about content-addressable data to be publish on InterPlanetary Network Indexer (IPNI), making it publicly queryable.

Concepts

Space

A namespace, often referred as a "space", is an owned resource that can be shared. It corresponds to a unique asymmetric cryptographic keypair and is identified by a did:key URI.

Capabilities

Index Add

Authorized agent MAY invoke /space/index/add capability on the [space] subject to submit [content index] for publishing on InterPlanetary Network Indexer (IPNI).

Index Add Invocation Example

Invocation example illustrates Alice requesting to add index under "bafy..blob" to be published on the network.

{
  "cmd": "/space/index/add",
  "sub": "did:key:zAlice",
  "iss": "did:key:zAlice",
  "aud": "did:web:web3.storage",
  "args": {
    "index": { "/": "bafy...blob" }
  }
}

Index Add Capability

Index Add Capability Schema

type AddIndex = {
  cmd: "/space/index/add"
  sub: SpaceDID
  args: {
    // Link is the Content Archive (CAR) containing
    // the `Index`.
    index: Link<ContentArchive<Index>>
  }
}

// Type describes a CAR format
type ContentArchive<T> = ByteView<{
  roots: [Block<T>]
  blocks: Block[]
}>

Index

Index Schema

Index schema is variant type keyed by the format descriptor label designed to allow format evolution through versioning and additional schema variants.

type Index = Variant<{
  "index/sharded/[email protected]": ShardedDAGIndex
}>

Sharded DAG Index

Sharded DAG index MUST describe complete set of blocks that make up the content DAG in terms of BlobSlices.

Sharded DAG Index Schema
type ShardedDAGIndex = {
  // content root CID
  content: Link<any>
  // links to blob indexes that contain blocks of the content DAG
  shards: Link<BlobIndex>[]
}

type BlobIndex = [
  // hash digest of the blob
  digest: Multihash
  // Index of blob slices
  slices: BlobSlice
]

type BlobSlice = [
  // hash digest of the slice
  digest: Multihash
  // Slice offset
  offset: Int
  // Slice size in bytes
  length: Int
]

type Multihash = bytes

ℹ️ Please note that shards is a list of BlobIndex links. This provide a flexibility of bundling blob indexes or externalizing them by linking to them.

It is RECOMMENDED to include BlobSlice in slices that spans full range of the blob to make it available. On the flip side it creates a choice to share only partial index of the blob when so desired.

It is RECOMMENDED to bundle all the BlobIndexs inside the Content Archive of the Index.

Sharded DAG Index Example

For the reader convenience we use link function to denote external blocks that should be linked

{
  "index/sharded/[email protected]": {
    "content": { "/": "bafy..dag" },
    "shards": [
      link([
        // blob multihash
        { "/": { "bytes": "blb...left" } },
        // sliced within the blob
        [
          [{ "/": { "bytes": "block..1"} }, 0, 128],
          [{ "/": { "bytes": "block..2"} }, 129, 256],
          [{ "/": { "bytes": "block..3"} }, 257, 384],
          [{ "/": { "bytes": "block..4"} }, 385, 512]
        ]
      ]),
      link([
        // blob multihash
        { "/": { "bytes": "blb...right" } },
        // sliced within the blob
        [
          [{ "/": { "bytes": "block..5"} }, 0, 128],
          [{ "/": { "bytes": "block..6"} }, 129, 256],
          [{ "/": { "bytes": "block..7"} }, 257, 384],
          [{ "/": { "bytes": "block..8"} }, 385, 512]
        ]
      ])
    ]
  }
}