Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

image-layout: ./refs/ -> index.json #533

Merged
merged 2 commits into from
Feb 8, 2017
Merged

image-layout: ./refs/ -> index.json #533

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
489e00e
manifest-list: `platform` is effective optional
vbatts Jan 26, 2017
0556a6b
image-layout: ./refs/ -> index.json
vbatts Jan 26, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 8 additions & 5 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -75,12 +75,14 @@ $(OUTPUT_DIRNAME)/$(DOC_FILENAME).html: $(DOC_FILES) $(FIGURE_FILES)
ls -sh $(shell readlink -f $@)
endif

validate-examples:
validate-examples: schema/fs.go
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Travis doesn't actually call validate-examples, and it's not a dependency of the test target either. If we're trying to rebuild schema/fs.go before any tests, you probably need to have:

test: schema/fs.go

as well.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

then i'll add validate-examples as a travis target.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

then i'll add validate-examples as a travis target.

That will double those tests, since make test, which we already call, also runs the example tests. Other make test tests depend on fs.go as well, so I think you can leave Travis alone and add the fs.go dependency to test.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh i see. ok.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe schema/fs.go doesn't even need to be commited then ... only generated. It often causes a lot of senseless rebasing.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#543

go test -run TestValidate ./schema

schema-fs:
schema/fs.go: $(wildcard schema/*.json) schema/gen.go
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure how well make will handle this in a fresh Git checkout. Maybe Travis should touch schema/gen.go to ensure a newer-than-fs.go mtime?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

presumably, a fresh checkout wouldn't need a fresh schema. the schema-fs ought to be commited when the JSON schema changes.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the schema-fs ought to be commited when the JSON schema changes.

But that doesn't always happen (it didn't in #411). Making sure it does always happen was what I was trying to do with #530. I'd thought the approach you were aiming for here was to always rebuild in CI so it didn't matter if the committed copy was stale.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

right. If travis runs the validate-examples, then it would fail CI for their PR, then they update schema-fs, commit and re-push.

cd schema && printf "%s\n\n%s\n" "$$(cat ../.header)" "$$(go generate)" > fs.go

schema-fs: schema/fs.go
@echo "generating schema fs"
@cd schema && printf "%s\n\n%s\n" "$$(cat ../.header)" "$$(go generate)" > fs.go

check-license:
@echo "checking license headers"
Expand All @@ -90,7 +92,7 @@ lint:
@echo "checking lint"
@./.tool/lint

test:
test: schema/fs.go
go test -race -cover $(shell go list ./... | grep -v /vendor/)

img/%.png: img/%.dot
Expand Down Expand Up @@ -129,4 +131,5 @@ clean:
clean \
lint \
docs \
test
test \
schema-fs
1 change: 1 addition & 0 deletions annotations.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,3 +21,4 @@ This specification defines the following annotation keys, intended for but not l
* **org.opencontainers.homepage** URL to find more information on the image (string, a URL with scheme HTTP or HTTPS)
* **org.opencontainers.documentation** URL to get documentation on the image (string, a URL with scheme HTTP or HTTPS)
* **org.opencontainers.source** URL to get source code for the binary files in the image (string, a URL with scheme HTTP or HTTPS)
* **org.opencontainers.ref.name** Name of the reference (string)
2 changes: 1 addition & 1 deletion config.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -229,4 +229,4 @@ Here is an example image configuration JSON document:
```

[rfc3339-s5.6]: https://tools.ietf.org/html/rfc3339#section-5.6
[runtime-platform]: https://github.com/opencontainers/runtime-spec/blob/v1.0.0-rc2/config.md#platform
[runtime-platform]: https://github.com/opencontainers/runtime-spec/blob/v1.0.0-rc3/config.md#platform
153 changes: 99 additions & 54 deletions image-layout.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,47 +3,45 @@
The OCI Image Layout is a slash separated layout of OCI content-addressable blobs and [location-addressable](https://en.wikipedia.org/wiki/Content-addressable_storage#Content-addressed_vs._location-addressed) references (refs).
This layout MAY be used in a variety of different transport mechanisms: archive formats (e.g. tar, zip), shared filesystem environments (e.g. nfs), or networked file fetching (e.g. http, ftp, rsync).

Given an image layout and a ref, a tool can create an [OCI Runtime Specification bundle](https://github.com/opencontainers/runtime-spec/blob/v1.0.0-rc2/bundle.md) by:
Given an image layout and a ref, a tool can create an [OCI Runtime Specification bundle](https://github.com/opencontainers/runtime-spec/blob/v1.0.0-rc3/bundle.md) by:

* Following the ref to find a [manifest](manifest.md#image-manifest), possibly via a [manifest list](manifest-list.md#manifest-list)
* [Applying the filesystem layers](layer.md#applying) in the specified order
* Converting the [image configuration](config.md) into an [OCI Runtime Specification `config.json`](https://github.com/opencontainers/runtime-spec/blob/v1.0.0-rc2/config.md)
* Converting the [image configuration](config.md) into an [OCI Runtime Specification `config.json`](https://github.com/opencontainers/runtime-spec/blob/v1.0.0-rc3/config.md)

# Content

The image layout MUST contain two top level directories:

- `blobs` contains content-addressable blobs.
A blob has no schema and should be considered opaque.
- `refs` contains [descriptors][descriptors].
Commonly pointing to an [image manifest](manifest.md#image-manifest) or an [image manifest list](manifest-list.md#oci-image-manifest-list-specification).

Both `blobs` and `refs` MAY be empty.

The image layout MUST also contain an `oci-layout` file:

- It MUST be a JSON object
- It MUST contain an `imageLayoutVersion` field
- The `imageLayoutVersion` value will align with the OCI Image Specification version at the time changes to the layout are made, and will pin a given version until changes to the layout are required
- It MAY include additional fields
The image layout is as follows:

- `blobs` directory
- Contains content-addressable blobs
- A blob has no schema and should be considered opaque
- Directory MUST exist and MAY be empty
- See [blobs](#blobs) section
- `oci-layout` file
- It MUST exist
- It MUST be a JSON object
- It MUST contain an `imageLayoutVersion` field
- See [oci-layout file](#oci-layout-file) section
- It MAY include additional fields
- `index.json` file
- It MUST exist
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This doesn't match your “OPTIONAL index.json” commit message and PR summaries. You'll probably want to update for consistency.

Personally, I have no problem with making index.json optional (it feels like a bigger requirement than an empty refs directory for folks who have no references).

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll update the message and summaries. It is a simple enough file to be there and it is logically consistent to have it be required if the ./refs/ directory was required.

- It MUST be a JSON object
- It MUST have the base properties of [manifest-list](manifest-list.md).
- See [index.json](#indexjson-file) section

## Example Layout

This is an example image layout:

```
$ cd example.com/app/
$ find .
.
./blobs
./blobs/sha256/e692418e4cbaf90ca69d05a66403747baa33ee08806650b51fab815ad7fc331f
./blobs/sha256/afff3924849e458c5ef237db5f89539274d5e609db5db935ed3959c90f1f2d51
./blobs/sha256/9b97579de92b1c195b85bb42a11011378ee549b02d7fe9c17bf2a6b35d5cb079
./blobs/sha256/5b0bcabd1ed22e9fb1310cf6c2dec7cdef19f0ad69efa1f392e94a4333501270
$ find . -type f
./index.json
./oci-layout
./refs
./refs/v1.0
./refs/stable-release
./blobs/sha256/3588d02542238316759cbf24502f4344ffcc8a60c803870022f335d1390c13b4
./blobs/sha256/4b0bc1c4050b03c95ef2a8e36e25feac42fd31283e8c30b3ee5df6b043155d3c
./blobs/sha256/7968321274dc6b6171697c33df7815310468e694ac5be0ec03ff053bb135e768
```

Blobs are named by their contents:
Expand All @@ -53,33 +51,6 @@ $ shasum -a 256 ./blobs/sha256/afff3924849e458c5ef237db5f89539274d5e609db5db935e
afff3924849e458c5ef237db5f89539274d5e609db5db935ed3959c90f1f2d51 ./blobs/sha256/afff3924849e458c5ef237db5f89539274d5e609db5db935ed3959c90f1f2d51
```

## Refs

Object names in the `refs` subdirectories MUST NOT include characters outside of the set of "A" to "Z", "a" to "z", "0" to "9", the hyphen `-`, the dot `.`, and the underscore `_`.

No semantic restriction is given for object names in the `refs` subdirectory.
Each object in the `refs` subdirectory MUST be of type `application/vnd.oci.descriptor.v1+json`.
In general the `mediaType` of this [descriptor][descriptors] object will be either `application/vnd.oci.image.manifest.list.v1+json` or `application/vnd.oci.image.manifest.v1+json` although future versions of the spec MAY use a different mediatype.

**Implementor's Note:**
A common use case of refs is representing "tags" for a container image.
For example, an image may have a tag for different versions or builds of the software.
In the wild you often see "tags" like "v1.0.0-vendor.0", "2.0.0-debug", etc.
Those tags will often be represented in an image-layout repository with matching refs names like "v1.0.0-vendor.0", "2.0.0-debug", etc.

### Example Ref

This is an example `v1.0` ref with a manifest-list descriptor:

```
$ cat ./refs/v1.0 | jq
{
"size": 4096,
"digest": "sha256:e692418e4cbaf90ca69d05a66403747baa33ee08806650b51fab815ad7fc331f",
"mediaType": "application/vnd.oci.image.manifest.list.v1+json"
}
```

## Blobs

Object names in the `blobs` subdirectories are composed of a directory for each hash algorithm, the children of which will contain the actual content.
Expand Down Expand Up @@ -156,4 +127,78 @@ $ cat ./blobs/sha256/e692418e4cbaf90ca69d05a66403747baa33ee08806650b51fab815ad7f
[tar stream]
```

## oci-layout file

This JSON object serves as a marker for the base of an Open Container Image Layout and to provide the version of the image-layout in use.
The `imageLayoutVersion` value will align with the OCI Image Specification version at the time changes to the layout are made, and will pin a given version until changes to the image layout are required.

### oci-layout Example

```json
{
"imageLayoutVersion": "1.0.0"
}
```

## index.json file

This REQUIRED file is the entry point for references and descriptors of the image-layout.
The [manifest-list](manifest-list.md) is a multi-descriptor entry point.

This index provides an established path (`/index.json`) to have an entry point for an image-layout and to discover auxiliary descriptors.

No semantic restriction is given for the "org.opencontainers.ref.name" annotation of descriptors.
In general the `mediaType` of each [descriptor][descriptors] object in the `manifests` field will be either `application/vnd.oci.image.manifest.list.v1+json` or `application/vnd.oci.image.manifest.v1+json`
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we remove the part about application/vnd.oci.image.manifest.list.v1+json? I don't see a use case for it and it just makes things complicated and unbounded.

In fact, if we ban, we can guarantee constant time processing for manifest resolution.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It needs to be there, because an image might have a ref/tag that points to a manifest-list, say for the "v1.0.0" or "stable" and has multiple platforms.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we encourage merging and flattening them in the index?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

interesting thought, thought that would mean possibly having duplicate ref.name's and the only difference is the platform field. That sound a tad messy, no?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@vbatts Just a thought. There are some implications to that. I think this is fine.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@stevvooe right on. Thankfully, leaving like it is currently will not exclude folks that want to flatten in the way you described.

Future versions of the spec MAY use a different mediatype (i.e. a new versioned format).
An encountered `mediaType` that is unknown SHOULD be safely ignored.


**Implementor's Note:**
A common use case of descriptors with a "org.opencontainers.ref.name" annotation is representing a "tag" for a container image.
For example, an image may have a tag for different versions or builds of the software.
In the wild you often see "tags" like "v1.0.0-vendor.0", "2.0.0-debug", etc.
Those tags will often be represented in an image-layout repository with matching "org.opencontainers.ref.name" annotations like "v1.0.0-vendor.0", "2.0.0-debug", etc.


### Index Example

```json,title=Manifest%20List&mediatype=application/vnd.oci.image.manifest.list.v1%2Bjson
{
"schemaVersion": 2,
"manifests": [
{
"mediaType": "application/vnd.oci.image.manifest.v1+json",
"size": 7143,
"digest": "sha256:0228f90e926ba6b96e4f39cf294b2586d38fbb5a1e385c05cd1ee40ea54fe7fd",
"annotations": {
"org.opencontainers.ref.name": "stable-release"
}
},
{
"mediaType": "application/vnd.oci.image.manifest.v1+json",
"size": 7143,
"digest": "sha256:e692418e4cbaf90ca69d05a66403747baa33ee08806650b51fab815ad7fc331f",
"annotations": {
"org.opencontainers.ref.name": "v1.0"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this mandatory? walking the reference dir and using the filename as the reference name worked but if this annotation isn't provided, what should implementos use as "the reference name"?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

walking the reference dir and using the filename as the reference name worked but if this annotation isn't provided, what should implementos use as "the reference name"?

Walking the reference directory didn't work over bare HTTP, and we're likely dropping refs/ anyway. I still don't think this annotation needs to be required though. If it's not set, the entry just doesn't have a “reference name”. You may still be able to address it through other means (e.g. maybe it's the only entry with a given media type, or it has some other annotation that you can use to identify it).

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

how would you refer to a given revision of an image w/o a ref name?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

how would you refer to a given revision of an image w/o a ref name?

  • By index in the array (e.g. manifests[0]), or
  • By another annotation (e.g. com.example.foo.bar="docs"), or
  • By media type (e.g. application/vnd.example.docs+xml).

Consumers which were based on referencing solely via org.opencontainers.ref.name would just ignore (and be unable to address) entries which didn't set that annotation.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@vbatts @stevvooe how is this working with docker or other means (skopeo) to load OCI images and give a reference (tag) to them?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

While we're figuring out the naming semantics, we probably want to also address whether the ref name MUST be unique or SHOULD be unique or MAY be duplicated. And for all of that, within what scope (the descriptor array? The referencing blob?).

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@runcom the org.opencontainers.ref.name will act as the pointers from the descriptors in the list.
This will not be that different from the manifest.json in the current docker save ... layout.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@runcom I would think it would just fall back to the digest if the annotation is missing.

}
},
{
"mediaType": "application/xml",
"size": 7143,
"digest": "sha256:b3d63d132d21c3ff4c35a061adf23cf43da8ae054247e32faa95494d904a007e",
"annotations": {
"org.freedesktop.specifications.metainfo.version": "1.0",
"org.freedesktop.specifications.metainfo.type": "AppStream"
}
}
],
"annotations": {
"com.example.index.revision": "r124356"
}
}
```

This illustrates an index that provides two named manifest references and an auxilary mediatype for this image layout.


[descriptors]: ./descriptor.md
24 changes: 11 additions & 13 deletions manifest-list.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,9 @@ For the media type(s) that this document is compatible with, see the [matrix][ma
- **`schemaVersion`** *int*

This REQUIRED property specifies the image manifest schema version.
For this version of the specification, this MUST be `2` to ensure backward compatibility with older versions of Docker. The value of this field will not change. This field MAY be removed in a future version of the specification.
For this version of the specification, this MUST be `2` to ensure backward compatibility with older versions of Docker.
The value of this field will not change.
This field MAY be removed in a future version of the specification.

- **`mediaType`** *string*

Expand All @@ -20,10 +22,10 @@ For the media type(s) that this document is compatible with, see the [matrix][ma

- **`manifests`** *array of objects*

This REQUIRED property contains a list of manifests for specific platforms.
While the property MUST be present, the size of the array MAY be zero.
This REQUIRED property contains a list of [manifests](manifest.md) for specific platforms.
While this property MUST be present, the size of the array MAY be zero.

Each object in `manifests` is a [descriptor](descriptor.md) with the following additional properties and restrictions:
Each object in `manifests` has the base properties of [descriptor](descriptor.md) with the following additional properties and restrictions:

- **`mediaType`** *string*

Expand All @@ -33,10 +35,13 @@ For the media type(s) that this document is compatible with, see the [matrix][ma
- [`application/vnd.oci.image.manifest.v1+json`](manifest.md)

Manifest lists concerned with portability SHOULD use one of the above media types.
Future versions of the spec MAY use a different mediatype (i.e. a new versioned format).
An encountered `mediaType` that is unknown SHOULD be safely ignored.

- **`platform`** *object*

This REQUIRED property describes the platform which the image in the manifest runs on.
This OPTIONAL property describes the platform which the image in the manifest runs on.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we make it as an optional property? Without this object, if a manifest list have multiple manifests, how can we know which platform (e.g., amd64, ppc64le) each manifest is for?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For those cases, it will be present. There is a lot of discussion above, but the idea is that there might be companion data that is not platform-specific.

@vbatts We may want to bolster this with something saying "This property SHOULD be present if its target is platform-specific." Maybe, even a must?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

alrighty

This property SHOULD be present if its target is platform-specific.

- **`architecture`** *string*

Expand Down Expand Up @@ -64,13 +69,6 @@ For the media type(s) that this document is compatible with, see the [matrix][ma

This OPTIONAL property specifies an array of strings, each specifying a mandatory CPU feature (for example `sse4` or `aes`).

- **`annotations`** *string-string map*

This OPTIONAL property contains arbitrary metadata for the manifest list.
This OPTIONAL property MUST use the [annotation rules](annotations.md#rules).

See [Pre-Defined Annotation Keys](annotations.md#pre-defined-annotation-keys).

## Example Manifest List

*Example showing a simple manifest list pointing to image manifests for two platforms:*
Expand Down Expand Up @@ -107,5 +105,5 @@ For the media type(s) that this document is compatible with, see the [matrix][ma
}
```

[runtime-platform2]: https://github.com/opencontainers/runtime-spec/blob/v1.0.0-rc2/config.md#platform
[runtime-platform2]: https://github.com/opencontainers/runtime-spec/blob/v1.0.0-rc3/config.md#platform
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we bump this to rc3, we probably want to update the config reference as well. But we can probably leave the rc2 → rc3 bump out of this PR.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i've updated the other references now.

[matrix]: media-types.md#compatibility-matrix
4 changes: 4 additions & 0 deletions schema/content-descriptor.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,10 @@
"type": "string",
"format": "uri"
}
},
"annotations": {
"id": "https://opencontainers.org/schema/image/descriptor/annotations",
"$ref": "defs-image.json#/definitions/annotations"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This catches us up with #501 (and it needs an fs.go bump too). With the spec change already hashed out in #501, this can probably get spun off into its own PR and landed faster than the index pivot as a whole.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i've updated the fs.go now. But what needs to get spun off?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nothing needs to get spun off. But adding annotations here is independent of the rest of this PR, so you could spin this whole hunk off into a separate PR if you wanted more, smaller PRs (vs. fewer, bigger PRs).

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh right. It could be independent. This PR makes an example of it though. So I fixed what I saw.

}
},
"required": [
Expand Down
7 changes: 5 additions & 2 deletions schema/defs-image.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,8 +17,7 @@
"required": [
"mediaType",
"size",
"digest",
"platform"
"digest"
],
"properties": {
"mediaType": {
Expand Down Expand Up @@ -78,6 +77,10 @@
}
}
}
},
"annotations": {
"id": "https://opencontainers.org/schema/image/descriptor/annotations",
"$ref": "#/definitions/annotations"
}
}
},
Expand Down
Loading