Add additional channel manifest metadata

[brson]

The current channel manifest is just a listing of files. It's already apparent that more metadata is needed. Right now multirust needs to know the archive date of nightlies, and the full version number (as written in rustc's 'version' file) would also be helpful.

Unfortunately I failed to version the existing manifest format so we probably need yet another file, say channel-rust-nightly-data that contains this extra metadata. This time make it versioned.

[brson]

Also add a version file to the cargo and rust installers in addition to the rustc one. That will allow the channel manifests for cargo to include the version.

[brson]

Essentially what should happen here is the following:

• Each installer needs to contain a version file that contains the version. The rustc build already has this file (see code in mk/dist.mk). The Cargo build also needs to produce this file. Finally, rust-packaging needs to be modified to copy the one from rustc over to the combined rust+cargo package.
• Once all our 'products' have version files, modify rust-buildbot's finish_dist method to create the channel-$product-nightly-data file.

This file needs to be parsable from shell scripts so I suggest a simple line-oriented key-value format, maybe like:

metadata-version: 1
product-version: (whatever the contents of the version file)
archive-date: (the date in the archives the artifacts are being uploaded to)

[brson]

Justification: the version number will help tools know what version is available w/o a download; the archive-date lets tools correlate the build currently-available on the release channels with the build on the archives, so e.g. somebody with today's nightly can know that they also have the nightly from today's archive date.

[steveklabnik]

Triage: I know metadata has changed, I'm not quite sure how.

[eternaleye]

How it has changed:

There is now a channel-rust-nightly.toml, which is generated by rust-buildbot/build-rust-manifest.py

This has (at the top level):

• A manifest-version, currently "2"
• The date it corresponds to
• a pkg dict/map, the keys of which are packages names like "cargo" or "rust"

Packages have:

• a version key, which looks like "0.10.0-nightly (fae9c53 2016-03-31)" or "1.8.0-beta.2 (2879d94 2016-03-22)"
• a target map, the keys of which are target triples like arm-unknown-linux-gnueabi

Targets have two possible forms, one in which the "available" key is true, and the other in which it is false.

If the target is available:

• The url key contains a link from which the package can be downloaded
• The hash key contains a hash of the file that the URL refers to

Note that there is no hash-type key, or any other namespacing; if anything other than SHA-256 (the current hash) is to be used, this would need to change.

It would probably be a good idea to either add one, or declare that hash will always be SHA-256 for legacy reasons, and use a hashes map from hash type to hash value in the future.

Such a change would be backwards-compatible so long as hash was still present, avoiding the need to bump the manifest-version.

If the target is not available, both keys are blank.

Independent of being available, a target may have a components array and an extensions array, the elements of which have a pkg key and optionally a target key (which if unset, has the same value as the target it occurs in). The referenced package target is then treated as either a component or extension of the one it is listed by.

The distinction between components and extensions seems to be that components are installed by default with the referencing target, while extensions are not.

[alexcrichton]

We've now done this with v2 manifests, so closing