Support placeholders for files.

[jsirois]

The idea is to extend placeholders from parameterizing commands to also parameterizing files.

Like so:

{
  "scie": {
    "lift": {
      "commands": {
        "exe": "{node}/node-v18.12.0-linux-arm64/bin/node"
      },
      "bindings": {
        "fetch": {
          "exe": "{ptex}"
        }
      },
      "files": [
        {
          "name": "ptex"
        },
        {
          "name": "node-v18.12.0-linux-arm64.tar.xz={scie.file.fetch}",
          "key": "node",
          "hash": "...",
          "size": "...",
          "type": "tar.xz"
        }
      ]
    }
  },
  "ptex": {
    "node-v18.12.0-linux-arm64.tar.xz": "https://nodejs.org/dist/v18.12.0/node-v18.12.0-linux-arm64.tar.xz"
  }
}

The idea is a new placeholder {scie.file.<binding>}. A file whose name ends in ={scie.file.<binding>} will not be processed by the boot-pack but instead be processed at install time. The protocol would be something like:

./<binding> <file json blob with stripped name> > ~/.nce/<hash>/<stripped name>

So if the binding command can accept file json and emit that file on stdout, that's all that needed to hydrate files from elsewhere than the scie itself. Since the config is strict for the scie lift manifest, but agnostic for other top-level keys, this allows the configured fetch binding to store extra metadata about where the file should be fetched from. This would allow an application like ptex to be written seperate from the scie-jump that could be included in scies to turn them into ptexes - portable thin executables.

Twitter had a use case for this in the context of distributing very large PEXes in their datacenters. If the scie conceptually contained a packed layout PEX, it could be shipped as a ptexed scie that just had the scie-jump and ptex binary blobs inside with all the rest of the application content, up to and including the Python interpreter, provided lazily by ptex on 1st boot.

With the syntax approach sketched out here the config format stays the same - only a new placeholder is introduced; so this idea can be deferred without worry of breaking the format.

[jsirois]

This still needs some doc before closing.

[jsirois]

I'm not sure if scie-jump will house specific docs for ptex, but the ptex docs are up for review in a-scie/ptex#11.