docs: tutorial/example for ngclient Updater

[jku]

EDIT: I think starting with out-of-the-box runnable example code is a better idea: see comment below (original issue follows:)

the updater module docs show how to write a client in five lines of code but they also mention dependencies:

    * A webserver running on http://localhost:8000, serving TUF repository
      metadata at "/tuf-repo/" and targets at "/targets/"
    * Local metadata directory "~/tufclient/metadata/" is writable and contains
      a root metadata version for the remote repository
    * Download directory "~/tufclient/targets/" is writable

would be nice to have really easy to follow instructions that handle all of that...

The tutorial should live within the documentation but not in the ngclient module docs, just as a page on its own. Actually writing the tutorial is probably a "good first issue"?

[jku]

I'm marking this good first issue even if it's not completely trivial: if you're interested, feel free to ask for a bit of advice.

[jku]

The alternative to a tutorial might be to write a small example client that can just be executed, and short documentation for it -- the content is going to be roughly the same but I feel like an example is more useful?

Possible initial implementation using the tests repository data

• client python code in examples/example-client.py
• examples/metadata/root.json copied from tests repository
• examples/targets/ for downloading targets
• client could have a few features like "list targets", "download target", "clear cache" (to remove local targets and metadata except root) -- but just download or list is fine for the first version
• examples/example-repository.py would just serve the test repository (tests/repository_data/repository/) over http
• Then the documentation is just "run 'example-repository.py' in one window and 'example-client.py list-targets' in another"

I would like to have a more featureful repo example (that would actually show the cool features of TUF) but this would be a start to show the client.

[kairoaraujo]

I'm picking up this issue.

[jku]

I got into potential future plans in chat, I think they look good enough to include here -- most of it not as something to implement as part of initial example but as goals for future:

so these potential variations of example code setups would be pretty much the same for the client part, what changes is the remote end: where is the metadata (and targets) served and how are they generated. Obviously the decisions affect the client in the sense that some client features would only make sense in some of these cases.

• local static server: python -m http.server -d tests/repository_data/repository/ -- use static metadata that already exists in python-tuf tests, serve it locally using http.server. This is a one-liner but does not show the cool things in TUF.
• local dynamic server: Use Metadata API to generate metadata from scratch, write a small server to serve it locally: user would still run this on their own machine. This allows some nice possibilities like
  • updating the timestamp every 30 seconds so client sees things changing as they update
  • making the server react to file changes in a directory: copy a file into targets directory, and the server updates metadata automatically to include that file as a target
  • etc ...
• Run an actual TUF repository somewhere on the internet. This is most complex but also feels much, much more real. In this case user does not need to start any servers on localhost: just run the client example and watch it securely download files from the internet
  • this requires infrastructure to run the timestamp and snapshot processes (see github workflows in https://github.com/jku/tuf-repo-test for a taste)
  • this also requires some content flow: something needs to be added into the metadata once in a while -- it could be python-tuf release files during the release process but could be something completely different, possibly automated, as well
  • note that actual target files don't need to be included in the repository: e.g. python-tuf release files can be pypi.org hosted, we just serve the metadata for the TUF repo
  • this might be doable with github actions in python-tuf repo deploying TUF metadata to e.g. python-tuf.github.io (which is currently unused)
  • even with the maintenance overhead I kind of like this because it would give us valuable first-hand info on running a repository

[kairoaraujo]

Before submitting the PR, I am sending a design proposal/draft for local static server

The idea is to start with a simple document describing how to run an example of a client code using the ngclient and the static metadata from tests/repository_data/repository/.

The document will have three basic steps.

1. Run the HTTP Server using the tests data repository
2. How to initialize the client to prepare the client infrastructure
3. Download files to show different scenarios
   In that step, we will highlight the TUF/ngclient behaviors.

The client code example:

• The example will be stored inside the examples/ in the root folder. The reason is that some people check examples, even before the documentation.

• The idea is a commented code, including references to the API and/or the Specification.

• The initial name suggested will be client_example as a small Python Command Line Interface (CLI) executable/script.

  • The metadata repository directory will be in $HOME/.local/share/<app-name-here>/
  • The download folder directory will be in download (local directory)

• The example will be a simple CLI with initial two parameters:

  • --init
    This parameter will create the client's infrastructure, the metadata folder (including the initial root.json), and the download folder.
    Running this example will be explicit about all the steps. The CLI will print the messages.

  ./client_example --init
  [INFO] metadata folder [<path>] created.
  [INFO] Bootstrap 'root.json' with a trusted source.
  [INFO] Download folder  ['downloads'] created.
  [INFO] Client initialization finished.

  The second scenario is the client already initialized, but the messages above will be the same.

  • --download <filename>.
    Along with the how-to documentation, the instruction will be to download different targets.
    The goal is to demonstrate flow, the terminology used in the framework, using the steps messages (printing as log), such as:

  ./client_example --download file.txt
  [INFO] Updating metadata from remote.
  [INFO] Getting target info.
  [INFO] Checking if the target <name> is cached.
  [INFO] Downloading target <name> to download <path name>.
  [INFO] File downloaded successfully.

[jku]

on the CLI design: looks good. Download could be a sub-command as well (so download file.txt instead of --download file.txt), then combining with --init would maybe look better: ./client_example --init download file1.txt.

I'm mostly thinking about if you add more commands later (like list-files) then if they are both options it looks like you could combine them (--list-files --download file1.txt) which is probably not what we want: if they are commands it's clear you should only give one command at a time.

One thing we didn't discuss is how do we handle the server startup in this first iteration? the client example could of course start a new local http server in the background each time it runs... but maybe the documentation just includes advice to run python3 -m http.server ... in another terminal? In the latter case you just need to make the port selection somehow trivial -- I don't think we want the user copy pasting port numbers... maybe hard code a specific port? any better ideas?

[jku]

That said, maybe we should not provide an example of --init at all (I'm not 100% sure if you meant downloading the initial root.json over http in a Trust-On-First-Use model)? Maybe making a copy of the root.json in the examples directory and bootstrapping the local metadata directory from there (in other words copying the file to local metadata dir) is a more responsible example?

If we do that, then I think --init can be implicit: if the root.json does not exist in the correct dir, copy it over

[kairoaraujo]

on the CLI design: looks good. Download could be a sub-command as well (so download file.txt instead of --download file.txt), then combining with --init would maybe look better: ./client_example --init download file1.txt.

I think using sub-commands also looks good, and it is something usual now, also helps to extend.
IMO, having --init as a parameter in combination can be valuable, but is it instructive? I will add but not in the documentation explicitly; however, we can change it easily in the doc only.

I'm mostly thinking about if you add more commands later (like list-files) then if they are both options it looks like you could combine them (--list-files --download file1.txt) which is probably not what we want: if they are commands it's clear you should only give one command at a time.

One thing we didn't discuss is how do we handle the server startup in this first iteration? the client example could of course start a new local http server in the background each time it runs... but maybe the documentation just includes advice to run python3 -m http.server ... in another terminal? In the latter case you just need to make the port selection somehow trivial -- I don't think we want the user copy pasting port numbers... maybe hard code a specific port? any better ideas?

I am planning to have it in the documentation as a kind of how-to...

The draft I am working on

Preparing

To have the example working in your machine, clone the python-tuf in your system.

$ git clone [email protected]:theupdateframework/python-tuf.git

Repository

As this example demonstrates how to use the python-tuf to build a client application, the repository will use static files.

The static files are available in the python-tuf repository, same as this
The static repository files are in
tests/repository_data/repository.

Run the repository using the Python3 built-in http module, and keep this
session running.

$ python3 -m http.server -d tests/repository_data/repository
Serving HTTP on :: port 8000 (http://[::]:8000/) ...

(...)

[kairoaraujo]

That said, maybe we should not provide an example of --init at all (I'm not 100% sure if you meant downloading the initial root.json over http in a Trust-On-First-Use model)? Maybe making a copy of the root.json in the examples directory and bootstrapping the local metadata directory from there (in other words copying the file to local metadata dir) is a more responsible example?

If we do that, then I think --init can be implicit: if the root.json does not exist in the correct dir, copy it over

Yes, I was thinking about it this morning when I was writing the code.
Bootstrapping from a copy stored with the example is best in the sense of giving the best practice IMO.

[kairoaraujo]

The initial version for the client example was merged by PR #1675. I will add here some next steps commented in this PR.

I think we can continue the discussion here and create new issues (smaller issues) to track them.

From this comment #1675 (review) (thanks @jku for the insights)

• Add the examples/client_example to RTD #1716
• Create simple test for examples/client_example #1717
• support other server locations than hard-coded localhost:8000
• support boostrapping root.json for other servers (like the repository generated by the repository example) somehow
• document the application decisions better (as an example "why is is local metadata stored in $HOME/.local/share/", "why - - not download first root.json over http?"). I didn't do this in the review as I think it's a tricky balance: too much detail means no-one will read it...
• start thinking about how to use examples to show that TUF actually works, and prevents attacks. This might be more related to repository examples but ...

[jku]

Thanks. I'd like to close this one to mark that the dependency for 1694 is handled...

Maybe it's ok that the main missing features are issues and remaining ideas are searchable here... or you could file another issue for the "support other servers" issue.

Closing this one, feel free to file new issues for the other ideas