### Step-by-step instruction
-will be here
+Fork the repo first. Make sure you uncheck "Copy the `main` branch only" to copy
+Git tags as well:
+
+
+Then enable workflows in your repo, for a Git tag to trigger CI:
+
-Let's do the same thing we did locally, but for your remote repo:
+Let's do the same thing we did locally, but for your remote repo. Don't forget
+to replace the URL:
```cli
$ gto assign cv-class --version v0.1.13 --stage dev \
@@ -118,7 +124,10 @@ this information, the step `Deploy (act on assigning a new stage)` was executed
### CI/CD execution example
-screenshot goes here
+
+
+Then enable workflows in your repo, for a Git tag to trigger CI:
+
+
+If you want to see more CI examples, check out
+[the example-repo](https://github.com/iterative/example-gto/actions).
diff --git a/content/docs/gto/user-guide/index.md b/content/docs/gto/user-guide/index.md
index 0a38de29..81f02544 100644
--- a/content/docs/gto/user-guide/index.md
+++ b/content/docs/gto/user-guide/index.md
@@ -1,5 +1,12 @@
# User Guide
+GTO helps you build a Artifact Registry out of your Git repository. It does so
+by creating Git tags of [special format](/doc/gto/user-guide) and managing
+[`artifacts.yaml` metafile](/doc/gto/user-guide). Since committing large files
+to Git is not a good practice, you shouldn't commit binary files to Git.
+Instead, use
+[DVC, Git-lfs, or any method commit pointers to the binary files instead](/doc/gto/user-guide/storages).
+
Our guides describe the major concepts in GTO and how it works comprehensively,
explaining when and how to use what, as well as inter-relationship between them.
From e3b90bdb14993c410cf6982d0728b21a54295a94 Mon Sep 17 00:00:00 2001
From: Alexander Guschin <1aguschin@gmail.com>
Date: Tue, 1 Nov 2022 22:26:05 +0600
Subject: [PATCH 10/38] add Git tags format
---
content/docs/gto/user-guide/git-tags.md | 24 +++++++++++++++++++++++-
1 file changed, 23 insertions(+), 1 deletion(-)
diff --git a/content/docs/gto/user-guide/git-tags.md b/content/docs/gto/user-guide/git-tags.md
index 8b8912ca..589141b4 100644
--- a/content/docs/gto/user-guide/git-tags.md
+++ b/content/docs/gto/user-guide/git-tags.md
@@ -1,3 +1,25 @@
# Git tags format
-...describe git tags format and it details...
+Each event (version registration, stage assignment, artifact deprecation) has
+it's own standard format of Git tags.
+
+All of them share two parts:
+
+1. `{artifact}` prefix part.
+2. `#{e}` counter at the end that can be omitted ("simple" Git tag format).
+
+Generally, `#{e}` counter is used, because Git doesn't allow to create two Git
+tags with the same name. If you want to have two Git tags that assign `dev`
+stage to `model` artifact without the counter (`model#dev`), that will require
+deleting the old Git tag first. Consequently, that doesn't allow you to preserve
+history of events that happened.
+
+By default, `#{e}` sometimes is omitted, sometimes not. We are setting defaults
+to omit using `#{e}` when it's rarely necessary, e.g. for version registrations
+and artifact deprecations.
+
+Git tag formats:
+
+- `{artifact_name}@{version_number}#{e}` for Version registration.
+- `{artifact_name}#{stage}#{e}` for Stage assignment.
+- `{artifact_name}@deprecated#{e}` for Artifact deprecation.
From eff9755d021a3376d312d73595d9d0554b564a9c Mon Sep 17 00:00:00 2001
From: Alexander Guschin <1aguschin@gmail.com>
Date: Tue, 1 Nov 2022 22:34:26 +0600
Subject: [PATCH 11/38] refine Git tags format
---
content/docs/gto/user-guide/git-tags.md | 17 ++++++++---------
1 file changed, 8 insertions(+), 9 deletions(-)
diff --git a/content/docs/gto/user-guide/git-tags.md b/content/docs/gto/user-guide/git-tags.md
index 589141b4..b4bacb7e 100644
--- a/content/docs/gto/user-guide/git-tags.md
+++ b/content/docs/gto/user-guide/git-tags.md
@@ -1,12 +1,17 @@
# Git tags format
-Each event (version registration, stage assignment, artifact deprecation) has
-it's own standard format of Git tags.
+Each event has it's own standard format of Git tags:
+
+- `{artifact_name}@{version_number}#{e}` for version registration.
+- `{artifact_name}@{version_number}!#{e}` for version deregistration.
+- `{artifact_name}#{stage}#{e}` for stage assignment.
+- `{artifact_name}#{stage}!#{e}` for stage unassignment.
+- `{artifact_name}@deprecated#{e}` for artifact deprecation.
All of them share two parts:
1. `{artifact}` prefix part.
-2. `#{e}` counter at the end that can be omitted ("simple" Git tag format).
+2. `#{e}` counter at the end that can be omitted (in "simple" Git tag format).
Generally, `#{e}` counter is used, because Git doesn't allow to create two Git
tags with the same name. If you want to have two Git tags that assign `dev`
@@ -17,9 +22,3 @@ history of events that happened.
By default, `#{e}` sometimes is omitted, sometimes not. We are setting defaults
to omit using `#{e}` when it's rarely necessary, e.g. for version registrations
and artifact deprecations.
-
-Git tag formats:
-
-- `{artifact_name}@{version_number}#{e}` for Version registration.
-- `{artifact_name}#{stage}#{e}` for Stage assignment.
-- `{artifact_name}@deprecated#{e}` for Artifact deprecation.
From 65f66ee73497caf9b6160592a9c1831f87dd8360 Mon Sep 17 00:00:00 2001
From: Alexander Guschin <1aguschin@gmail.com>
Date: Tue, 1 Nov 2022 23:25:03 +0600
Subject: [PATCH 12/38] write page: artifacts.yaml metafile
---
content/docs/gto/user-guide/artifacts.md | 59 +++++++++++++++++++-----
content/docs/gto/user-guide/git-tags.md | 2 +-
2 files changed, 49 insertions(+), 12 deletions(-)
diff --git a/content/docs/gto/user-guide/artifacts.md b/content/docs/gto/user-guide/artifacts.md
index a99aef3b..97be377d 100644
--- a/content/docs/gto/user-guide/artifacts.md
+++ b/content/docs/gto/user-guide/artifacts.md
@@ -1,14 +1,51 @@
-# `artifacts.yaml` metafile
+# artifacts.yaml metafile
-Registering versions and assigning stages is handy since you can trigger a
-CI/CD, but the Git tag itself doesn't contain path to the artifact, type of it
-(it could be `model` or `dataset`), or any other information you may find
-useful. For simple projects (e.g. single artifact) we can assume the details in
-a downstream system. But for more advanced cases, we should codify them in the
-registry itself.
+Using Git tag to register versions and assign stages is handy, but the Git tag
+itself doesn't contain path to the artifact, type of it (it could be `model` or
+`dataset`), or any other information you may find useful. For simple projects
+(e.g. single artifact) we can assume the details in a downstream system. But for
+more advanced cases, we should codify them in the registry itself.
-To keep this metainformation, GTO uses `artifacts.yaml` file and commands like
-`gto annotate`. Please refer to [User Guide](/doc/gto/user-guide). for more
-details.
+To keep this metainformation, GTO uses `artifacts.yaml` file. Commands like
+[`annotate`](/doc/gto/command-reference/annotate) and
+[`remove`](/doc/gto/command-reference/remote) are used to modify it, while
+[`describe`](/doc/gto/command-reference/describe) helps get them when they're
+needed.
-...tell about how to annotate and remove, about the file structure...
+Let's see an example of `artifacts.yaml` (another example can be found in
+[example-gto](https://github.com/iterative/example-gto/blob/main/artifacts.yaml)):
+
+```yaml
+awesome:
+ description: This model is very cool
+ labels:
+ - ml
+ - cool
+ path: models/awesome.pkl
+ type: model
+```
+
+By default GTO saves artifact as `virtual`. Use the `--must_exist` flag to tell
+GTO the artifact file is committed to Git.
+
+
+
+### Virtual vs. Physical artifacts
+
+- Physical files/directories are committed to the repo. When you create a new
+ version or assign a stage to it, Git guarantees that it's immutable -- you can
+ return a year later and get the same artifact by providing a version.
+
+- Virtual artifacts could be an external path (e.g. `s3://mybucket/myfile`) or a
+ local path to a metafile representing an externally stored artifact file (as
+ [with DVC](https://dvc.org/doc/start/data-management)). In this case, GTO
+ can't pin versions to a physical state of the artifact and guarantee it's
+ immutability later, e.g. if `s3://mybucket/myfile` changes the registry won't
+ know it, nor have a way to recover the original file.
+
+> In future versions, we will support additional enrichments: useful information
+> that other tools like [DVC](https://dvc.org/) and [MLEM](https://mlem.ai/) can
+> provide about the artifacts. This will allow treating DVC repo outputs as
+> usual artifacts instead of `virtual` ones.
+
+
diff --git a/content/docs/gto/user-guide/git-tags.md b/content/docs/gto/user-guide/git-tags.md
index b4bacb7e..448ae8c3 100644
--- a/content/docs/gto/user-guide/git-tags.md
+++ b/content/docs/gto/user-guide/git-tags.md
@@ -1,6 +1,6 @@
# Git tags format
-Each event has it's own standard format of Git tags:
+All events have the standard formats of Git tags:
- `{artifact_name}@{version_number}#{e}` for version registration.
- `{artifact_name}@{version_number}!#{e}` for version deregistration.
From 129b17fbc3752e5ffd9596c755baa25373622389 Mon Sep 17 00:00:00 2001
From: Alexander Guschin <1aguschin@gmail.com>
Date: Wed, 2 Nov 2022 16:55:53 +0600
Subject: [PATCH 13/38] user-guide/{artifacts,config,downstream}
---
content/docs/gto/user-guide/artifacts.md | 2 +-
content/docs/gto/user-guide/config.md | 19 ++++++++++-
content/docs/gto/user-guide/downstream.md | 40 ++++++++++++++++++++++-
3 files changed, 58 insertions(+), 3 deletions(-)
diff --git a/content/docs/gto/user-guide/artifacts.md b/content/docs/gto/user-guide/artifacts.md
index 97be377d..a43683d6 100644
--- a/content/docs/gto/user-guide/artifacts.md
+++ b/content/docs/gto/user-guide/artifacts.md
@@ -1,4 +1,4 @@
-# artifacts.yaml metafile
+# Annotations in artifacts.yaml
Using Git tag to register versions and assign stages is handy, but the Git tag
itself doesn't contain path to the artifact, type of it (it could be `model` or
diff --git a/content/docs/gto/user-guide/config.md b/content/docs/gto/user-guide/config.md
index e3dfa00e..cbac3235 100644
--- a/content/docs/gto/user-guide/config.md
+++ b/content/docs/gto/user-guide/config.md
@@ -1,3 +1,20 @@
# Configuring types and stages
-...tell about limitations in config to allowed types and stages...
+To configure GTO, use file `.gto` in the root of your repo:
+
+```yaml
+# .gto config file
+types: [model, dataset] # list of allowed Types
+stages: [dev, stage, prod] # list of allowed Stages
+```
+
+When allowed Stages or Types are specified, GTO will check commands you run and
+error out if you provided a value that doesn't exist in the config. Note, that
+GTO applies the config from the workspace, so if want to apply the config from
+`main` branch, you need to check it out first with `git checkout main`.
+
+Alternatively, you can use environment variables (note the `GTO_` prefix)
+
+```cli
+$ GTO_EMOJIS=false gto show
+```
diff --git a/content/docs/gto/user-guide/downstream.md b/content/docs/gto/user-guide/downstream.md
index b1699f13..75ee7601 100644
--- a/content/docs/gto/user-guide/downstream.md
+++ b/content/docs/gto/user-guide/downstream.md
@@ -1,3 +1,41 @@
# Acting downstream
-...tell about CI/CD, webhooks, and external services watching the repo...
+Once Git tag was pushed, you can start acting downstream. One of the popular
+options here is to use CI/CD, setting it to be triggered by pushing a Git tag.
+For general details, check out something like
+[GitHub Actions](https://github.com/features/actions),
+[GitLab CI/CD](https://docs.gitlab.com/ee/ci/) or
+[Circle CI](https://circleci.com).
+
+The other option is to
+[configure webhooks](https://docs.github.com/en/rest/webhooks) that will send
+HTTP requests to your server upon pushing Git tags to the remote.
+
+Finally, you can configure your server to query your Git provider via something
+like REST API to check if changes happened. As an example, check out
+[Github REST API](https://docs.github.com/en/rest).
+
+## Getting started with CI/CD
+
+To act upon registrations and assignments (Git tags), you can create simple CI
+workflow. Check out
+[the example workflow in `example-gto` repo](https://github.com/iterative/example-gto/blob/main/.github/workflows/gto-act-on-tags.yml).
+The workflow uses [the GTO GH Action](https://github.com/iterative/gto-action)
+that fetches all Git tags (to correctly interpret the Registry) and finds out
+the version of the artifact that was registered, or the stage that was assigned,
+so you could use them in later steps of the CI.
+
+
+## Inspecting Git tags
+
+You can use `gto check-ref` to interpret the Git tag:
+
+```console
+$ gto check-ref -r build/example-gto churn#prod#3
+✅ Stage "prod" was assigned to version "v3.0.0" of artifact "churn"
+```
+
+For machine-consumable format, use `--json` flag or output specific pieces of
+information with `--name`, `--version`, `--stage` or `--event`.
+
+TODO: add `gto describe` here.
From c596dc96a9a72f48c5c8cf14bc71b49de89f192e Mon Sep 17 00:00:00 2001
From: Alexander Guschin <1aguschin@gmail.com>
Date: Wed, 2 Nov 2022 16:56:17 +0600
Subject: [PATCH 14/38] fix lint
---
content/docs/gto/user-guide/config.md | 4 ++--
content/docs/gto/user-guide/downstream.md | 1 -
2 files changed, 2 insertions(+), 3 deletions(-)
diff --git a/content/docs/gto/user-guide/config.md b/content/docs/gto/user-guide/config.md
index cbac3235..8a44acb4 100644
--- a/content/docs/gto/user-guide/config.md
+++ b/content/docs/gto/user-guide/config.md
@@ -4,8 +4,8 @@ To configure GTO, use file `.gto` in the root of your repo:
```yaml
# .gto config file
-types: [model, dataset] # list of allowed Types
-stages: [dev, stage, prod] # list of allowed Stages
+types: [model, dataset] # list of allowed Types
+stages: [dev, stage, prod] # list of allowed Stages
```
When allowed Stages or Types are specified, GTO will check commands you run and
diff --git a/content/docs/gto/user-guide/downstream.md b/content/docs/gto/user-guide/downstream.md
index 75ee7601..fecafd82 100644
--- a/content/docs/gto/user-guide/downstream.md
+++ b/content/docs/gto/user-guide/downstream.md
@@ -25,7 +25,6 @@ that fetches all Git tags (to correctly interpret the Registry) and finds out
the version of the artifact that was registered, or the stage that was assigned,
so you could use them in later steps of the CI.
-
## Inspecting Git tags
You can use `gto check-ref` to interpret the Git tag:
From 9f42acea59515634a1e896503c8cd03e932d9ef9 Mon Sep 17 00:00:00 2001
From: Alexander Guschin <1aguschin@gmail.com>
Date: Wed, 2 Nov 2022 17:24:09 +0600
Subject: [PATCH 15/38] fixes on feedback
---
content/docs/gto/get-started.md | 34 ++++++++++++------------
content/docs/gto/user-guide/artifacts.md | 2 +-
2 files changed, 18 insertions(+), 18 deletions(-)
diff --git a/content/docs/gto/get-started.md b/content/docs/gto/get-started.md
index 7575409d..947bbdaa 100644
--- a/content/docs/gto/get-started.md
+++ b/content/docs/gto/get-started.md
@@ -5,12 +5,12 @@ description:
# Get Started
-GTO helps you build an Artifact Registry on top of a Git repository (with a
-special case of Machine Learning Model Registry). You can register relevant
-versions of your files (e.g. ML model releases) and assign them to different
-deployment environments (testing, shadow, production, etc.). Git-native
-mechanisms are used, so you can automate the delivery of your ML project with
-CI/CD, and adopt a GitOps approach in general.
+GTO helps you build an Artifact Registry on top of a Git repository. You can
+register relevant versions of your files or directories from any source (e.g. ML
+model releases) and assign them to different deployment environments (testing,
+shadow, production, etc.). Git-native mechanisms are used, so you can automate
+the delivery of your software project with CI/CD, and adopt a GitOps approach in
+general.
This Get Started will walk you through basic GTO concepts and actions you would
like to do in the Artifact Registry.
@@ -25,8 +25,8 @@ $ git clone https://github.com/iterative/example-gto
$ cd example-gto
```
-This repo represents a simple example of Machine Learning Model Registry. Let's
-review it:
+This repo represents a simple example of Machine Learning Model Registry (that
+is just a special case of an Artifact Registry for us). Let's review it:
```cli
$ gto show
@@ -87,20 +87,20 @@ The power of using Git tags to register versions and assign stages is simple: we
can act upon them in well-known way - in CI/CD.
To see how it works, let's fork the
-[example-gto repo](https://github.com/iterative/example-gto/fork) and push the
-tag we just created to GitHub. For CI/CD to start, you'll need to enable them on
-the "Actions" page of your fork.
+[example-gto repo](https://github.com/iterative/example-gto) and push the tag we
+just created to GitHub. For CI/CD to start, you'll need to enable them on the
+"Actions" page of your fork.
-### Step-by-step instruction
+### Step-by-step instruction to fork the repo and enable Github workflows
-Fork the repo first. Make sure you uncheck "Copy the `main` branch only" to copy
-Git tags as well:
-
+1. [Fork the repo](https://github.com/iterative/example-gto/fork). Make sure you
+ uncheck "Copy the `main` branch only" to copy Git tags as well:
+
-Then enable workflows in your repo, for a Git tag to trigger CI:
-
+2. Enable workflows in your repo, for a Git tag to trigger CI:
+
diff --git a/content/docs/gto/user-guide/artifacts.md b/content/docs/gto/user-guide/artifacts.md
index a43683d6..d00776b5 100644
--- a/content/docs/gto/user-guide/artifacts.md
+++ b/content/docs/gto/user-guide/artifacts.md
@@ -16,7 +16,7 @@ Let's see an example of `artifacts.yaml` (another example can be found in
[example-gto](https://github.com/iterative/example-gto/blob/main/artifacts.yaml)):
```yaml
-awesome:
+awesome: # artifact name
description: This model is very cool
labels:
- ml
From 931dac19f10a3fc62261abbf121f3ab910276dc7 Mon Sep 17 00:00:00 2001
From: Alexander Guschin <1aguschin@gmail.com>
Date: Thu, 10 Nov 2022 18:15:12 +0600
Subject: [PATCH 16/38] Apply suggestions from code review
Co-authored-by: Jorge Orpinel
+1. [Fork the repo](https://github.com/iterative/example-gto/fork). Make sure you
+ uncheck "Copy the `main` branch only" to copy Git tags as well:
+
-Then enable workflows in your repo, for a Git tag to trigger CI:
-
+2. Enable workflows in your repo, for a Git tag to trigger CI:
+
-### Step-by-step instruction to fork the repo and enable Github workflows
+### Click to set up a working repository to try this.
+
+Let's fork the
+[example-gto repo](https://github.com/iterative/example-gto) repo
+(you'll need a [GitHub](https://github.com/signup) account first). For
+CI/CD to start, you'll need to enable it on the "Actions" page of your
+fork.
1. [Fork the repo](https://github.com/iterative/example-gto/fork). Make sure you
uncheck "Copy the `main` branch only" to copy Git tags as well:
diff --git a/content/docs/gto/user-guide/artifacts.md b/content/docs/gto/user-guide/artifacts.md
index d00776b5..095f1acf 100644
--- a/content/docs/gto/user-guide/artifacts.md
+++ b/content/docs/gto/user-guide/artifacts.md
@@ -8,7 +8,7 @@ more advanced cases, we should codify them in the registry itself.
To keep this metainformation, GTO uses `artifacts.yaml` file. Commands like
[`annotate`](/doc/gto/command-reference/annotate) and
-[`remove`](/doc/gto/command-reference/remote) are used to modify it, while
+[`remove`](/doc/gto/command-reference/remove) are used to modify it, while
[`describe`](/doc/gto/command-reference/describe) helps get them when they're
needed.
@@ -39,7 +39,7 @@ GTO the artifact file is committed to Git.
- Virtual artifacts could be an external path (e.g. `s3://mybucket/myfile`) or a
local path to a metafile representing an externally stored artifact file (as
[with DVC](https://dvc.org/doc/start/data-management)). In this case, GTO
- can't pin versions to a physical state of the artifact and guarantee it's
+ can't pin versions to a physical state of the artifact and guarantee its
immutability later, e.g. if `s3://mybucket/myfile` changes the registry won't
know it, nor have a way to recover the original file.
diff --git a/content/docs/gto/user-guide/git-tags.md b/content/docs/gto/user-guide/git-tags.md
index 448ae8c3..ae1aba73 100644
--- a/content/docs/gto/user-guide/git-tags.md
+++ b/content/docs/gto/user-guide/git-tags.md
@@ -10,7 +10,7 @@ All events have the standard formats of Git tags:
All of them share two parts:
-1. `{artifact}` prefix part.
+1. `{artifact_name}` prefix part.
2. `#{e}` counter at the end that can be omitted (in "simple" Git tag format).
Generally, `#{e}` counter is used, because Git doesn't allow to create two Git
From 76dd0447d0531d6d41ad0072dac274e26fe7a61e Mon Sep 17 00:00:00 2001
From: Alexander Guschin <1aguschin@gmail.com>
Date: Thu, 10 Nov 2022 19:19:47 +0600
Subject: [PATCH 17/38] command-reference index page
---
content/docs/gto/command-reference/index.md | 25 ++++++++++++++++++---
content/docs/gto/get-started.md | 2 +-
content/docs/gto/user-guide/git-tags.md | 7 ++++++
3 files changed, 30 insertions(+), 4 deletions(-)
diff --git a/content/docs/gto/command-reference/index.md b/content/docs/gto/command-reference/index.md
index 0de40a34..1bbdd940 100644
--- a/content/docs/gto/command-reference/index.md
+++ b/content/docs/gto/command-reference/index.md
@@ -3,8 +3,27 @@
GTO is a command line tool. Here, we provide the specifications, complete
descriptions, and comprehensive usage examples for different `gto` commands.
-For a list of all commands, type `gto -h`
+For a list of all commands, type `gto -h`.
-## Typical GTO workflow
+## Read the registry
-...
+- Show the registry state, highest version, or what's assigned in stage with
+ `gto show`
+- Find out the artifact version registered/assigned with ref with
+ `gto check-ref`
+- Show a journal of registry operations with `gto history`
+- Print list of stages used in the registry with `gto stages`
+
+## Modify artifacts
+
+- Assign stage to specific artifact version with `gto assign`
+- Deprecate artifact, deregister a version, or unassign a stage with
+ `gto deprecate`
+- Create an artifact version to signify an important, published or released
+ iteration with `gto register`
+
+## Manage artifact enrichments
+
+- Update artifact metadata annotations with `gto annotate`
+- Display enrichments for an artifact with `gto describe`
+- Remove the enrichment for given artifact with `gto remove`
diff --git a/content/docs/gto/get-started.md b/content/docs/gto/get-started.md
index bbf4a225..6fadd09a 100644
--- a/content/docs/gto/get-started.md
+++ b/content/docs/gto/get-started.md
@@ -138,7 +138,7 @@ If you want to see more CI examples, check out
Thanks for completing this Get Started!
- If you want how to specify artifact's metainformation like `path`, `type` and
- `description`, check out [User Guide](/doc/gto/user-guide).
+ `description`, check out [User Guide](/doc/gto/user-guide/artifacts).
- If you want to learn about using DVC to keep your artifact binaries in remote
storages, check out [DVC docs](https://dvc.org/doc).
- If you want to learn more about Studio, check out
diff --git a/content/docs/gto/user-guide/git-tags.md b/content/docs/gto/user-guide/git-tags.md
index ae1aba73..44b4214b 100644
--- a/content/docs/gto/user-guide/git-tags.md
+++ b/content/docs/gto/user-guide/git-tags.md
@@ -1,5 +1,12 @@
# Git tags format
+
+
+You can work with GTO without knowing the used Git tag names conventions, since
+commands like `gto register` and `gto assign` abstracts that away from you.
+
+
+
All events have the standard formats of Git tags:
- `{artifact_name}@{version_number}#{e}` for version registration.
From f2e7af04e72f78b503adeb3168eaecd19e2c215f Mon Sep 17 00:00:00 2001
From: Alexander Guschin <1aguschin@gmail.com>
Date: Thu, 10 Nov 2022 20:53:12 +0600
Subject: [PATCH 18/38] first version of command reference
---
.../docs/gto/command-reference/annotate.md | 34 ++++++++++++
content/docs/gto/command-reference/assign.md | 46 ++++++++++++++++
.../docs/gto/command-reference/check-ref.md | 32 +++++++++++
.../docs/gto/command-reference/deprecate.md | 40 ++++++++++++++
.../docs/gto/command-reference/describe.md | 30 +++++++++++
content/docs/gto/command-reference/history.md | 33 ++++++++++++
.../docs/gto/command-reference/register.md | 47 ++++++++++++++++
content/docs/gto/command-reference/remove.md | 27 ++++++++++
content/docs/gto/command-reference/show.md | 50 +++++++++++++++++
content/docs/gto/command-reference/stages.md | 25 +++++++++
content/docs/gto/get-started.md | 9 ++--
content/docs/sidebar.json | 54 ++++++++++++++++++-
12 files changed, 420 insertions(+), 7 deletions(-)
create mode 100644 content/docs/gto/command-reference/annotate.md
create mode 100644 content/docs/gto/command-reference/assign.md
create mode 100644 content/docs/gto/command-reference/check-ref.md
create mode 100644 content/docs/gto/command-reference/deprecate.md
create mode 100644 content/docs/gto/command-reference/describe.md
create mode 100644 content/docs/gto/command-reference/history.md
create mode 100644 content/docs/gto/command-reference/register.md
create mode 100644 content/docs/gto/command-reference/remove.md
create mode 100644 content/docs/gto/command-reference/show.md
create mode 100644 content/docs/gto/command-reference/stages.md
diff --git a/content/docs/gto/command-reference/annotate.md b/content/docs/gto/command-reference/annotate.md
new file mode 100644
index 00000000..999cf5a9
--- /dev/null
+++ b/content/docs/gto/command-reference/annotate.md
@@ -0,0 +1,34 @@
+# annotate
+
+Update artifact metadata annotations
+
+## Synopsis
+
+```usage
+usage: gto annotate [-r ] [--type ] [--path ]
+ [-e] [--label ] [-d ] [--commit]
+ [--push] [-h]
+ name
+
+arguments:
+ name Artifact name
+```
+
+## Description
+
+## Options
+
+- `-r `, `--repo ` - Local or remote repository [default: .]
+- `--type ` - Artifact type
+- `--path ` - Artifact path
+- `-e`, `--must-exist` - Verify artifact is committed to Git
+- `--label ` - Labels to add to artifact
+- `-d `, `--description ` - Artifact description
+- `--commit` - Automatically commit changes due to this command (experimental)
+- `--push` - Push created commit automatically (experimental) - will set
+ commit=True
+- `-h`, `--help` - Show this message and exit.
+
+## Examples
+
+$ gto annotate nn --type model --path models/neural_network.h5
diff --git a/content/docs/gto/command-reference/assign.md b/content/docs/gto/command-reference/assign.md
new file mode 100644
index 00000000..ed19fab5
--- /dev/null
+++ b/content/docs/gto/command-reference/assign.md
@@ -0,0 +1,46 @@
+# assign
+
+Assign stage to specific artifact version
+
+## Synopsis
+
+```usage
+usage: gto assign [-r ] [--version ]
+ [--stage ] [-m ]
+ [--simple ] [--force] [--push] [--sr]
+ [-h]
+ name [ref]
+
+arguments:
+ name Artifact name
+ [ref] Git reference to use
+```
+
+## Description
+
+## Options
+
+- `-r `, `--repo ` - Local or remote repository [default: .]
+- `--version ` - If you provide REF, this will be used to name new version
+- `--stage ` - Stage to assign
+- `-m `, `--message ` - Message to annotate git tag with
+- `--simple ` - [auto, true, false] Use simple notation, e.g. rf#prod
+ instead of rf#prod-5 [default: auto]
+- `--force` - Create a git tag even if it already exists and is in effect
+- `--push` - Push created tag automatically (experimental)
+- `--sr`, `--skip-registration` - Don't register a version at specified commit
+- `-h`, `--help` - Show this message and exit.
+
+## Examples
+
+ Assign "nn" to "prod" at specific ref:
+ $ gto assign nn abcd123 --stage prod
+
+ Assign specific version:
+ $ gto assign nn --version v1.0.0 --stage prod
+
+ Assign at specific ref and name version explicitly:
+ $ gto assign nn abcd123 --version v1.0.0 --stage prod
+
+ Assign without increment:
+ $ gto assign nn HEAD --stage prod --simple
diff --git a/content/docs/gto/command-reference/check-ref.md b/content/docs/gto/command-reference/check-ref.md
new file mode 100644
index 00000000..30ed3437
--- /dev/null
+++ b/content/docs/gto/command-reference/check-ref.md
@@ -0,0 +1,32 @@
+# check-ref
+
+Find out the artifact version registered/assigned with ref
+
+## Synopsis
+
+```usage
+usage: gto check-ref [-r ] [--json] [--name] [--version]
+ [--event] [--stage] [-h]
+ ref
+
+arguments:
+ ref Git reference to analyze
+```
+
+## Description
+
+## Options
+
+- `-r `, `--repo ` - Local or remote repository [default: .]
+- `--json` - Print output in json format
+- `--name` - Show artifact name
+- `--version` - Output artifact version
+- `--event` - Show event
+- `--stage` - Show artifact stage
+- `-h`, `--help` - Show this message and exit.
+
+## Examples
+
+ $ gto check-ref rf@v1.0.0
+ $ gto check-ref rf#prod --name
+ $ gto check-ref rf#prod --version
diff --git a/content/docs/gto/command-reference/deprecate.md b/content/docs/gto/command-reference/deprecate.md
new file mode 100644
index 00000000..21e5fe4c
--- /dev/null
+++ b/content/docs/gto/command-reference/deprecate.md
@@ -0,0 +1,40 @@
+# deprecate
+
+Deprecate artifact, deregister a version, or unassign a stage
+
+## Synopsis
+
+```usage
+usage: gto deprecate [-r ] [-m ] [--simple ]
+ [--force] [-d] [--push] [-h]
+ name [version] [stage]
+
+arguments:
+ name Artifact name
+ [version] Artifact version
+ [stage] Stage to unassign
+```
+
+## Description
+
+## Options
+
+- `-r `, `--repo ` - Local or remote repository [default: .]
+- `-m `, `--message ` - Message to annotate git tag with
+- `--simple ` - [auto, true, false] Use simple notation, e.g. rf#prod
+ instead of rf#prod-5 [default: auto]
+- `--force` - Create a git tag even if it already exists and is in effect
+- `-d`, `--delete` - Delete the git tag(s) instead of creating the new one
+- `--push` - Push created tag automatically (experimental)
+- `-h`, `--help` - Show this message and exit.
+
+## Examples
+
+ Deprecate an artifact:
+ $ gto deprecate nn
+
+ Deprecate a version:
+ $ gto deprecate nn v0.0.1
+
+ Unassign a stage:
+ $ gto deprecate nn v0.0.1 prod
diff --git a/content/docs/gto/command-reference/describe.md b/content/docs/gto/command-reference/describe.md
new file mode 100644
index 00000000..76dbe177
--- /dev/null
+++ b/content/docs/gto/command-reference/describe.md
@@ -0,0 +1,30 @@
+# describe
+
+Display enrichments for an artifact
+
+## Synopsis
+
+```usage
+usage: gto describe [-r ] [--rev ] [--type] [--path]
+ [--description] [-h]
+ name
+
+arguments:
+ name Artifact name
+```
+
+## Description
+
+## Options
+
+- `-r `, `--repo ` - Local or remote repository [default: .]
+- `--rev ` - Repo revision to use
+- `--type` - Show type
+- `--path` - Show path
+- `--description` - Show description
+- `-h`, `--help` - Show this message and exit.
+
+## Examples
+
+ $ gto describe nn --rev HEAD
+ $ gto describe nn@v0.0.1
diff --git a/content/docs/gto/command-reference/history.md b/content/docs/gto/command-reference/history.md
new file mode 100644
index 00000000..2f2f47d1
--- /dev/null
+++ b/content/docs/gto/command-reference/history.md
@@ -0,0 +1,33 @@
+# history
+
+Show a journal of registry operations
+
+## Synopsis
+
+```usage
+usage: gto history [-r ] [-a] [-A] [--json] [--plain]
+ [--asc] [-h]
+ [name]
+
+arguments:
+ [name] Artifact name to show. If empty, show all.
+```
+
+## Description
+
+## Options
+
+- `-r `, `--repo ` - Local or remote repository [default: .]
+- `-a`, `--all-branches` - Read heads from all branches
+- `-A`, `--all-commits` - Read all commits
+- `--json` - Print output in json format
+- `--plain` - Print table in grep-able format
+- `--ascending`, `--asc` - Show new first
+- `-h`, `--help` - Show this message and exit.
+
+## Examples
+
+ $ gto history nn
+
+ Use --all-branches and --all-commits to read more than just HEAD:
+ $ gto history nn --all-commits
diff --git a/content/docs/gto/command-reference/register.md b/content/docs/gto/command-reference/register.md
new file mode 100644
index 00000000..c0b6b625
--- /dev/null
+++ b/content/docs/gto/command-reference/register.md
@@ -0,0 +1,47 @@
+# register
+
+Create an artifact version to signify an important, published or released
+iteration
+
+## Synopsis
+
+```usage
+usage: gto register [-r ] [--ver ] [-m ]
+ [--simple ] [--force] [--bump-major]
+ [--bump-minor] [--bump-patch] [--push] [-h]
+ name [ref]
+
+arguments:
+ name Artifact name
+ [ref] Git reference to use for registration [default: HEAD]
+```
+
+## Description
+
+## Options
+
+- `-r `, `--repo ` - Local or remote repository [default: .]
+- `--version `, `--ver ` - Version name in SemVer format
+- `-m `, `--message ` - Message to annotate git tag with
+- `--simple ` - [auto, true, false] Use simple notation, e.g. rf#prod
+ instead of rf#prod-5 [default: auto]
+- `--force` - Create a git tag even if it already exists and is in effect
+- `--bump-major` - Bump major version
+- `--bump-minor` - Bump minor version
+- `--bump-patch` - Bump patch version
+- `--push` - Push created tag automatically (experimental)
+- `-h`, `--help` - Show this message and exit.
+
+## Examples
+
+ Register new version at HEAD:
+ $ gto register nn
+
+ Register new version at a specific ref:
+ $ gto register nn abc1234
+
+ Assign version name explicitly:
+ $ gto register nn --version v1.0.0
+
+ Choose a part to bump version by:
+ $ gto register nn --bump-minor
diff --git a/content/docs/gto/command-reference/remove.md b/content/docs/gto/command-reference/remove.md
new file mode 100644
index 00000000..51486810
--- /dev/null
+++ b/content/docs/gto/command-reference/remove.md
@@ -0,0 +1,27 @@
+# remove
+
+Remove the enrichment for given artifact
+
+## Synopsis
+
+```usage
+usage: gto remove [-r ] [--commit] [--push] [-h]
+ name
+
+arguments:
+ name Artifact name
+```
+
+## Description
+
+## Options
+
+- `-r `, `--repo ` - Local or remote repository [default: .]
+- `--commit` - Automatically commit changes due to this command (experimental)
+- `--push` - Push created commit automatically (experimental) - will set
+ commit=True
+- `-h`, `--help` - Show this message and exit.
+
+## Examples
+
+ $ gto remove nn
diff --git a/content/docs/gto/command-reference/show.md b/content/docs/gto/command-reference/show.md
new file mode 100644
index 00000000..b63d9cc2
--- /dev/null
+++ b/content/docs/gto/command-reference/show.md
@@ -0,0 +1,50 @@
+# show
+
+Show the registry state, highest version, or what's assigned in stage
+
+## Synopsis
+
+```usage
+usage: gto show [-r ] [-a] [-A] [--json] [--plain]
+ [--name] [--version] [--stage] [--ref]
+ [--ro] [--av ] [--vs ]
+ [--sort ] [-h]
+ [name]
+
+arguments:
+ [name] Artifact name to show. If empty, show registry
+```
+
+## Description
+
+## Options
+
+- `-r `, `--repo ` - Local or remote repository [default: .]
+- `-a`, `--all-branches` - Read heads from all branches
+- `-A`, `--all-commits` - Read all commits
+- `--json` - Print output in json format
+- `--plain` - Print table in grep-able format
+- `--name` - Show artifact name
+- `--version` - Output artifact version
+- `--stage` - Show artifact stage
+- `--ref` - Show ref
+- `--ro`, `--registered-only` - Show only registered versions
+- `--av `, `--assignments-per-version ` - Show N last stages
+ for each version. -1 for all [default: -1]
+- `--vs `, `--versions-per-stage ` - Show N last versions for
+ each stage. -1 for all. Applied after 'assignments-per-version' [default: 1]
+- `--sort ` - Order assignments by timestamp or semver [default:
+ timestamp]
+- `-h`, `--help` - Show this message and exit.
+
+## Examples
+
+ Show the registry:
+ $ gto show
+
+ Show versions of specific artifact in registry:
+ $ gto show nn
+
+ Show greatest version or what's in stage:
+ $ gto show nn@greatest
+ $ gto show nn#prod
diff --git a/content/docs/gto/command-reference/stages.md b/content/docs/gto/command-reference/stages.md
new file mode 100644
index 00000000..e9318b3b
--- /dev/null
+++ b/content/docs/gto/command-reference/stages.md
@@ -0,0 +1,25 @@
+# stages
+
+Print list of stages used in the registry
+
+## Synopsis
+
+```usage
+usage: gto stages [-r ] [--allowed] [--used] [--json]
+ [-h]
+```
+
+## Description
+
+## Options
+
+- `-r `, `--repo ` - Local or remote repository [default: .]
+- `--allowed` - Show allowed stages from config
+- `--used` - Show stages that were ever used (from all git tags)
+- `--json` - Print output in json format
+- `-h`, `--help` - Show this message and exit.
+
+## Examples
+
+ $ gto stages
+ $ gto stages --allowed
diff --git a/content/docs/gto/get-started.md b/content/docs/gto/get-started.md
index 6fadd09a..16964f9e 100644
--- a/content/docs/gto/get-started.md
+++ b/content/docs/gto/get-started.md
@@ -86,16 +86,13 @@ The `gto show` output confirms our expectation.
The power of using Git tags to register versions and assign stages is simple: we
can act upon them in well-known way - in CI/CD.
-
- Non-exhaustive list of scenarios GTO can help with
+
+ Using GTO to build a Model Registry
diff --git a/content/docs/gto/user-guide/mlem.md b/content/docs/gto/mlem.md
similarity index 62%
rename from content/docs/gto/user-guide/mlem.md
rename to content/docs/gto/mlem.md
index b1376957..be396e3a 100644
--- a/content/docs/gto/user-guide/mlem.md
+++ b/content/docs/gto/mlem.md
@@ -1,3 +1,5 @@
# Deploy models with MLEM
...tell about MLEM and show `example-gto` repo `mlem` branch...
+
+...(I'll remove this page before merging this PR)...
diff --git a/content/docs/gto/user-guide.md b/content/docs/gto/user-guide.md
new file mode 100644
index 00000000..c5a58f5b
--- /dev/null
+++ b/content/docs/gto/user-guide.md
@@ -0,0 +1,109 @@
+# User Guide
+
+GTO helps you build a Artifact Registry out of your Git repository. It does so
+by creating Git tags of special format and managing `artifacts.yaml` metafile.
+Since committing large files to Git is not a good practice, you should consider
+not committing your artifacts to Git. Instead, use [DVC](/doc/gto/dvc), Git-lfs,
+or any method commit pointers to the binary files instead.
+
+## Annotations in artifacts.yaml
+
+Using Git tag to register versions and assign stages is handy, but the Git tag
+itself doesn't contain path to the artifact, type of it (it could be `model` or
+`dataset`), or any other information you may find useful. For simple projects
+(e.g. single artifact) we can assume the details in a downstream system. But for
+more advanced cases, we should codify them in the registry itself.
+
+To keep this metainformation, GTO uses `artifacts.yaml` file. Commands like
+`gto annotate` and `gto remove` are used to modify it, while `gto describe`
+helps get them when they're needed.
+
+If you would like to see an example of `artifacts.yaml`, check out the
+[example-gto](https://github.com/iterative/example-gto/blob/main/artifacts.yaml)
+repo.
+
+## Acting downstream
+
+Once Git tag was pushed, you can start acting downstream. One of the popular
+options here is to use CI/CD, setting it to be triggered by pushing a Git tag.
+For general details, check out something like
+[GitHub Actions](https://github.com/features/actions),
+[GitLab CI/CD](https://docs.gitlab.com/ee/ci/) or
+[Circle CI](https://circleci.com).
+
+The other option is to
+[configure webhooks](https://docs.github.com/en/rest/webhooks) that will send
+HTTP requests to your server upon pushing Git tags to the remote.
+
+Finally, you can configure your server to query your Git provider via something
+like REST API to check if changes happened. As an example, check out
+[Github REST API](https://docs.github.com/en/rest).
+
+### Getting started with CI/CD
+
+To act upon registrations and assignments (Git tags), you can create a simple CI
+workflow. To see an example, check out
+[the workflow in `example-gto` repo](https://github.com/iterative/example-gto/blob/main/.github/workflows/gto-act-on-tags.yml).
+The workflow uses [the GTO GH Action](https://github.com/iterative/gto-action)
+that fetches all Git tags (to correctly interpret the Registry), finds out the
+`version` of the artifact that was registered, the `stage` that was assigned,
+and annotations details such as `path`, `type`, `description`, etc, so you could
+use them in the next steps of the CI.
+
+### Helpful commands
+
+If you would like to set up CI/CD, but don't want to use GTO GH Action, check
+out `gto show`, `gto check-ref` and `gto describe` commands.
+
+## Configuring types and stages
+
+To configure GTO, use file `.gto` in the root of your repo:
+
+```yaml
+# .gto config file
+types: [model, dataset] # list of allowed Types
+stages: [dev, stage, prod] # list of allowed Stages
+```
+
+When allowed Stages or Types are specified, GTO will check commands you run and
+error out if you provided a value that doesn't exist in the config. Note, that
+GTO applies the config from the workspace, so if want to apply the config from
+`main` branch, you need to check it out first with `git checkout main`.
+
+Alternatively, you can use environment variables (note the `GTO_` prefix)
+
+```cli
+$ GTO_EMOJIS=false gto show
+```
+
+## Git tags format
+
+
+
+You can work with GTO without knowing the used Git tag names conventions, since
+commands like `gto register` and `gto assign` abstracts that away from you.
+
+
+
+All events have the standard formats of Git tags:
+
+- `{artifact_name}@{version_number}#{e}` for version registration.
+- `{artifact_name}@{version_number}!#{e}` for version deregistration.
+- `{artifact_name}#{stage}#{e}` for stage assignment.
+- `{artifact_name}#{stage}!#{e}` for stage unassignment.
+- `{artifact_name}@deprecated#{e}` for artifact deprecation.
+
+All of them share two parts:
+
+1. `{artifact_name}` prefix part.
+2. `#{e}` counter at the end that can be omitted (in "simple" Git tag format).
+
+Generally, `#{e}` counter is used, because Git doesn't allow to create two Git
+tags with the same name. If you want to have two Git tags that assign `dev`
+stage to `model` artifact without the counter (`model#dev`), that will require
+deleting the old Git tag first. Consequently, that doesn't allow you to preserve
+history of events that happened.
+
+By default, `#{e}` sometimes is omitted, sometimes not. We are setting defaults
+to omit using `#{e}` when it's rarely necessary, e.g. for version registrations
+and artifact deprecations.
diff --git a/content/docs/gto/user-guide/artifacts.md b/content/docs/gto/user-guide/artifacts.md
deleted file mode 100644
index b3e63d00..00000000
--- a/content/docs/gto/user-guide/artifacts.md
+++ /dev/null
@@ -1,15 +0,0 @@
-# Annotations in artifacts.yaml
-
-Using Git tag to register versions and assign stages is handy, but the Git tag
-itself doesn't contain path to the artifact, type of it (it could be `model` or
-`dataset`), or any other information you may find useful. For simple projects
-(e.g. single artifact) we can assume the details in a downstream system. But for
-more advanced cases, we should codify them in the registry itself.
-
-To keep this metainformation, GTO uses `artifacts.yaml` file. Commands like
-`gto annotate` and `gto remove` are used to modify it, while `gto describe`
-helps get them when they're needed.
-
-If you would like to see an example of `artifacts.yaml`, check out the
-[example-gto](https://github.com/iterative/example-gto/blob/main/artifacts.yaml)
-repo.
diff --git a/content/docs/gto/user-guide/config.md b/content/docs/gto/user-guide/config.md
deleted file mode 100644
index 8a44acb4..00000000
--- a/content/docs/gto/user-guide/config.md
+++ /dev/null
@@ -1,20 +0,0 @@
-# Configuring types and stages
-
-To configure GTO, use file `.gto` in the root of your repo:
-
-```yaml
-# .gto config file
-types: [model, dataset] # list of allowed Types
-stages: [dev, stage, prod] # list of allowed Stages
-```
-
-When allowed Stages or Types are specified, GTO will check commands you run and
-error out if you provided a value that doesn't exist in the config. Note, that
-GTO applies the config from the workspace, so if want to apply the config from
-`main` branch, you need to check it out first with `git checkout main`.
-
-Alternatively, you can use environment variables (note the `GTO_` prefix)
-
-```cli
-$ GTO_EMOJIS=false gto show
-```
diff --git a/content/docs/gto/user-guide/downstream.md b/content/docs/gto/user-guide/downstream.md
deleted file mode 100644
index 596c819a..00000000
--- a/content/docs/gto/user-guide/downstream.md
+++ /dev/null
@@ -1,32 +0,0 @@
-# Acting downstream
-
-Once Git tag was pushed, you can start acting downstream. One of the popular
-options here is to use CI/CD, setting it to be triggered by pushing a Git tag.
-For general details, check out something like
-[GitHub Actions](https://github.com/features/actions),
-[GitLab CI/CD](https://docs.gitlab.com/ee/ci/) or
-[Circle CI](https://circleci.com).
-
-The other option is to
-[configure webhooks](https://docs.github.com/en/rest/webhooks) that will send
-HTTP requests to your server upon pushing Git tags to the remote.
-
-Finally, you can configure your server to query your Git provider via something
-like REST API to check if changes happened. As an example, check out
-[Github REST API](https://docs.github.com/en/rest).
-
-## Getting started with CI/CD
-
-To act upon registrations and assignments (Git tags), you can create a simple CI
-workflow. To see an example, check out
-[the workflow in `example-gto` repo](https://github.com/iterative/example-gto/blob/main/.github/workflows/gto-act-on-tags.yml).
-The workflow uses [the GTO GH Action](https://github.com/iterative/gto-action)
-that fetches all Git tags (to correctly interpret the Registry), finds out the
-`version` of the artifact that was registered, the `stage` that was assigned,
-and annotations details such as `path`, `type`, `description`, etc, so you could
-use them in the next steps of the CI.
-
-## Helpful commands
-
-If you would like to set up CI/CD, but don't want to use GTO GH Action, check
-out `gto show`, `gto check-ref` and `gto describe` commands.
diff --git a/content/docs/gto/user-guide/git-tags.md b/content/docs/gto/user-guide/git-tags.md
deleted file mode 100644
index 44b4214b..00000000
--- a/content/docs/gto/user-guide/git-tags.md
+++ /dev/null
@@ -1,31 +0,0 @@
-# Git tags format
-
-
-
-You can work with GTO without knowing the used Git tag names conventions, since
-commands like `gto register` and `gto assign` abstracts that away from you.
-
-
-
-All events have the standard formats of Git tags:
-
-- `{artifact_name}@{version_number}#{e}` for version registration.
-- `{artifact_name}@{version_number}!#{e}` for version deregistration.
-- `{artifact_name}#{stage}#{e}` for stage assignment.
-- `{artifact_name}#{stage}!#{e}` for stage unassignment.
-- `{artifact_name}@deprecated#{e}` for artifact deprecation.
-
-All of them share two parts:
-
-1. `{artifact_name}` prefix part.
-2. `#{e}` counter at the end that can be omitted (in "simple" Git tag format).
-
-Generally, `#{e}` counter is used, because Git doesn't allow to create two Git
-tags with the same name. If you want to have two Git tags that assign `dev`
-stage to `model` artifact without the counter (`model#dev`), that will require
-deleting the old Git tag first. Consequently, that doesn't allow you to preserve
-history of events that happened.
-
-By default, `#{e}` sometimes is omitted, sometimes not. We are setting defaults
-to omit using `#{e}` when it's rarely necessary, e.g. for version registrations
-and artifact deprecations.
diff --git a/content/docs/gto/user-guide/index.md b/content/docs/gto/user-guide/index.md
deleted file mode 100644
index 81f02544..00000000
--- a/content/docs/gto/user-guide/index.md
+++ /dev/null
@@ -1,17 +0,0 @@
-# User Guide
-
-GTO helps you build a Artifact Registry out of your Git repository. It does so
-by creating Git tags of [special format](/doc/gto/user-guide) and managing
-[`artifacts.yaml` metafile](/doc/gto/user-guide). Since committing large files
-to Git is not a good practice, you shouldn't commit binary files to Git.
-Instead, use
-[DVC, Git-lfs, or any method commit pointers to the binary files instead](/doc/gto/user-guide/storages).
-
-Our guides describe the major concepts in GTO and how it works comprehensively,
-explaining when and how to use what, as well as inter-relationship between them.
-
-The topics here range from more foundational (impacting many parts of GTO) to
-more specific and advanced things you can do.
-
-Please choose from the navigation sidebar to the left, or click the `Next`
-button below ↘
diff --git a/content/docs/sidebar.json b/content/docs/sidebar.json
index ffdef7b5..c93a38ed 100644
--- a/content/docs/sidebar.json
+++ b/content/docs/sidebar.json
@@ -636,54 +636,20 @@
"label": "Get Started",
"source": "get-started.md"
},
- {
- "slug": "use-cases",
- "label": "Use Cases",
- "source": "use-cases/index.md",
- "children": [
- {
- "slug": "model-registry",
- "source": "model-registry.md",
- "label": "Model Registry"
- }
- ]
- },
{
"slug": "user-guide",
"label": "User Guide",
- "source": "user-guide/index.md",
- "children": [
- {
- "slug": "git-tags",
- "source": "git-tags.md",
- "label": "Git tags format"
- },
- {
- "slug": "artifacts",
- "source": "artifacts.md",
- "label": "Annotations in artifacts.yaml"
- },
- {
- "slug": "downstream",
- "source": "downstream.md",
- "label": "Acting downstream. CI/CD"
- },
- {
- "slug": "config",
- "source": "config.md",
- "label": "Configuring types and stages"
- },
- {
- "slug": "storages",
- "source": "storages.md",
- "label": "Storing large artifacts"
- },
- {
- "slug": "mlem",
- "source": "mlem.md",
- "label": "Deploying models with MLEM"
- }
- ]
+ "source": "user-guide.md"
+ },
+ {
+ "slug": "dvc",
+ "source": "dvc.md",
+ "label": "GTO with DVC"
+ },
+ {
+ "slug": "mlem",
+ "source": "mlem.md",
+ "label": "GTO with MLEM"
},
{
"slug": "command-reference",
From 21930be0562da6eb3069b9f3ecb1c528690293e4 Mon Sep 17 00:00:00 2001
From: Alexander Guschin <1aguschin@gmail.com>
Date: Thu, 17 Nov 2022 19:17:04 +0600
Subject: [PATCH 25/38] shorten command-reference/index
---
content/docs/gto/command-reference/index.md | 30 +++++----------------
1 file changed, 7 insertions(+), 23 deletions(-)
diff --git a/content/docs/gto/command-reference/index.md b/content/docs/gto/command-reference/index.md
index 9f5d9262..a0ae400c 100644
--- a/content/docs/gto/command-reference/index.md
+++ b/content/docs/gto/command-reference/index.md
@@ -1,27 +1,11 @@
# GTO Command Reference
-GTO is a command line tool (`gto` binary). For a list of all commands, type
-`gto -h`.
+GTO is a command line tool (see [installation](/doc/gto/install)). For a list of
+all commands, type `gto -h`.
-## Read the registry
+## Typical GTO workflow
-- Show the registry state, highest version, or what's assigned in stage with
- `gto show`
-- Find out the artifact version registered/assigned with ref with
- `gto check-ref`
-- Show a journal of registry operations with `gto history`
-- Print list of stages used in the registry with `gto stages`
-
-## Modify artifacts
-
-- Assign stage to specific artifact version with `gto assign`
-- Deprecate artifact, deregister a version, or unassign a stage with
- `gto deprecate`
-- Create an artifact version to signify an important, published or released
- iteration with `gto register`
-
-## Manage artifact enrichments
-
-- Update artifact metadata annotations with `gto annotate`
-- Display enrichments for an artifact with `gto describe`
-- Remove the enrichment for given artifact with `gto remove`
+- Register artifact versions with `gto register`
+- Assign stages to them with `gto assign`
+- Read the registry with `gto show` and `gto history`
+- Edit artifact metadata with `gto annotate` and read it with `gto describe`
From 0b25aecc1faee21b55659564a23b1d8c1ebc5c0a Mon Sep 17 00:00:00 2001
From: Alexander Guschin <1aguschin@gmail.com>
Date: Thu, 17 Nov 2022 19:24:10 +0600
Subject: [PATCH 26/38] hide next steps in gs
---
content/docs/gto/get-started.md | 12 ++++--------
1 file changed, 4 insertions(+), 8 deletions(-)
diff --git a/content/docs/gto/get-started.md b/content/docs/gto/get-started.md
index 76fb2f3a..a0638c09 100644
--- a/content/docs/gto/get-started.md
+++ b/content/docs/gto/get-started.md
@@ -143,11 +143,7 @@ If you want to see more CI examples, check out
Thanks for completing this Get Started!
-- If you want how to specify artifact's metainformation like `path`, `type` and
- `description`, check out [User Guide](/doc/gto/user-guide/artifacts).
-- If you want to learn about using DVC to keep your artifact binaries in remote
- storages, check out [DVC docs](https://dvc.org/doc).
-- If you want to learn more about Studio, check out
- [Studio docs](https://dvc.org/doc/studio).
-- If you want to learn about using MLEM to deploying your model upon GTO stage
- assignments, check out [MLEM docs](/doc/).
+
From 68f2b98c40c8907d02d201bb62ed45bc537759ef Mon Sep 17 00:00:00 2001
From: Alexander Guschin <1aguschin@gmail.com>
Date: Thu, 17 Nov 2022 19:43:46 +0600
Subject: [PATCH 27/38] reveal why-gto section
---
content/docs/gto/index.md | 4 +-
content/docs/gto/use-cases/model-registry.md | 62 -------------------
.../gto/{use-cases/index.md => why-gto.md} | 10 ++-
content/docs/sidebar.json | 5 ++
4 files changed, 14 insertions(+), 67 deletions(-)
delete mode 100644 content/docs/gto/use-cases/model-registry.md
rename content/docs/gto/{use-cases/index.md => why-gto.md} (96%)
diff --git a/content/docs/gto/index.md b/content/docs/gto/index.md
index 49878390..f68313e1 100644
--- a/content/docs/gto/index.md
+++ b/content/docs/gto/index.md
@@ -1,8 +1,8 @@
# GTO Documentation
**GTO** is a tool for creating an Artifact Registry in your Git repository. One
-of the special cases we would like to highlight is creating a **Machine Learning
-Model Registry**.
+of the special cases we would like to highlight is creating a
+[Machine Learning Model Registry](/doc/use-cases/model-registry).
Such a registry serves as a centralized place to store and operationalize your
artifacts along with their metadata; manage model life-cycle, versions &
diff --git a/content/docs/gto/use-cases/model-registry.md b/content/docs/gto/use-cases/model-registry.md
deleted file mode 100644
index b0f8225d..00000000
--- a/content/docs/gto/use-cases/model-registry.md
+++ /dev/null
@@ -1,62 +0,0 @@
-# Machine Learning Model Registry
-
-A **model registry** is a tool to catalog ML models and their versions. Models
-from your data science projects can be discovered, tested, shared, deployed, and
-audited from there. [DVC](https://github.com/iterative/dvc), GTO, and [MLEM]
-enable these capabilities on top of Git, so you can stick to en existing
-software engineering stack. No more divide between ML engineering and
-operations!
-
-[mlem]: /doc
-
-ML model registries give your team key capabilities:
-
-- Collect and organize model [versions] from different sources effectively,
- preserving their data provenance and lineage information.
-- Share metadata including [metrics and plots][mp] to help use and evaluate
- models.
-- A standard interface to access all your ML artifacts, from early-stage
- [experiments] to production-ready models.
-- Deploy specific models on different environments (dev, shadow, prod, etc.)
- without touching the applications that consume them.
-- For security, control who can manage models, and audit their usage trails.
-
-[versions]: https://dvc.org/doc/use-cases/versioning-data-and-model-files
-[mp]: https://dvc.org/doc/start/metrics-parameters-plots
-[experiments]: https://dvc.org/doc/user-guide/experiment-management
-
-Many of these benefits are built into DVC: Your [modeling process] and
-[performance data][mp] become **codified** in Git-based DVC
-repositories, making it possible to reproduce and manage models with
-standard Git workflows (along with code). Large model files are stored
-separately and efficiently, and can be pushed to [remote storage] -- a scalable
-access point for [sharing].
-
-
-
-See also [Data Registry](https://dvc.org/doc/use-cases/data-registry).
-
-
-
-To make a Git-native registry (on top of DVC or not), one option is to use GTO
-(Git Tag Ops). It tags ML model releases and promotions, and links them to
-artifacts in the repo using versioned annotations. This creates abstractions for
-your models, which lets you **manage their lifecycle** freely and directly from
-Git.
-
-And to **productionize** the models, you can save and build them with the [MLEM]
-Python API or CLI, which automagically captures all the context needed to
-distribute them. It can store model files on the cloud (by itself or with DVC),
-list and transfer them within locations, wrap them as a local REST server, or
-even containerize and deploy them to cloud providers!
-
-This ecosystem of tools from [Iterative](https://iterative.ai/) brings your ML
-process into [GitOps]. This means you can manage and deliver ML models with
-software engineering methods such as continuous integration (CI/CD), which can
-sync with the state of the artifacts in your registry.
-
-[modeling process]: https://dvc.org/doc/start/data-pipelines
-[remote storage]: https://dvc.org/doc/command-reference/remote
-[sharing]: https://dvc.org/doc/start/data-and-model-access
-[via cml]: https://cml.dev/doc/cml-with-dvc
-[gitops]: https://www.gitops.tech/
diff --git a/content/docs/gto/use-cases/index.md b/content/docs/gto/why-gto.md
similarity index 96%
rename from content/docs/gto/use-cases/index.md
rename to content/docs/gto/why-gto.md
index f89a3ae3..39e3e9c2 100644
--- a/content/docs/gto/use-cases/index.md
+++ b/content/docs/gto/why-gto.md
@@ -1,8 +1,12 @@
-# Use Cases
+# Why GTO?
**GTO** is a tool for creating an Artifact Registry in your Git repository. One
of the special cases we would like to highlight is creating a
-[Machine Learning Model Registry](/doc/gto/use-cases/model-registry).
+[Machine Learning Model Registry](/doc/use-cases/model-registry).
+
+
Date: Thu, 17 Nov 2022 10:47:07 -0500
Subject: [PATCH 28/38] Update theme to 0.1.24 for args linker fix
---
package.json | 2 +-
yarn.lock | 8 ++++----
2 files changed, 5 insertions(+), 5 deletions(-)
diff --git a/package.json b/package.json
index 2a4cc042..912f1c99 100644
--- a/package.json
+++ b/package.json
@@ -21,7 +21,7 @@
"prepare": "husky install"
},
"dependencies": {
- "@dvcorg/gatsby-theme-iterative": "0.1.23",
+ "@dvcorg/gatsby-theme-iterative": "0.1.24",
"@dvcorg/websites-server": "^0.0.10",
"@octokit/graphql": "5.0.1",
"@sentry/gatsby": "^7.13.0",
diff --git a/yarn.lock b/yarn.lock
index 52201adb..70f34107 100644
--- a/yarn.lock
+++ b/yarn.lock
@@ -1084,10 +1084,10 @@
resolved "https://registry.yarnpkg.com/@csstools/convert-colors/-/convert-colors-1.4.0.tgz#ad495dc41b12e75d588c6db8b9834f08fa131eb7"
integrity sha512-5a6wqoJV/xEdbRNKVo6I4hO3VjyDq//8q2f9I6PBAvMesJHFauXDorcNCsr9RzvsZnaWi5NYCcfyqP1QeFHFbw==
-"@dvcorg/gatsby-theme-iterative@0.1.23":
- version "0.1.23"
- resolved "https://registry.yarnpkg.com/@dvcorg/gatsby-theme-iterative/-/gatsby-theme-iterative-0.1.23.tgz#8e13533008999fa8398a69525f920557a00bce45"
- integrity sha512-LrEQvAHL0fPRsHw7Ymi4DNxw7H6CHJnIPQYbGbj1h8PI+IvBoTrnhpLoGtOPJSna/EUILrErWP+i1FSFcP1oAQ==
+"@dvcorg/gatsby-theme-iterative@0.1.24":
+ version "0.1.24"
+ resolved "https://registry.yarnpkg.com/@dvcorg/gatsby-theme-iterative/-/gatsby-theme-iterative-0.1.24.tgz#156264e5cb72e21c5dff1926306ff0121683c79e"
+ integrity sha512-NFmiq+tJ7UkrkGY+36KC9XkIqtaJFxiz19RN9Zq14pHA8ZKjSxBG+qqFyR+kxo7KcF9CIJivoSHNQ10YZzhiOg==
dependencies:
"@reach/portal" "^0.17.0"
"@reach/router" "^1.3.4"
From f374516eea7938d1b5adccce5c3cabe5cd3c4147 Mon Sep 17 00:00:00 2001
From: Alexander Guschin <1aguschin@gmail.com>
Date: Mon, 21 Nov 2022 19:37:38 +0600
Subject: [PATCH 29/38] fix links
---
content/docs/gto/command-reference/assign.md | 2 +-
content/docs/gto/command-reference/deprecate.md | 2 +-
content/docs/gto/command-reference/register.md | 2 +-
content/docs/gto/command-reference/stages.md | 2 +-
content/docs/gto/get-started.md | 2 +-
content/docs/gto/user-guide.md | 2 +-
6 files changed, 6 insertions(+), 6 deletions(-)
diff --git a/content/docs/gto/command-reference/assign.md b/content/docs/gto/command-reference/assign.md
index 868151a4..784ff50b 100644
--- a/content/docs/gto/command-reference/assign.md
+++ b/content/docs/gto/command-reference/assign.md
@@ -29,7 +29,7 @@ Created git tag 'awesome-model#prod#1' that assigns stage to 'v0.0.1'
```
GTO creates a special Git tag in
-[the standard format](/doc/gto/user-guide/git-tags).
+[the standard format](/doc/gto/user-guide#git-tags-format).
## Options
diff --git a/content/docs/gto/command-reference/deprecate.md b/content/docs/gto/command-reference/deprecate.md
index 9bb32bd6..4729efe3 100644
--- a/content/docs/gto/command-reference/deprecate.md
+++ b/content/docs/gto/command-reference/deprecate.md
@@ -46,7 +46,7 @@ Created git tag 'awesome-model#prod!#2' that unassigns a stage from 'v0.0.1'
### Unassigning a stage: some details and options
GTO creates a special Git tag in
-[the standard format](/doc/gto/user-guide/git-tags).
+[the standard format](/doc/gto/user-guide#git-tags-format).
Note, that later you can create this stage again, if you need to, by calling
`$ gto assign` again.
diff --git a/content/docs/gto/command-reference/register.md b/content/docs/gto/command-reference/register.md
index 4a6c574f..2911fffa 100644
--- a/content/docs/gto/command-reference/register.md
+++ b/content/docs/gto/command-reference/register.md
@@ -28,7 +28,7 @@ Created git tag 'awesome-model@v0.0.1' that registers a version
```
GTO creates a special Git tag for the artifact version, in
-[the standard format](/doc/gto/user-guide/git-tags).
+[the standard format](/doc/gto/user-guide#git-tags-format).
The version is now associated to the current Git commit (`HEAD`). You can use
another Git commit if you provide it's hexsha as an additional argument, like
diff --git a/content/docs/gto/command-reference/stages.md b/content/docs/gto/command-reference/stages.md
index 72997030..0950f0a2 100644
--- a/content/docs/gto/command-reference/stages.md
+++ b/content/docs/gto/command-reference/stages.md
@@ -12,7 +12,7 @@ usage: gto stages [-r ] [--allowed] [--used] [--json]
## Description
This command is used to get a list of stages used in the registry - whether
-directly from [the config file](/doc/gto/user-guide/config), or from all GTO Git
+directly from [the config file](/doc/gto/user-guide#configuring-gto), or from all GTO Git
tags that exist.
## Options
diff --git a/content/docs/gto/get-started.md b/content/docs/gto/get-started.md
index a0638c09..89bb517c 100644
--- a/content/docs/gto/get-started.md
+++ b/content/docs/gto/get-started.md
@@ -89,7 +89,7 @@ GTO [uses Git tags] to register artifact versions and assign them to stages.
This means we can act upon these operations in any Git-based system such as many
CI/CD platforms.
-[uses git tags]: /doc/gto/user-guide/git-tags
+[uses git tags]: /doc/gto/user-guide#git-tags-format
Date: Tue, 22 Nov 2022 17:02:10 -0600
Subject: [PATCH 36/38] Update content/docs/gto/get-started.md
---
content/docs/gto/get-started.md | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/content/docs/gto/get-started.md b/content/docs/gto/get-started.md
index 8e5580c9..0451e087 100644
--- a/content/docs/gto/get-started.md
+++ b/content/docs/gto/get-started.md
@@ -6,8 +6,8 @@ description:
# Get Started
GTO helps you build an Artifact Registry on top of a Git repository. You can
-register relevant versions of your files or directories from any source (e.g. ML
-model releases) and assign them to different deployment environments (testing,
+register relevant versions of your files or directories (e.g. ML model releases)
+from any source (other repos) and assign them to different deployment stages (testing,
shadow, production, etc.). Git-native mechanisms are used, so you can automate
the delivery of your software project with CI/CD, and adopt a GitOps approach in
general.
From 8f9b6a3b3169e75b0538e55a8aa10b5c4854c5fe Mon Sep 17 00:00:00 2001
From: Alexander Guschin <1aguschin@gmail.com>
Date: Wed, 23 Nov 2022 13:10:00 +0600
Subject: [PATCH 37/38] Apply suggestions from code review
Co-authored-by: Jorge Orpinel
---
.../docs/gto/command-reference/annotate.md | 2 +-
content/docs/gto/command-reference/index.md | 4 +-
content/docs/gto/get-started.md | 127 +++++++++---------
content/docs/gto/index.md | 12 +-
content/docs/gto/install.md | 12 +-
content/docs/gto/user-guide.md | 23 ++--
6 files changed, 90 insertions(+), 90 deletions(-)
diff --git a/content/docs/gto/command-reference/annotate.md b/content/docs/gto/command-reference/annotate.md
index 6d75989e..df43d5bd 100644
--- a/content/docs/gto/command-reference/annotate.md
+++ b/content/docs/gto/command-reference/annotate.md
@@ -1,6 +1,6 @@
# annotate
-Create or update metainformation about an artifact.
+Create or update metadata about an artifact.
## Synopsis
diff --git a/content/docs/gto/command-reference/index.md b/content/docs/gto/command-reference/index.md
index a0ae400c..030aec4e 100644
--- a/content/docs/gto/command-reference/index.md
+++ b/content/docs/gto/command-reference/index.md
@@ -1,7 +1,7 @@
# GTO Command Reference
-GTO is a command line tool (see [installation](/doc/gto/install)). For a list of
-all commands, type `gto -h`.
+GTO is a command line tool. It works on top of Git. For a list of all commands,
+type `gto -h`.
## Typical GTO workflow
diff --git a/content/docs/gto/get-started.md b/content/docs/gto/get-started.md
index 0451e087..9ece6530 100644
--- a/content/docs/gto/get-started.md
+++ b/content/docs/gto/get-started.md
@@ -5,22 +5,18 @@ description:
# Get Started
-GTO helps you build an Artifact Registry on top of a Git repository. You can
-register relevant versions of your files or directories (e.g. ML model releases)
-from any source (other repos) and assign them to different deployment stages (testing,
-shadow, production, etc.). Git-native mechanisms are used, so you can automate
-the delivery of your software project with CI/CD, and adopt a GitOps approach in
-general.
+GTO helps you manage machine learning artifact versions in a Git repository, and
+their deployment stages (testing, shadow, production, etc.).
-Assuming GTO is already [installed](/doc/gto/install) in your active Python
-environment, let's clone an example [ML Model Registry] and review it with
-`git show`:
+Assuming GTO is already [installed](/doc/gto/install) in your Python environment,
+let's clone an [example model registry] and review it's current state with `gto show`:
-[ml model registry]: /doc/use-cases/model-registry
+[example model registry]: https://github.com/iterative/example-gto
```cli
$ git clone https://github.com/iterative/example-gto
$ cd example-gto
+
$ gto show
╒══════════╤══════════╤════════╤═════════╤════════════╕
│ name │ latest │ #dev │ #prod │ #staging │
@@ -38,9 +34,9 @@ are shown.
## Registering a new version
-Registering a version is usually done to mark significant changes to the
-artifact. Let's assume this is the case for the `cv-class` model in `HEAD` and
-we would like to create a new version of it:
+`gto register` lets you mark significant artifact versions (e.g. an ML model
+release). Let's register a new version of `cv-class` and check the registry
+status again:
```cli
$ gto register cv-class
@@ -48,7 +44,6 @@ Created git tag 'cv-class@v0.1.14' that registers version
To push the changes upstream, run:
git push origin cv-class@v0.1.14
-# this version will be shown in `gto show` now:
$ gto show
╒══════════╤══════════╤════════╤═════════╤════════════╕
│ name │ latest │ #dev │ #prod │ #staging │
@@ -59,10 +54,13 @@ $ gto show
╘══════════╧══════════╧════════╧═════════╧════════════╛
```
+This creates a Git tag attached to the latest Git commit (`HEAD`) which bumps the
+artifact's version automatically (in this case from `v0.1.13` to `v0.1.14`).
+
## Assigning stages
-Let's assume the version we just registered looks very promising, and we want to
-promote it to `dev` to test it:
+The version we just registered looks very promising. You can promote it with
+`gto assign`, for example to the `dev` stage (for testing):
```cli
$ gto assign cv-class --stage dev
@@ -70,7 +68,6 @@ Created git tag 'cv-class#dev#1' that assigns stage to version 'v0.1.14'
To push the changes upstream, run:
git push origin cv-class#dev#1
-# stage assignment can be seen with `gto show`:
$ gto show
╒══════════╤══════════╤═════════╤═════════╤════════════╕
│ name │ latest │ #dev │ #prod │ #staging │
@@ -81,78 +78,81 @@ $ gto show
╘══════════╧══════════╧═════════╧═════════╧════════════╛
```
-We can push to Git repository to start a CI job!
+This also creates a Git tag, which associates the latest version of `cv-class`
+(`v0.1.14`) to `dev`.
-## Act on registrations and assignments in CI/CD
+## Act in CI/CD upon registrations and assignments
-GTO [uses Git tags] to register artifact versions and assign them to stages.
-This means we can act upon these operations in any Git-based system such as many
-CI/CD platforms.
+You may have noticed that `gto` reminds you how to `git push` the [tags] created
+during registrations and promotions. The benefit of these Git-native mechanism is
+that you can act upon GTO operations in any Git-based system downstream, for
+example automating model deployments with CI/CD.
-[uses git tags]: /doc/gto/user-guide#git-tags-format
+[tags]: /doc/gto/user-guide#git-tags-message-format
-
+To see what the example repo's [CI/CD jobs] look like, see its [GitHub Actions] page.
-"GTO: figure out what was registered/promoted" step interprets the Git tag that
-triggered the workflow and passes the information to the next steps. The
-information is used later to trigger "Publish" or "Deploy" steps (since the CI
-on the screenshot was triggered by `cv-class#dev#1` Git tag, the "Publish" step
-was skipped).
+[ci/cd jobs]:
+ https://github.com/iterative/example-gto/blob/main/.github/workflows/gto-act-on-tags.yml
+[github actions]: https://github.com/iterative/example-gto/actions
-If you want to see CI examples without reproducing this Get Started, you can
-check out them in
-[the example-repo](https://github.com/iterative/example-gto/actions).
+
## What's next?
@@ -161,10 +161,7 @@ Thanks for completing this Get Started!
- Learn how to
[specify important artifact's metainformation](/doc/gto/user-guide#annotations-in-artifactsyaml)
like `path`, `type` and `description` in the Artifact registry.
-- Learn more about [acting in CI/CD](/doc/gto/user-guide#acting-downstream) upon
+- Learn more about [acting in CI/CD](/doc/gto/user-guide#acting-in-ci-cd) upon
version registrations and stage assignments.
- Reach us out in [GH issues](https://github.com/iterative/gto/issues) or in
[Discord](https://discord.com/invite/dvwXA2N) to get your questions answered!
-
-
diff --git a/content/docs/gto/index.md b/content/docs/gto/index.md
index f68313e1..abcc8e32 100644
--- a/content/docs/gto/index.md
+++ b/content/docs/gto/index.md
@@ -1,11 +1,11 @@
# GTO Documentation
-**GTO** is a tool for creating an Artifact Registry in your Git repository. One
-of the special cases we would like to highlight is creating a
+**GTO** (Git Tag Ops) is a tool for creating an Artifact Registry
+in your Git repository. An important special case is a
[Machine Learning Model Registry](/doc/use-cases/model-registry).
Such a registry serves as a centralized place to store and operationalize your
-artifacts along with their metadata; manage model life-cycle, versions &
+artifacts along with their metadata; manage artifact's life-cycle, versions &
releases, and easily automate tests and deployments using GitOps.
@@ -35,5 +35,7 @@ specific help. We are very responsive ⚡.
✅ Check out our [GitHub repository](https://github.com/iterative/gto) and give
us a ⭐ if you like the project!
-✅ Contribute to MLEM [on GitHub](https://github.com/iterative/gto) or help us
-improve this [documentation](https://github.com/iterative/mlem.ai) 🙏.
+✅ Contribute to GTO [on GitHub](https://github.com/iterative/gto) or help us
+improve this
+[documentation](https://github.com/iterative/mlem.ai/tree/main/content/docs/gto)
+🙏.
diff --git a/content/docs/gto/install.md b/content/docs/gto/install.md
index 84fab543..edbad28e 100644
--- a/content/docs/gto/install.md
+++ b/content/docs/gto/install.md
@@ -1,18 +1,16 @@
# Installation
-To create an Artifact Registry with GTO, you only need a Git repo and GTO
-package installed. There's no need to set up any services or databases, compared
-to many other Model Registry offerings.
+You'll need [Python](https://www.python.org/) to install GTO, and
+[Git](https://git-scm.com/) to use it.
To check whether GTO is installed in your environment, run `which gto`. To check
which version is installed, run `gto --version`.
## Install as a Python library
-GTO is a Python library. It works on any OS. You can install it with a package
+GTO is distributed as a Python library, so it works on any OS. You can install it with a package
manager like [pip](https://pypi.org/project/pip/) or
-[Conda](https://docs.conda.io/en/latest/), or as a Python
-[requirement](https://pip.pypa.io/en/latest/user_guide/#requirements-files).
+[Conda](https://docs.conda.io/en/latest/).
@@ -29,5 +27,5 @@ encapsulate your local environment.
$ pip install gto
```
-This will install the `gto` command-line interface (CLI) and make the Python API
+This will install the [`gto`](https://mlem.ai/doc/gto/command-reference) command-line interface (CLI) and make the Python API
available for use in code.
diff --git a/content/docs/gto/user-guide.md b/content/docs/gto/user-guide.md
index eff8a03f..34fadbbf 100644
--- a/content/docs/gto/user-guide.md
+++ b/content/docs/gto/user-guide.md
@@ -1,7 +1,9 @@
# User Guide
-GTO helps you build a Artifact Registry out of your Git repository. It does so
-by creating Git tags of special format and managing `artifacts.yaml` metafile.
+GTO helps you build an Artifact Registry out of your Git repository. It creates
+annotated [Git tags](https://git-scm.com/book/en/v2/Git-Basics-Tagging) with a
+[special format](#git-tag-message-format) in their message, and manages an
+`artifacts.yaml` file.
Since committing large files to Git is not a good practice, you should consider
not committing your artifacts to Git. Instead, use [DVC](https://dvc.org/doc),
Git-lfs, or any method commit pointers to the binary files instead.
@@ -11,8 +13,9 @@ Git-lfs, or any method commit pointers to the binary files instead.
Using Git tag to register versions and assign stages is handy, but the Git tag
itself doesn't contain path to the artifact, type of it (it could be `model` or
`dataset`), or any other information you may find useful. For simple projects
-(e.g. single artifact) we can assume the details in a downstream system. But for
-more advanced cases, we should codify them in the registry itself.
+(e.g. single artifact) we can assume the details
+[in a CI/CD system](#acting-in-ci-cd) downstream. But for more advanced cases,
+we should codify them in the registry itself.
To keep this metainformation, GTO uses `artifacts.yaml` file. Commands like
`gto annotate` and `gto remove` are used to modify it, while `gto describe`
@@ -22,10 +25,10 @@ If you would like to see an example of `artifacts.yaml`, check out the
[example-gto](https://github.com/iterative/example-gto/blob/main/artifacts.yaml)
repo.
-## Acting downstream
+## Acting in CI/CD
-Once Git tag was pushed, you can start acting downstream. One of the popular
-options here is to use CI/CD, setting it to be triggered by pushing a Git tag.
+Once Git tags are pushed, you can start acting in systems downstream. A
+popular options is to use CI/CD (triggered when Git tags are pushed).
For general details, check out something like
[GitHub Actions](https://github.com/features/actions),
[GitLab CI/CD](https://docs.gitlab.com/ee/ci/) or
@@ -76,12 +79,12 @@ Alternatively, you can use environment variables (note the `GTO_` prefix)
$ GTO_EMOJIS=false gto show
```
-## Git tags format
+## Git tag message format
-You can work with GTO without knowing the used Git tag names conventions, since
-commands like `gto register` and `gto assign` abstracts that away from you.
+You can work with GTO without knowing these conventions, since
+[`gto` commands](/doc/command-reference) take care of everything for you.
From c7ae5ff4f588e6eac13c71bc6cd9ead1b6ebea89 Mon Sep 17 00:00:00 2001
From: Alexander Guschin <1aguschin@gmail.com>
Date: Wed, 23 Nov 2022 13:35:31 +0600
Subject: [PATCH 38/38] yarn format
---
content/docs/gto/get-started.md | 19 ++++++++++---------
content/docs/gto/index.md | 4 ++--
content/docs/gto/install.md | 8 ++++----
content/docs/gto/user-guide.md | 14 +++++++-------
4 files changed, 23 insertions(+), 22 deletions(-)
diff --git a/content/docs/gto/get-started.md b/content/docs/gto/get-started.md
index 9ece6530..1a5b2321 100644
--- a/content/docs/gto/get-started.md
+++ b/content/docs/gto/get-started.md
@@ -8,8 +8,9 @@ description:
GTO helps you manage machine learning artifact versions in a Git repository, and
their deployment stages (testing, shadow, production, etc.).
-Assuming GTO is already [installed](/doc/gto/install) in your Python environment,
-let's clone an [example model registry] and review it's current state with `gto show`:
+Assuming GTO is already [installed](/doc/gto/install) in your Python
+environment, let's clone an [example model registry] and review it's current
+state with `gto show`:
[example model registry]: https://github.com/iterative/example-gto
@@ -54,8 +55,8 @@ $ gto show
╘══════════╧══════════╧════════╧═════════╧════════════╛
```
-This creates a Git tag attached to the latest Git commit (`HEAD`) which bumps the
-artifact's version automatically (in this case from `v0.1.13` to `v0.1.14`).
+This creates a Git tag attached to the latest Git commit (`HEAD`) which bumps
+the artifact's version automatically (in this case from `v0.1.13` to `v0.1.14`).
## Assigning stages
@@ -84,8 +85,8 @@ This also creates a Git tag, which associates the latest version of `cv-class`
## Act in CI/CD upon registrations and assignments
You may have noticed that `gto` reminds you how to `git push` the [tags] created
-during registrations and promotions. The benefit of these Git-native mechanism is
-that you can act upon GTO operations in any Git-based system downstream, for
+during registrations and promotions. The benefit of these Git-native mechanism
+is that you can act upon GTO operations in any Git-based system downstream, for
example automating model deployments with CI/CD.
[tags]: /doc/gto/user-guide#git-tags-message-format
@@ -129,8 +130,7 @@ $ git push origin cv-class@v0.1.14
```
Alternatively, GTO operations can target another `--repo` directly. Let's try
-the [stage assignment](#assigning-stages) again for example, but on your
-remote:
+the [stage assignment](#assigning-stages) again for example, but on your remote:
```cli
# Replace myuser with your GitHub user below.
@@ -146,7 +146,8 @@ the target repo.
-To see what the example repo's [CI/CD jobs] look like, see its [GitHub Actions] page.
+To see what the example repo's [CI/CD jobs] look like, see its [GitHub Actions]
+page.
[ci/cd jobs]:
https://github.com/iterative/example-gto/blob/main/.github/workflows/gto-act-on-tags.yml
diff --git a/content/docs/gto/index.md b/content/docs/gto/index.md
index abcc8e32..5400eb98 100644
--- a/content/docs/gto/index.md
+++ b/content/docs/gto/index.md
@@ -1,7 +1,7 @@
# GTO Documentation
-**GTO** (Git Tag Ops) is a tool for creating an Artifact Registry
-in your Git repository. An important special case is a
+**GTO** (Git Tag Ops) is a tool for creating an Artifact Registry in your Git
+repository. An important special case is a
[Machine Learning Model Registry](/doc/use-cases/model-registry).
Such a registry serves as a centralized place to store and operationalize your
diff --git a/content/docs/gto/install.md b/content/docs/gto/install.md
index edbad28e..dd759e5f 100644
--- a/content/docs/gto/install.md
+++ b/content/docs/gto/install.md
@@ -8,8 +8,8 @@ which version is installed, run `gto --version`.
## Install as a Python library
-GTO is distributed as a Python library, so it works on any OS. You can install it with a package
-manager like [pip](https://pypi.org/project/pip/) or
+GTO is distributed as a Python library, so it works on any OS. You can install
+it with a package manager like [pip](https://pypi.org/project/pip/) or
[Conda](https://docs.conda.io/en/latest/).
@@ -27,5 +27,5 @@ encapsulate your local environment.
$ pip install gto
```
-This will install the [`gto`](https://mlem.ai/doc/gto/command-reference) command-line interface (CLI) and make the Python API
-available for use in code.
+This will install the [`gto`](https://mlem.ai/doc/gto/command-reference)
+command-line interface (CLI) and make the Python API available for use in code.
diff --git a/content/docs/gto/user-guide.md b/content/docs/gto/user-guide.md
index 34fadbbf..268416f3 100644
--- a/content/docs/gto/user-guide.md
+++ b/content/docs/gto/user-guide.md
@@ -3,10 +3,10 @@
GTO helps you build an Artifact Registry out of your Git repository. It creates
annotated [Git tags](https://git-scm.com/book/en/v2/Git-Basics-Tagging) with a
[special format](#git-tag-message-format) in their message, and manages an
-`artifacts.yaml` file.
-Since committing large files to Git is not a good practice, you should consider
-not committing your artifacts to Git. Instead, use [DVC](https://dvc.org/doc),
-Git-lfs, or any method commit pointers to the binary files instead.
+`artifacts.yaml` file. Since committing large files to Git is not a good
+practice, you should consider not committing your artifacts to Git. Instead, use
+[DVC](https://dvc.org/doc), Git-lfs, or any method commit pointers to the binary
+files instead.
## Annotations in artifacts.yaml
@@ -27,9 +27,9 @@ repo.
## Acting in CI/CD
-Once Git tags are pushed, you can start acting in systems downstream. A
-popular options is to use CI/CD (triggered when Git tags are pushed).
-For general details, check out something like
+Once Git tags are pushed, you can start acting in systems downstream. A popular
+options is to use CI/CD (triggered when Git tags are pushed). For general
+details, check out something like
[GitHub Actions](https://github.com/features/actions),
[GitLab CI/CD](https://docs.gitlab.com/ee/ci/) or
[Circle CI](https://circleci.com).
### Click to set up a working repository to try this.
-Let's fork the
-[example-gto repo](https://github.com/iterative/example-gto) repo
-(you'll need a [GitHub](https://github.com/signup) account first). For
-CI/CD to start, you'll need to enable it on the "Actions" page of your
-fork.
+Let's fork the [example-gto repo](https://github.com/iterative/example-gto) repo
+(you'll need a [GitHub](https://github.com/signup) account first). For CI/CD to
+start, you'll need to enable it on the "Actions" page of your fork.
1. [Fork the repo](https://github.com/iterative/example-gto/fork). Make sure you
uncheck "Copy the `main` branch only" to copy Git tags as well:
diff --git a/content/docs/sidebar.json b/content/docs/sidebar.json
index 9fcd0b9d..ffdef7b5 100644
--- a/content/docs/sidebar.json
+++ b/content/docs/sidebar.json
@@ -688,7 +688,59 @@
{
"slug": "command-reference",
"label": "Command Reference",
- "source": "command-reference/index.md"
+ "source": "command-reference/index.md",
+ "children": [
+ {
+ "slug": "annotate",
+ "source": "annotate.md",
+ "label": "annotate"
+ },
+ {
+ "slug": "assign",
+ "source": "assign.md",
+ "label": "assign"
+ },
+ {
+ "slug": "check-ref",
+ "source": "check-ref.md",
+ "label": "check-ref"
+ },
+ {
+ "slug": "deprecate",
+ "source": "deprecate.md",
+ "label": "deprecate"
+ },
+ {
+ "slug": "describe",
+ "source": "describe.md",
+ "label": "describe"
+ },
+ {
+ "slug": "history",
+ "source": "history.md",
+ "label": "history"
+ },
+ {
+ "slug": "register",
+ "source": "register.md",
+ "label": "register"
+ },
+ {
+ "slug": "remove",
+ "source": "remove.md",
+ "label": "remove"
+ },
+ {
+ "slug": "show",
+ "source": "show.md",
+ "label": "show"
+ },
+ {
+ "slug": "stages",
+ "source": "stages.md",
+ "label": "stages"
+ }
+ ]
}
]
}
From 0f5b8339c4f6e0e73bbd611d8710cbbe1716288d Mon Sep 17 00:00:00 2001
From: Alexander Guschin <1aguschin@gmail.com>
Date: Thu, 10 Nov 2022 20:55:03 +0600
Subject: [PATCH 19/38] lightweight fixes
---
content/docs/gto/command-reference/annotate.md | 2 +-
content/docs/gto/command-reference/assign.md | 2 +-
content/docs/gto/command-reference/check-ref.md | 2 +-
content/docs/gto/command-reference/deprecate.md | 2 +-
content/docs/gto/command-reference/describe.md | 2 +-
content/docs/gto/command-reference/history.md | 2 +-
content/docs/gto/command-reference/register.md | 2 +-
content/docs/gto/command-reference/remove.md | 2 +-
content/docs/gto/command-reference/show.md | 2 +-
content/docs/gto/command-reference/stages.md | 2 +-
10 files changed, 10 insertions(+), 10 deletions(-)
diff --git a/content/docs/gto/command-reference/annotate.md b/content/docs/gto/command-reference/annotate.md
index 999cf5a9..18238c44 100644
--- a/content/docs/gto/command-reference/annotate.md
+++ b/content/docs/gto/command-reference/annotate.md
@@ -1,6 +1,6 @@
# annotate
-Update artifact metadata annotations
+Update artifact metadata annotations.
## Synopsis
diff --git a/content/docs/gto/command-reference/assign.md b/content/docs/gto/command-reference/assign.md
index ed19fab5..d6b553e6 100644
--- a/content/docs/gto/command-reference/assign.md
+++ b/content/docs/gto/command-reference/assign.md
@@ -1,6 +1,6 @@
# assign
-Assign stage to specific artifact version
+Assign stage to specific artifact version.
## Synopsis
diff --git a/content/docs/gto/command-reference/check-ref.md b/content/docs/gto/command-reference/check-ref.md
index 30ed3437..e3f93afa 100644
--- a/content/docs/gto/command-reference/check-ref.md
+++ b/content/docs/gto/command-reference/check-ref.md
@@ -1,6 +1,6 @@
# check-ref
-Find out the artifact version registered/assigned with ref
+Find out the artifact version registered/assigned with ref.
## Synopsis
diff --git a/content/docs/gto/command-reference/deprecate.md b/content/docs/gto/command-reference/deprecate.md
index 21e5fe4c..890ce73a 100644
--- a/content/docs/gto/command-reference/deprecate.md
+++ b/content/docs/gto/command-reference/deprecate.md
@@ -1,6 +1,6 @@
# deprecate
-Deprecate artifact, deregister a version, or unassign a stage
+Deprecate artifact, deregister a version, or unassign a stage.
## Synopsis
diff --git a/content/docs/gto/command-reference/describe.md b/content/docs/gto/command-reference/describe.md
index 76dbe177..fa624963 100644
--- a/content/docs/gto/command-reference/describe.md
+++ b/content/docs/gto/command-reference/describe.md
@@ -1,6 +1,6 @@
# describe
-Display enrichments for an artifact
+Display enrichments for an artifact.
## Synopsis
diff --git a/content/docs/gto/command-reference/history.md b/content/docs/gto/command-reference/history.md
index 2f2f47d1..7709cc4f 100644
--- a/content/docs/gto/command-reference/history.md
+++ b/content/docs/gto/command-reference/history.md
@@ -1,6 +1,6 @@
# history
-Show a journal of registry operations
+Show a journal of registry operations.
## Synopsis
diff --git a/content/docs/gto/command-reference/register.md b/content/docs/gto/command-reference/register.md
index c0b6b625..56397020 100644
--- a/content/docs/gto/command-reference/register.md
+++ b/content/docs/gto/command-reference/register.md
@@ -1,7 +1,7 @@
# register
Create an artifact version to signify an important, published or released
-iteration
+iteration.
## Synopsis
diff --git a/content/docs/gto/command-reference/remove.md b/content/docs/gto/command-reference/remove.md
index 51486810..3433266e 100644
--- a/content/docs/gto/command-reference/remove.md
+++ b/content/docs/gto/command-reference/remove.md
@@ -1,6 +1,6 @@
# remove
-Remove the enrichment for given artifact
+Remove the enrichment for given artifact.
## Synopsis
diff --git a/content/docs/gto/command-reference/show.md b/content/docs/gto/command-reference/show.md
index b63d9cc2..6d78c1d7 100644
--- a/content/docs/gto/command-reference/show.md
+++ b/content/docs/gto/command-reference/show.md
@@ -1,6 +1,6 @@
# show
-Show the registry state, highest version, or what's assigned in stage
+Show the registry state, highest version, or what's assigned in stage.
## Synopsis
diff --git a/content/docs/gto/command-reference/stages.md b/content/docs/gto/command-reference/stages.md
index e9318b3b..fe140d34 100644
--- a/content/docs/gto/command-reference/stages.md
+++ b/content/docs/gto/command-reference/stages.md
@@ -1,6 +1,6 @@
# stages
-Print list of stages used in the registry
+Print list of stages used in the registry.
## Synopsis
From 9eab27ffa4680abcc0137790803453db6fd60547 Mon Sep 17 00:00:00 2001
From: Alexander Guschin <1aguschin@gmail.com>
Date: Fri, 11 Nov 2022 16:58:42 +0600
Subject: [PATCH 20/38] update annotate, assign, check-ref
---
.../docs/gto/command-reference/annotate.md | 23 +++++++++---
content/docs/gto/command-reference/assign.md | 35 ++++++++++++-------
.../docs/gto/command-reference/check-ref.md | 16 +++++----
.../docs/gto/command-reference/deprecate.md | 8 ++---
.../docs/gto/command-reference/register.md | 8 ++---
content/docs/gto/user-guide/downstream.md | 28 ++++++---------
6 files changed, 70 insertions(+), 48 deletions(-)
diff --git a/content/docs/gto/command-reference/annotate.md b/content/docs/gto/command-reference/annotate.md
index 18238c44..a386ffb9 100644
--- a/content/docs/gto/command-reference/annotate.md
+++ b/content/docs/gto/command-reference/annotate.md
@@ -16,6 +16,25 @@ arguments:
## Description
+The command adds the artifact annotation to `artifacts.yaml` file.
+
+For example:
+
+```console
+$ gto annotate nn --type model --path models/neural_network.h5
+```
+
+will create
+
+```yaml
+artifacts:
+ nn:
+ type: model
+ path: models/neural_network.h5
+```
+
+This information can be later retrieved by running `gto describe` command.
+
## Options
- `-r `, `--repo ` - Local or remote repository [default: .]
@@ -28,7 +47,3 @@ arguments:
- `--push` - Push created commit automatically (experimental) - will set
commit=True
- `-h`, `--help` - Show this message and exit.
-
-## Examples
-
-$ gto annotate nn --type model --path models/neural_network.h5
diff --git a/content/docs/gto/command-reference/assign.md b/content/docs/gto/command-reference/assign.md
index d6b553e6..b8c75c94 100644
--- a/content/docs/gto/command-reference/assign.md
+++ b/content/docs/gto/command-reference/assign.md
@@ -18,29 +18,40 @@ arguments:
## Description
+The command assigns a stage to the artifact version.
+
+For example, this will assign version "v1.0.0" of artifact "nn" to stage "prod":
+
+```cli
+$ gto assign nn --version v1.0.0 --stage prod
+Created git tag 'nn#prod#1' that assigns stage to version 'v1.0.0'
+```
+
## Options
- `-r `, `--repo ` - Local or remote repository [default: .]
- `--version ` - If you provide REF, this will be used to name new version
- `--stage ` - Stage to assign
-- `-m `, `--message ` - Message to annotate git tag with
-- `--simple ` - [auto, true, false] Use simple notation, e.g. rf#prod
- instead of rf#prod-5 [default: auto]
-- `--force` - Create a git tag even if it already exists and is in effect
+- `-m `, `--message ` - Message to annotate the Git tag with
+- `--simple ` - Use simple notation, e.g. `rf#prod` instead of `rf#prod-5`
+ [supported values: auto, true, false] [default: auto]
+- `--force` - Create the Git tag even if it already exists and is in effect
- `--push` - Push created tag automatically (experimental)
- `--sr`, `--skip-registration` - Don't register a version at specified commit
- `-h`, `--help` - Show this message and exit.
## Examples
- Assign "nn" to "prod" at specific ref:
- $ gto assign nn abcd123 --stage prod
+Assign artifact "nn" to "prod" at specific Git ref instead of supplying artifact
+version (note that this will also register a version if it doesn't exist):
- Assign specific version:
- $ gto assign nn --version v1.0.0 --stage prod
+```cli
+$ gto assign nn abcd123 --stage prod
+```
- Assign at specific ref and name version explicitly:
- $ gto assign nn abcd123 --version v1.0.0 --stage prod
+Assign stage at specific Git ref and name the version explicitly (this assumes
+that version was not registered yet):
- Assign without increment:
- $ gto assign nn HEAD --stage prod --simple
+```cli
+$ gto assign nn abcd123 --version v1.0.0 --stage prod
+```
diff --git a/content/docs/gto/command-reference/check-ref.md b/content/docs/gto/command-reference/check-ref.md
index e3f93afa..94d506a2 100644
--- a/content/docs/gto/command-reference/check-ref.md
+++ b/content/docs/gto/command-reference/check-ref.md
@@ -15,6 +15,16 @@ arguments:
## Description
+You can use `gto check-ref` to interpret a Git tag:
+
+```console
+$ gto check-ref -r build/example-gto churn#prod#3
+✅ Stage "prod" was assigned to version "v3.0.0" of artifact "churn"
+```
+
+For machine-consumable format, use `--json` flag or output specific pieces of
+information with `--name`, `--version`, `--stage` or `--event`.
+
## Options
- `-r `, `--repo ` - Local or remote repository [default: .]
@@ -24,9 +34,3 @@ arguments:
- `--event` - Show event
- `--stage` - Show artifact stage
- `-h`, `--help` - Show this message and exit.
-
-## Examples
-
- $ gto check-ref rf@v1.0.0
- $ gto check-ref rf#prod --name
- $ gto check-ref rf#prod --version
diff --git a/content/docs/gto/command-reference/deprecate.md b/content/docs/gto/command-reference/deprecate.md
index 890ce73a..baa6fe1e 100644
--- a/content/docs/gto/command-reference/deprecate.md
+++ b/content/docs/gto/command-reference/deprecate.md
@@ -20,10 +20,10 @@ arguments:
## Options
- `-r `, `--repo ` - Local or remote repository [default: .]
-- `-m `, `--message ` - Message to annotate git tag with
-- `--simple ` - [auto, true, false] Use simple notation, e.g. rf#prod
- instead of rf#prod-5 [default: auto]
-- `--force` - Create a git tag even if it already exists and is in effect
+- `-m `, `--message ` - Message to annotate the Git tag with
+- `--simple ` - Use simple notation, e.g. `rf#prod` instead of `rf#prod-5`
+ [supported values: auto, true, false] [default: auto]
+- `--force` - Create the Git tag even if it already exists and is in effect
- `-d`, `--delete` - Delete the git tag(s) instead of creating the new one
- `--push` - Push created tag automatically (experimental)
- `-h`, `--help` - Show this message and exit.
diff --git a/content/docs/gto/command-reference/register.md b/content/docs/gto/command-reference/register.md
index 56397020..841dd425 100644
--- a/content/docs/gto/command-reference/register.md
+++ b/content/docs/gto/command-reference/register.md
@@ -22,10 +22,10 @@ arguments:
- `-r `, `--repo ` - Local or remote repository [default: .]
- `--version `, `--ver ` - Version name in SemVer format
-- `-m `, `--message ` - Message to annotate git tag with
-- `--simple ` - [auto, true, false] Use simple notation, e.g. rf#prod
- instead of rf#prod-5 [default: auto]
-- `--force` - Create a git tag even if it already exists and is in effect
+- `-m `, `--message ` - Message to annotate the Git tag with
+- `--simple ` - Use simple notation, e.g. `rf#prod` instead of `rf#prod-5`
+ [supported values: auto, true, false] [default: auto]
+- `--force` - Create the Git tag even if it already exists and is in effect
- `--bump-major` - Bump major version
- `--bump-minor` - Bump minor version
- `--bump-patch` - Bump patch version
diff --git a/content/docs/gto/user-guide/downstream.md b/content/docs/gto/user-guide/downstream.md
index fecafd82..cca5d3b0 100644
--- a/content/docs/gto/user-guide/downstream.md
+++ b/content/docs/gto/user-guide/downstream.md
@@ -17,24 +17,16 @@ like REST API to check if changes happened. As an example, check out
## Getting started with CI/CD
-To act upon registrations and assignments (Git tags), you can create simple CI
-workflow. Check out
-[the example workflow in `example-gto` repo](https://github.com/iterative/example-gto/blob/main/.github/workflows/gto-act-on-tags.yml).
+To act upon registrations and assignments (Git tags), you can create a simple CI
+workflow. To see an example, check out
+[the workflow in `example-gto` repo](https://github.com/iterative/example-gto/blob/main/.github/workflows/gto-act-on-tags.yml).
The workflow uses [the GTO GH Action](https://github.com/iterative/gto-action)
-that fetches all Git tags (to correctly interpret the Registry) and finds out
-the version of the artifact that was registered, or the stage that was assigned,
-so you could use them in later steps of the CI.
+that fetches all Git tags (to correctly interpret the Registry), finds out the
+`version` of the artifact that was registered, the `stage` that was assigned,
+and annotations details such as `path`, `type`, `description`, etc, so you could
+use them in the next steps of the CI.
-## Inspecting Git tags
+## Helpful commands
-You can use `gto check-ref` to interpret the Git tag:
-
-```console
-$ gto check-ref -r build/example-gto churn#prod#3
-✅ Stage "prod" was assigned to version "v3.0.0" of artifact "churn"
-```
-
-For machine-consumable format, use `--json` flag or output specific pieces of
-information with `--name`, `--version`, `--stage` or `--event`.
-
-TODO: add `gto describe` here.
+If you would like to set up CI/CD, but don't want to use GTO GH Action, check
+out `gto check-ref` and `gto describe` commands.
From 65c9f512aa4d3593d2deea51a6a53737bc85d506 Mon Sep 17 00:00:00 2001
From: Alexander Guschin <1aguschin@gmail.com>
Date: Fri, 11 Nov 2022 20:12:30 +0600
Subject: [PATCH 21/38] write all other commands
---
.../docs/gto/command-reference/annotate.md | 53 ++++++--
content/docs/gto/command-reference/assign.md | 14 +-
.../docs/gto/command-reference/check-ref.md | 2 +-
.../docs/gto/command-reference/deprecate.md | 123 +++++++++++++++--
.../docs/gto/command-reference/describe.md | 18 ++-
content/docs/gto/command-reference/history.md | 31 ++++-
.../docs/gto/command-reference/register.md | 16 +++
content/docs/gto/command-reference/remove.md | 7 +-
content/docs/gto/command-reference/show.md | 124 ++++++++++++++++--
content/docs/gto/command-reference/stages.md | 9 +-
content/docs/gto/user-guide/artifacts.md | 46 +------
content/docs/gto/user-guide/downstream.md | 2 +-
12 files changed, 345 insertions(+), 100 deletions(-)
diff --git a/content/docs/gto/command-reference/annotate.md b/content/docs/gto/command-reference/annotate.md
index a386ffb9..6ee896e2 100644
--- a/content/docs/gto/command-reference/annotate.md
+++ b/content/docs/gto/command-reference/annotate.md
@@ -16,25 +16,62 @@ arguments:
## Description
-The command adds the artifact annotation to `artifacts.yaml` file.
+The command adds the artifact annotation to `artifacts.yaml` file. Feel free to
+modify the file directly!
-For example:
+For example,
-```console
-$ gto annotate nn --type model --path models/neural_network.h5
+```cli
+$ gto annotate awesome-model \
+ --type model \
+ --path models/neural_network.h5 \
+ --label ml
+ --label cool
+ --description "This model is very cool"
```
will create
```yaml
-artifacts:
- nn:
- type: model
- path: models/neural_network.h5
+awesome-model:
+ description: This model is very cool
+ labels:
+ - ml
+ - cool
+ path: models/neural_network.h5
+ type: model
```
This information can be later retrieved by running `gto describe` command.
+> Don't forget to commit `artifacts.yaml` with Git to associate it with the
+> latest artifact version and stage in any copy of the repo.
+
+By default GTO saves artifact as `virtual`. Use the `--must_exist` flag to tell
+GTO the artifact file is committed to Git.
+
+`, `--repo ` - Local or remote repository [default: .]
diff --git a/content/docs/gto/command-reference/assign.md b/content/docs/gto/command-reference/assign.md
index b8c75c94..868151a4 100644
--- a/content/docs/gto/command-reference/assign.md
+++ b/content/docs/gto/command-reference/assign.md
@@ -18,15 +18,19 @@ arguments:
## Description
-The command assigns a stage to the artifact version.
-
-For example, this will assign version "v1.0.0" of artifact "nn" to stage "prod":
+To assign an actionable stage for a specific artifact version use the same
+`gto assign` command. Stages can mark the artifact readiness for a specific
+consumer. You can plug in a real downsteam system via CI/CD or web hooks, e.g.
+to redeploy an ML model.
```cli
-$ gto assign nn --version v1.0.0 --stage prod
-Created git tag 'nn#prod#1' that assigns stage to version 'v1.0.0'
+$ gto assign awesome-model --version v0.0.1 --stage prod
+Created git tag 'awesome-model#prod#1' that assigns stage to 'v0.0.1'
```
+GTO creates a special Git tag in
+[the standard format](/doc/gto/user-guide/git-tags).
+
## Options
- `-r `, `--repo ` - Local or remote repository [default: .]
diff --git a/content/docs/gto/command-reference/check-ref.md b/content/docs/gto/command-reference/check-ref.md
index 94d506a2..f6c721b9 100644
--- a/content/docs/gto/command-reference/check-ref.md
+++ b/content/docs/gto/command-reference/check-ref.md
@@ -17,7 +17,7 @@ arguments:
You can use `gto check-ref` to interpret a Git tag:
-```console
+```cli
$ gto check-ref -r build/example-gto churn#prod#3
✅ Stage "prod" was assigned to version "v3.0.0" of artifact "churn"
```
diff --git a/content/docs/gto/command-reference/deprecate.md b/content/docs/gto/command-reference/deprecate.md
index baa6fe1e..9bb32bd6 100644
--- a/content/docs/gto/command-reference/deprecate.md
+++ b/content/docs/gto/command-reference/deprecate.md
@@ -17,6 +17,118 @@ arguments:
## Description
+The command supports three use cases:
+
+```cli
+# deprecate an artifact:
+$ gto deprecate nn
+
+# deprecate a version:
+$ gto deprecate nn v0.0.1
+
+# unassign a stage:
+$ gto deprecate nn v0.0.1 prod
+```
+
+### Unassigning a stage
+
+Sometimes you need to mark an artifact version no longer ready for a specific
+consumer, and maybe signal a downstream system about this. You can use
+`gto deprecate` for that:
+
+```cli
+$ gto deprecate awesome-model v0.0.1 prod
+Created git tag 'awesome-model#prod!#2' that unassigns a stage from 'v0.0.1'
+```
+
+`, `--repo ` - Local or remote repository [default: .]
@@ -27,14 +139,3 @@ arguments:
- `-d`, `--delete` - Delete the git tag(s) instead of creating the new one
- `--push` - Push created tag automatically (experimental)
- `-h`, `--help` - Show this message and exit.
-
-## Examples
-
- Deprecate an artifact:
- $ gto deprecate nn
-
- Deprecate a version:
- $ gto deprecate nn v0.0.1
-
- Unassign a stage:
- $ gto deprecate nn v0.0.1 prod
diff --git a/content/docs/gto/command-reference/describe.md b/content/docs/gto/command-reference/describe.md
index fa624963..bbfc83f7 100644
--- a/content/docs/gto/command-reference/describe.md
+++ b/content/docs/gto/command-reference/describe.md
@@ -15,6 +15,19 @@ arguments:
## Description
+To get details about an artifact (from `artifacts.yaml`) use `gto describe`:
+
+```cli
+$ gto describe churn -r https://github.com/iterative/example-gto
+{
+ "type": "model",
+ "path": "models/churn.pkl",
+ "virtual": false
+}
+```
+
+The output is in JSON format for ease of parsing programatically.
+
## Options
- `-r `, `--repo ` - Local or remote repository [default: .]
@@ -23,8 +36,3 @@ arguments:
- `--path` - Show path
- `--description` - Show description
- `-h`, `--help` - Show this message and exit.
-
-## Examples
-
- $ gto describe nn --rev HEAD
- $ gto describe nn@v0.0.1
diff --git a/content/docs/gto/command-reference/history.md b/content/docs/gto/command-reference/history.md
index 7709cc4f..b05b8702 100644
--- a/content/docs/gto/command-reference/history.md
+++ b/content/docs/gto/command-reference/history.md
@@ -15,6 +15,30 @@ arguments:
## Description
+This command prints a journal of the events that happened to an artifact. This
+allows you to audit the changes.
+
+```cli
+$ gto history churn -r https://github.com/iterative/example-gto
+╒═════════════════════╤════════════╤══════════════╤═══════════╤═════════╤══════════╤═════════════════╕
+│ timestamp │ artifact │ event │ version │ stage │ commit │ ref │
+╞═════════════════════╪════════════╪══════════════╪═══════════╪═════════╪══════════╪═════════════════╡
+│ 2022-11-09 13:40:33 │ churn │ assignment │ v3.1.1 │ dev │ 2f2a8de │ churn#dev#5 │
+│ 2022-11-09 13:40:33 │ churn │ registration │ v3.1.1 │ - │ 2f2a8de │ churn@v3.1.1 │
+│ 2022-11-08 09:53:53 │ churn │ commit │ v3.1.1 │ - │ 2f2a8de │ 2f2a8de │
+│ 2022-11-07 06:07:13 │ churn │ assignment │ v3.1.0 │ dev │ 064f173 │ churn#dev#4 │
+│ 2022-11-06 02:20:33 │ churn │ assignment │ v3.0.0 │ prod │ ddae695 │ churn#prod#3 │
+│ 2022-11-04 22:33:53 │ churn │ assignment │ v3.1.0 │ staging │ 064f173 │ churn#staging#2 │
+│ 2022-11-03 18:47:13 │ churn │ assignment │ v3.0.0 │ dev │ ddae695 │ churn#dev#1 │
+│ 2022-11-02 15:00:33 │ churn │ registration │ v3.1.0 │ - │ 064f173 │ churn@v3.1.0 │
+│ 2022-11-01 11:13:53 │ churn │ commit │ v3.1.0 │ - │ 064f173 │ 064f173 │
+│ 2022-10-28 23:53:53 │ churn │ registration │ v3.0.0 │ - │ ddae695 │ churn@v3.0.0 │
+│ 2022-10-27 20:07:13 │ churn │ commit │ v3.0.0 │ - │ ddae695 │ ddae695 │
+╘═════════════════════╧════════════╧══════════════╧═══════════╧═════════╧══════════╧═════════════════╛
+```
+
+Use `--all-branches` and `--all-commits` to read more than just HEAD.
+
## Options
- `-r `, `--repo ` - Local or remote repository [default: .]
@@ -24,10 +48,3 @@ arguments:
- `--plain` - Print table in grep-able format
- `--ascending`, `--asc` - Show new first
- `-h`, `--help` - Show this message and exit.
-
-## Examples
-
- $ gto history nn
-
- Use --all-branches and --all-commits to read more than just HEAD:
- $ gto history nn --all-commits
diff --git a/content/docs/gto/command-reference/register.md b/content/docs/gto/command-reference/register.md
index 841dd425..4a6c574f 100644
--- a/content/docs/gto/command-reference/register.md
+++ b/content/docs/gto/command-reference/register.md
@@ -18,6 +18,22 @@ arguments:
## Description
+Registering a version is usually done to mark significant changes to the
+artifact. To release a new version (including the very first one), use
+`gto register`.
+
+```cli
+$ gto register awesome-model HEAD --version v0.0.1
+Created git tag 'awesome-model@v0.0.1' that registers a version
+```
+
+GTO creates a special Git tag for the artifact version, in
+[the standard format](/doc/gto/user-guide/git-tags).
+
+The version is now associated to the current Git commit (`HEAD`). You can use
+another Git commit if you provide it's hexsha as an additional argument, like
+`$ gto register awesome-model abc1234`.
+
## Options
- `-r `, `--repo ` - Local or remote repository [default: .]
diff --git a/content/docs/gto/command-reference/remove.md b/content/docs/gto/command-reference/remove.md
index 3433266e..bb673cd2 100644
--- a/content/docs/gto/command-reference/remove.md
+++ b/content/docs/gto/command-reference/remove.md
@@ -14,6 +14,9 @@ arguments:
## Description
+This command removes the artifact annotation from `artifacts.yaml`. Don't forget
+to commit the change.
+
## Options
- `-r `, `--repo ` - Local or remote repository [default: .]
@@ -21,7 +24,3 @@ arguments:
- `--push` - Push created commit automatically (experimental) - will set
commit=True
- `-h`, `--help` - Show this message and exit.
-
-## Examples
-
- $ gto remove nn
diff --git a/content/docs/gto/command-reference/show.md b/content/docs/gto/command-reference/show.md
index 6d78c1d7..68e961a6 100644
--- a/content/docs/gto/command-reference/show.md
+++ b/content/docs/gto/command-reference/show.md
@@ -17,6 +17,118 @@ arguments:
## Description
+This is the entire state of the registry: all artifacts, their latest versions,
+and the versions in each stage.
+
+```cli
+$ gto show -r https://github.com/iterative/example-gto
+╒══════════╤══════════╤════════╤═════════╤════════════╕
+│ name │ latest │ #dev │ #prod │ #staging │
+╞══════════╪══════════╪════════╪═════════╪════════════╡
+│ churn │ v3.1.1 │ v3.1.1 │ v3.0.0 │ v3.1.0 │
+│ segment │ v0.4.1 │ v0.4.1 │ - │ - │
+│ cv-class │ v0.1.13 │ - │ - │ - │
+╘══════════╧══════════╧════════╧═════════╧════════════╛
+```
+
+Here we'll see artifacts that have Git tags or are annotated in
+`artifacts.yaml`. The artifacts that have annotation, but have no Git tags, are
+considered yet `unregistered` and will be marked with an asterisk, e.g.
+`*annotated`. Use `--all-branches` or `--all-commits` to read `artifacts.yaml`
+from more commits than just `HEAD`.
+
+Add an artifact name to print all of its versions instead:
+
+```cli
+$ gto show churn
+╒════════════╤═══════════╤══════════════╤═════════════════════╤══════════════╕
+│ artifact │ version │ stage │ created_at │ ref │
+╞════════════╪═══════════╪══════════════╪═════════════════════╪══════════════╡
+│ churn │ v3.1.0 │ dev, staging │ 2022-08-28 16:58:50 │ churn@v3.1.0 │
+│ churn │ v3.0.0 │ prod │ 2022-08-24 01:52:10 │ churn@v3.0.0 │
+╘════════════╧═══════════╧══════════════╧═════════════════════╧══════════════╛
+```
+
+Note, that by default, assignments are sorted by the creation time (the latest
+assignment wins). You can sort them by Semver with `--sort semver` option (the
+greatest version in stage wins).
+
+Finally, you can show the greatest version of the artifact, or what's currently
+in a stage, using shortcuts like:
+
+```cli
+$ gto show churn@greatest
+╒════════════╤═══════════╤═════════╤═════════════════════╤══════════════╕
+│ artifact │ version │ stage │ created_at │ ref │
+╞════════════╪═══════════╪═════════╪═════════════════════╪══════════════╡
+│ churn │ v3.1.1 │ dev │ 2022-11-09 13:40:33 │ churn@v3.1.1 │
+╘════════════╧═══════════╧═════════╧═════════════════════╧══════════════╛
+
+$ gto show churn#prod
+╒════════════╤═══════════╤═════════╤═════════════════════╤══════════════╕
+│ artifact │ version │ stage │ created_at │ ref │
+╞════════════╪═══════════╪═════════╪═════════════════════╪══════════════╡
+│ churn │ v3.0.0 │ prod │ 2022-10-28 23:53:53 │ churn@v3.0.0 │
+╘════════════╧═══════════╧═════════╧═════════════════════╧══════════════╛
+```
+
+By default, GTO is configured to adhere to what we call "Environments"
+mechanics: a single artifact version can be in multiple stages, but if you take
+a specific stage, there will be only one version in it.
+
+There are two other approaches that you may want to use - see the details under
+collapsible sections below.
+
+`, `--repo ` - Local or remote repository [default: .]
@@ -36,15 +148,3 @@ arguments:
- `--sort ` - Order assignments by timestamp or semver [default:
timestamp]
- `-h`, `--help` - Show this message and exit.
-
-## Examples
-
- Show the registry:
- $ gto show
-
- Show versions of specific artifact in registry:
- $ gto show nn
-
- Show greatest version or what's in stage:
- $ gto show nn@greatest
- $ gto show nn#prod
diff --git a/content/docs/gto/command-reference/stages.md b/content/docs/gto/command-reference/stages.md
index fe140d34..72997030 100644
--- a/content/docs/gto/command-reference/stages.md
+++ b/content/docs/gto/command-reference/stages.md
@@ -11,6 +11,10 @@ usage: gto stages [-r ] [--allowed] [--used] [--json]
## Description
+This command is used to get a list of stages used in the registry - whether
+directly from [the config file](/doc/gto/user-guide/config), or from all GTO Git
+tags that exist.
+
## Options
- `-r `, `--repo ` - Local or remote repository [default: .]
@@ -18,8 +22,3 @@ usage: gto stages [-r ] [--allowed] [--used] [--json]
- `--used` - Show stages that were ever used (from all git tags)
- `--json` - Print output in json format
- `-h`, `--help` - Show this message and exit.
-
-## Examples
-
- $ gto stages
- $ gto stages --allowed
diff --git a/content/docs/gto/user-guide/artifacts.md b/content/docs/gto/user-guide/artifacts.md
index 095f1acf..b3e63d00 100644
--- a/content/docs/gto/user-guide/artifacts.md
+++ b/content/docs/gto/user-guide/artifacts.md
@@ -7,45 +7,9 @@ itself doesn't contain path to the artifact, type of it (it could be `model` or
more advanced cases, we should codify them in the registry itself.
To keep this metainformation, GTO uses `artifacts.yaml` file. Commands like
-[`annotate`](/doc/gto/command-reference/annotate) and
-[`remove`](/doc/gto/command-reference/remove) are used to modify it, while
-[`describe`](/doc/gto/command-reference/describe) helps get them when they're
-needed.
+`gto annotate` and `gto remove` are used to modify it, while `gto describe`
+helps get them when they're needed.
-Let's see an example of `artifacts.yaml` (another example can be found in
-[example-gto](https://github.com/iterative/example-gto/blob/main/artifacts.yaml)):
-
-```yaml
-awesome: # artifact name
- description: This model is very cool
- labels:
- - ml
- - cool
- path: models/awesome.pkl
- type: model
-```
-
-By default GTO saves artifact as `virtual`. Use the `--must_exist` flag to tell
-GTO the artifact file is committed to Git.
-
-
---
.../docs/gto/command-reference/annotate.md | 12 +++---
content/docs/gto/command-reference/index.md | 7 +---
content/docs/gto/get-started.md | 40 ++++++-------------
3 files changed, 21 insertions(+), 38 deletions(-)
diff --git a/content/docs/gto/command-reference/annotate.md b/content/docs/gto/command-reference/annotate.md
index 6ee896e2..b991f882 100644
--- a/content/docs/gto/command-reference/annotate.md
+++ b/content/docs/gto/command-reference/annotate.md
@@ -44,8 +44,12 @@ awesome-model:
This information can be later retrieved by running `gto describe` command.
-> Don't forget to commit `artifacts.yaml` with Git to associate it with the
-> latest artifact version and stage in any copy of the repo.
+
+
+Don't forget to commit `artifacts.yaml` with Git to associate it with the
+latest artifact version and stage in any copy of the repo.
+
+
By default GTO saves artifact as `virtual`. Use the `--must_exist` flag to tell
GTO the artifact file is committed to Git.
@@ -65,10 +69,6 @@ GTO the artifact file is committed to Git.
immutability later, e.g. if `s3://mybucket/myfile` changes the registry won't
know it, nor have a way to recover the original file.
-> In future versions, we will support additional enrichments: useful information
-> that other tools like [DVC](https://dvc.org/) and [MLEM](https://mlem.ai/) can
-> provide about the artifacts. This will allow treating DVC repo outputs as
-> usual artifacts instead of `virtual` ones.
diff --git a/content/docs/gto/command-reference/index.md b/content/docs/gto/command-reference/index.md
index 1bbdd940..fb0ef6ef 100644
--- a/content/docs/gto/command-reference/index.md
+++ b/content/docs/gto/command-reference/index.md
@@ -1,9 +1,6 @@
-# Using GTO Commands
+# GTO Command Reference
-GTO is a command line tool. Here, we provide the specifications, complete
-descriptions, and comprehensive usage examples for different `gto` commands.
-
-For a list of all commands, type `gto -h`.
+GTO is a command line tool (`gto` binary). For a list of all commands, type `gto -h`.
## Read the registry
diff --git a/content/docs/gto/get-started.md b/content/docs/gto/get-started.md
index 16964f9e..a596d9c4 100644
--- a/content/docs/gto/get-started.md
+++ b/content/docs/gto/get-started.md
@@ -12,23 +12,13 @@ shadow, production, etc.). Git-native mechanisms are used, so you can automate
the delivery of your software project with CI/CD, and adopt a GitOps approach in
general.
-This Get Started will walk you through basic GTO concepts and actions you would
-like to do in the Artifact Registry.
-
-## Showing the current state
-
Assuming GTO is already [installed](/doc/gto/install) in your active Python
-environment, let's clone the example repo:
+environment, let's clone an example [ML Model Registry] and review it
+with `git show`:
```cli
$ git clone https://github.com/iterative/example-gto
$ cd example-gto
-```
-
-This repo represents a simple example of Machine Learning Model Registry (that
-is just a special case of an Artifact Registry for us). Let's review it:
-
-```cli
$ gto show
╒══════════╤══════════╤════════╤═════════╤════════════╕
│ name │ latest │ #dev │ #prod │ #staging │
@@ -39,18 +29,14 @@ $ gto show
╘══════════╧══════════╧════════╧═════════╧════════════╛
```
-Here we have 3 models: `churn`, `segment` and `cv-class`. The latest versions of
-them are shown in the column named `latest`. The latest is selected as the one
-having the greatest [SemVer](https://semver.org).
-
-Model versions could be promoted to different stages. Here we have 3 of them:
-`dev`, `prod` and `staging`. When a model was never promoted to a stage, we see
-`-` in the field.
+3 artifacts (models `churn`, `segment`, and `cv-class`) and their `latest`
+versions (per [SemVer](https://semver.org)) are listed. We also have 3 stages:
+`dev`, `prod`, and `staging`. The model versions (if any) assigned to each
+stage are shown.
## Registering versions and assigning stages
-GTO can [register version](/doc/gto/command-reference/register) of artifacts and
-[assign stages to them](/doc/gto/command-reference/assign). Both functionalities
+You can `gto register` artifacts and `gto assign` them to stages. Both functionalities
work in a similar way, so let's walkthough only one of them here.
Let's assume the version `v0.1.13` of `cv-class` looks very promising, and now
@@ -64,8 +50,7 @@ To push the changes upstream, run:
```
GTO created a Git tag with a special format that contains instruction to assign
-a stage to a version. We can push to Git repository to start the CI, but let's
-ensure that changed our Registry first.
+a stage to a version.
```cli
$ gto show
@@ -79,12 +64,13 @@ $ gto show
╘══════════╧══════════╧═════════╧═════════╧════════════╛
```
-The `gto show` output confirms our expectation.
+We can push to Git repository to start a CI job!
## Acting downstream
-The power of using Git tags to register versions and assign stages is simple: we
-can act upon them in well-known way - in CI/CD.
+GTO [uses Git tags] to register artifact versions and assign them to stages. This
+means we can act upon these operations in any Git-based system such as many
+CI/CD platforms.
+
+### Virtual vs. Physical artifacts
+
+- Physical files/directories are committed to the repo. When you create a new
+ version or assign a stage to it, Git guarantees that it's immutable -- you can
+ return a year later and get the same artifact by providing a version.
+
+- Virtual artifacts could be an external path (e.g. `s3://mybucket/myfile`) or a
+ local path to a metafile representing an externally stored artifact file (as
+ [with DVC](https://dvc.org/doc/start/data-management)). In this case, GTO
+ can't pin versions to a physical state of the artifact and guarantee its
+ immutability later, e.g. if `s3://mybucket/myfile` changes the registry won't
+ know it, nor have a way to recover the original file.
+
+> In future versions, we will support additional enrichments: useful information
+> that other tools like [DVC](https://dvc.org/) and [MLEM](https://mlem.ai/) can
+> provide about the artifacts. This will allow treating DVC repo outputs as
+> usual artifacts instead of `virtual` ones.
+
+
+
## Options
- `-r
+
+### Unassigning a stage: some details and options
+
+GTO creates a special Git tag in
+[the standard format](/doc/gto/user-guide/git-tags).
+
+Note, that later you can create this stage again, if you need to, by calling
+`$ gto assign` again.
+
+You also may want to delete the git tag instead of creating a new one. This is
+useful if you don't want to keep extra tags in you Git repo, don't need history
+and don't want to trigger a CI/CD or another downstream system. For that, you
+can use:
+
+```cli
+$ gto deprecate awesome-model v0.0.1 prod --delete
+Deleted git tag 'awesome-model#prod#1' that assigned a stage to 'v0.0.1'
+To push the changes upstream, run:
+git push origin awesome-model#prod#1 --delete
+```
+
+
+
+### Deregister a version
+
+Sometimes you need mark a specific artifact version as a no longer ready for
+usage. You could just delete a git tag, but if you want to preserve a history of
+the actions, you may again use `gto deprecate`.
+
+```cli
+$ gto deprecate awesome-model v0.0.1
+Created git tag 'awesome-model@v0.0.1!' that deregistered a version.
+```
+
+
+
+### Deregister a version: some details and options
+
+If you want to deregister the version by deleting the Git tags itself, you could
+use
+
+```cli
+$ gto deprecate awesome-model v0.0.1 --delete
+Deleted git tag 'awesome-model@v0.0.1' that registered a version.
+Deleted git tag 'awesome-model#prod#1' that assigned a stage to 'v0.0.1'.
+Deleted git tag 'awesome-model#prod!#2' that unassigned a stage to 'v0.0.1'.
+To push the changes upstream, run:
+git push origin awesome-model@v0.0.1 awesome-model#prod#1 awesome-model#prod!#2 --delete
+```
+
+This includes all Git tags related to the version: a tag that registered it and
+all tags that assigned stages to it.
+
+
+
+### Deprecating an artifact
+
+Sometimes you need to need to mark the artifact as "deprecated", usually meaning
+it's outdated and will no longer be developed. To do this, you could run:
+
+```cli
+$ gto deprecate awesome-model
+Created Git tag 'awesome-model@deprecated' that deprecates an artifact.
+```
+
+
+
+### Deprecating an artifact: some details and options
+
+With `awesome-model@deprecated` Git tag the artifact will be considered
+deprecated until you register a new version or assign a new stage to it after
+the deprecation.
+
+If you want to deprecate an artifact by deleting git tags, you'll need to delete
+all of them for the artifact. You could do that with
+
+```cli
+$ gto deprecate awesome-model --delete
+Deleted git tag 'awesome-model@v0.0.1' that registered a version.
+Deleted git tag 'awesome-model#prod#1' that assigned a stage to 'v0.0.1'.
+Deleted git tag 'awesome-model#prod!#2' that unassigned a stage to 'v0.0.1'.
+To push the changes upstream, run:
+git push origin awesome-model@v0.0.1 awesome-model#prod#1 awesome-model#prod!#2 --delete
+```
+
+
+
## Options
- `-r
+
+### Enable multiple versions in the same Stage workflow
+
+Note: this functionality is experimental and subject to change. If you find it
+useful, please share your feedback in GH issues to help us make it stable.
+
+If you would like to see more than a single version assigned in a stage, use
+`--vs` (short for `--versions-per-stage`), e.g. `-1` to show all versions.
+
+```cli
+$ gto show churn --vs -1
+╒════════════╤═══════════╤══════════════╤═════════════════════╤══════════════╕
+│ artifact │ version │ stage │ created_at │ ref │
+╞════════════╪═══════════╪══════════════╪═════════════════════╪══════════════╡
+│ churn │ v3.1.0 │ dev, staging │ 2022-08-28 16:58:50 │ churn@v3.1.0 │
+│ churn │ v3.0.0 │ dev, prod │ 2022-08-24 01:52:10 │ churn@v3.0.0 │
+╘════════════╧═══════════╧══════════════╧═════════════════════╧══════════════╛
+```
+
+
+
+
+
+### Enable Kanban-like workflow
+
+Note: this functionality is experimental and subject to change. If you find it
+useful, please share your feedback in GH issues to help us make it stable.
+
+If you would like the latest stage to replace all the previous stages for an
+artifact version, use `--vs` flag combined with `--av`
+(`--assignments-per-version` for short):
+
+```cli
+$ gto show churn --av 1 --vs -1
+╒════════════╤═══════════╤═════════╤═════════════════════╤══════════════╕
+│ artifact │ version │ stage │ created_at │ ref │
+╞════════════╪═══════════╪═════════╪═════════════════════╪══════════════╡
+│ churn │ v3.1.0 │ staging │ 2022-08-28 16:58:50 │ churn@v3.1.0 │
+│ churn │ v3.0.0 │ dev │ 2022-08-24 01:52:10 │ churn@v3.0.0 │
+╘════════════╧═══════════╧═════════╧═════════════════════╧══════════════╛
+```
+
+In this case the version will always have a single stage (or have no stage at
+all). This resembles Kanban workflow, when you "move" your artifact version from
+one column ("stage-1") to another ("stage-2"). This is how MLFlow and some other
+Model Registries work.
+
+
+
## Options
- `-r
-
-### Virtual vs. Physical artifacts
-
-- Physical files/directories are committed to the repo. When you create a new
- version or assign a stage to it, Git guarantees that it's immutable -- you can
- return a year later and get the same artifact by providing a version.
-
-- Virtual artifacts could be an external path (e.g. `s3://mybucket/myfile`) or a
- local path to a metafile representing an externally stored artifact file (as
- [with DVC](https://dvc.org/doc/start/data-management)). In this case, GTO
- can't pin versions to a physical state of the artifact and guarantee its
- immutability later, e.g. if `s3://mybucket/myfile` changes the registry won't
- know it, nor have a way to recover the original file.
-
-> In future versions, we will support additional enrichments: useful information
-> that other tools like [DVC](https://dvc.org/) and [MLEM](https://mlem.ai/) can
-> provide about the artifacts. This will allow treating DVC repo outputs as
-> usual artifacts instead of `virtual` ones.
-
-
+If you would like to see an example of `artifacts.yaml`, check out the
+[example-gto](https://github.com/iterative/example-gto/blob/main/artifacts.yaml)
+repo.
diff --git a/content/docs/gto/user-guide/downstream.md b/content/docs/gto/user-guide/downstream.md
index cca5d3b0..596c819a 100644
--- a/content/docs/gto/user-guide/downstream.md
+++ b/content/docs/gto/user-guide/downstream.md
@@ -29,4 +29,4 @@ use them in the next steps of the CI.
## Helpful commands
If you would like to set up CI/CD, but don't want to use GTO GH Action, check
-out `gto check-ref` and `gto describe` commands.
+out `gto show`, `gto check-ref` and `gto describe` commands.
From e5a75b336a42ff529c828d459dff5a00c86d7d55 Mon Sep 17 00:00:00 2001
From: Alexander Guschin <1aguschin@gmail.com>
Date: Thu, 17 Nov 2022 16:55:09 +0600
Subject: [PATCH 22/38] Apply suggestions from code review
Co-authored-by: Jorge Orpinel
@@ -98,7 +84,7 @@ start, you'll need to enable it on the "Actions" page of your fork.
uncheck "Copy the `main` branch only" to copy Git tags as well:
-2. Enable workflows in your repo, for a Git tag to trigger CI:
+2. Enable workflows in your repo for a new Git tag to trigger CI:
From 570b77a1ab570d506c45be36cd8ddbc5b14e90c2 Mon Sep 17 00:00:00 2001
From: Alexander Guschin <1aguschin@gmail.com>
Date: Thu, 17 Nov 2022 18:03:54 +0600
Subject: [PATCH 23/38] return register to GS
---
.../docs/gto/command-reference/annotate.md | 2 +-
content/docs/gto/get-started.md | 99 +++++++++----------
2 files changed, 49 insertions(+), 52 deletions(-)
diff --git a/content/docs/gto/command-reference/annotate.md b/content/docs/gto/command-reference/annotate.md
index b991f882..a941c0a9 100644
--- a/content/docs/gto/command-reference/annotate.md
+++ b/content/docs/gto/command-reference/annotate.md
@@ -1,6 +1,6 @@
# annotate
-Update artifact metadata annotations.
+Create or update metainformation about an artifact.
## Synopsis
diff --git a/content/docs/gto/get-started.md b/content/docs/gto/get-started.md
index a596d9c4..76fb2f3a 100644
--- a/content/docs/gto/get-started.md
+++ b/content/docs/gto/get-started.md
@@ -13,8 +13,10 @@ the delivery of your software project with CI/CD, and adopt a GitOps approach in
general.
Assuming GTO is already [installed](/doc/gto/install) in your active Python
-environment, let's clone an example [ML Model Registry] and review it
-with `git show`:
+environment, let's clone an example [ML Model Registry] and review it with
+`git show`:
+
+[ml model registry]: /doc/use-cases/model-registry
```cli
$ git clone https://github.com/iterative/example-gto
@@ -30,48 +32,65 @@ $ gto show
```
3 artifacts (models `churn`, `segment`, and `cv-class`) and their `latest`
-versions (per [SemVer](https://semver.org)) are listed. We also have 3 stages:
-`dev`, `prod`, and `staging`. The model versions (if any) assigned to each
-stage are shown.
-
-## Registering versions and assigning stages
+versions (per [SemVer](https://semver.org)) are listed. We also have 3 stages:
+`dev`, `prod`, and `staging`. The model versions (if any) assigned to each stage
+are shown.
-You can `gto register` artifacts and `gto assign` them to stages. Both functionalities
-work in a similar way, so let's walkthough only one of them here.
+## Registering a new version
-Let's assume the version `v0.1.13` of `cv-class` looks very promising, and now
-we want to promote it to `dev` to test it:
+Registering a version is usually done to mark significant changes to the
+artifact. Let's assume this is the case for the `cv-class` model in `HEAD` and
+we would like to create a new version of it:
```cli
-$ gto assign cv-class --version v0.1.13 --stage dev
-Created git tag 'cv-class#dev#1' that assigns stage to version 'v0.1.13'
+$ gto register cv-class
+Created git tag 'cv-class@v0.1.14' that registers version
To push the changes upstream, run:
- git push origin cv-class#dev#1
+ git push origin cv-class@v0.1.14
+
+# this version will be shown in `gto show` now:
+$ gto show
+╒══════════╤══════════╤════════╤═════════╤════════════╕
+│ name │ latest │ #dev │ #prod │ #staging │
+╞══════════╪══════════╪════════╪═════════╪════════════╡
+│ churn │ v3.1.1 │ v3.1.1 │ v3.0.0 │ v3.1.0 │
+│ segment │ v0.4.1 │ v0.4.1 │ - │ - │
+│ cv-class │ v0.1.14 │ - │ - │ - │
+╘══════════╧══════════╧════════╧═════════╧════════════╛
```
-GTO created a Git tag with a special format that contains instruction to assign
-a stage to a version.
+## Assigning stages
+
+Let's assume the version we just registered looks very promising, and we want to
+promote it to `dev` to test it:
```cli
+$ gto assign cv-class --stage dev
+Created git tag 'cv-class#dev#1' that assigns stage to version 'v0.1.14'
+To push the changes upstream, run:
+ git push origin cv-class#dev#1
+
+# stage assignment can be seen with `gto show`:
$ gto show
╒══════════╤══════════╤═════════╤═════════╤════════════╕
│ name │ latest │ #dev │ #prod │ #staging │
╞══════════╪══════════╪═════════╪═════════╪════════════╡
-│ churn │ v3.1.1 │ v3.1.1 │ v3.0.0 │ v3.1.1 │
+│ churn │ v3.1.1 │ v3.1.1 │ v3.0.0 │ v3.1.0 │
│ segment │ v0.4.1 │ v0.4.1 │ - │ - │
-│ cv-class │ v0.1.13 │ v0.1.13 │ - │ - │
-│ awesome │ v0.0.1 │ - │ - │ - │
+│ cv-class │ v0.1.14 │ v0.1.14 │ - │ - │
╘══════════╧══════════╧═════════╧═════════╧════════════╛
```
We can push to Git repository to start a CI job!
-## Acting downstream
+## Act on new versions and stage assignments in CI/CD
-GTO [uses Git tags] to register artifact versions and assign them to stages. This
-means we can act upon these operations in any Git-based system such as many
+GTO [uses Git tags] to register artifact versions and assign them to stages.
+This means we can act upon these operations in any Git-based system such as many
CI/CD platforms.
+[uses git tags]: /doc/gto/user-guide/git-tags
+
-2. Enable workflows in your repo, for a Git tag to trigger CI:
+2. Enable workflows in your repo for a new Git tag to trigger CI:
### Click to set up a working repository to try this.
@@ -93,15 +112,19 @@ Let's do the same thing we did locally, but for your remote repo. Don't forget
to replace the URL:
```cli
-$ gto assign cv-class --version v0.1.13 --stage dev \
+# since we didn't register a version on remote, GTO will do that for us:
+$ gto assign cv-class --stage dev \
--repo https://github.com/aguschin/example-gto
-Created git tag 'cv-class#dev#1' that assigns stage to version 'v0.1.13'
+Created git tag 'cv-class@v0.1.14' that registers a version
+Running `git push origin cv-class@v0.1.14`
+Successfully pushed git tag cv-class@v0.1.14 on remote.
+Created git tag 'cv-class#dev#1' that assigns stage to version 'v0.1.14'
Running `git push origin cv-class#dev#1`
Successfully pushed git tag cv-class#dev#1 on remote.
```
Now the CI/CD should start, and you should see that we found out: it was
-`cv-class` artifact, version `v0.1.13` that was assigned to `dev` stage. Using
+`cv-class` artifact, version `v0.1.14` that was assigned to `dev` stage. Using
this information, the step `Deploy (act on assigning a new stage)` was executed
(while `Publish (act on registering a new version)` was skipped):
@@ -128,29 +151,3 @@ Thanks for completing this Get Started!
[Studio docs](https://dvc.org/doc/studio).
- If you want to learn about using MLEM to deploying your model upon GTO stage
assignments, check out [MLEM docs](/doc/).
-
-
From d747e562374717f82b457c35278157753048c097 Mon Sep 17 00:00:00 2001
From: Alexander Guschin <1aguschin@gmail.com>
Date: Thu, 17 Nov 2022 18:21:07 +0600
Subject: [PATCH 24/38] simplify the docs structure, hide 'why gto?'
---
.../docs/gto/command-reference/annotate.md | 5 +-
content/docs/gto/command-reference/index.md | 3 +-
.../gto/{user-guide/storages.md => dvc.md} | 2 +
content/docs/gto/index.md | 4 +-
content/docs/gto/{user-guide => }/mlem.md | 2 +
content/docs/gto/user-guide.md | 109 ++++++++++++++++++
content/docs/gto/user-guide/artifacts.md | 15 ---
content/docs/gto/user-guide/config.md | 20 ----
content/docs/gto/user-guide/downstream.md | 32 -----
content/docs/gto/user-guide/git-tags.md | 31 -----
content/docs/gto/user-guide/index.md | 17 ---
content/docs/sidebar.json | 56 ++-------
12 files changed, 130 insertions(+), 166 deletions(-)
rename content/docs/gto/{user-guide/storages.md => dvc.md} (58%)
rename content/docs/gto/{user-guide => }/mlem.md (62%)
create mode 100644 content/docs/gto/user-guide.md
delete mode 100644 content/docs/gto/user-guide/artifacts.md
delete mode 100644 content/docs/gto/user-guide/config.md
delete mode 100644 content/docs/gto/user-guide/downstream.md
delete mode 100644 content/docs/gto/user-guide/git-tags.md
delete mode 100644 content/docs/gto/user-guide/index.md
diff --git a/content/docs/gto/command-reference/annotate.md b/content/docs/gto/command-reference/annotate.md
index a941c0a9..6d75989e 100644
--- a/content/docs/gto/command-reference/annotate.md
+++ b/content/docs/gto/command-reference/annotate.md
@@ -46,8 +46,8 @@ This information can be later retrieved by running `gto describe` command.
-Don't forget to commit `artifacts.yaml` with Git to associate it with the
-latest artifact version and stage in any copy of the repo.
+Don't forget to commit `artifacts.yaml` with Git to associate it with the latest
+artifact version and stage in any copy of the repo.
@@ -69,7 +69,6 @@ GTO the artifact file is committed to Git.
immutability later, e.g. if `s3://mybucket/myfile` changes the registry won't
know it, nor have a way to recover the original file.
-
## Options
diff --git a/content/docs/gto/command-reference/index.md b/content/docs/gto/command-reference/index.md
index fb0ef6ef..9f5d9262 100644
--- a/content/docs/gto/command-reference/index.md
+++ b/content/docs/gto/command-reference/index.md
@@ -1,6 +1,7 @@
# GTO Command Reference
-GTO is a command line tool (`gto` binary). For a list of all commands, type `gto -h`.
+GTO is a command line tool (`gto` binary). For a list of all commands, type
+`gto -h`.
## Read the registry
diff --git a/content/docs/gto/user-guide/storages.md b/content/docs/gto/dvc.md
similarity index 58%
rename from content/docs/gto/user-guide/storages.md
rename to content/docs/gto/dvc.md
index f1ba9dcb..315a042b 100644
--- a/content/docs/gto/user-guide/storages.md
+++ b/content/docs/gto/dvc.md
@@ -1,3 +1,5 @@
## Working with large artifacts
...Git-lfs, DVC and what else is there...
+
+...(I'll remove this page before merging this PR)...
diff --git a/content/docs/gto/index.md b/content/docs/gto/index.md
index e8861b26..49878390 100644
--- a/content/docs/gto/index.md
+++ b/content/docs/gto/index.md
@@ -18,8 +18,8 @@ releases, and easily automate tests and deployments using GitOps.
Study the detailed inner-workings of GTO in its user guide.
-
+
+## Why do we need such a Registry?
Such a registry serves as a centralized place to store and operationalize your
artifacts along with their metadata; manage model life-cycle, versions &
@@ -21,7 +25,7 @@ GTO helps you achieve all of them in a [GitOps](https://www.gitops.tech) way. If
you would like to see an example, please follow
[Get Started](/doc/gto/get-started).
-## Why GTO?
+
In Software Engineering, Git is a heart of the Software system. The code is
committed to Git and CI/CD triggers on new commits making the downstream action
diff --git a/content/docs/sidebar.json b/content/docs/sidebar.json
index c93a38ed..dd36d9e1 100644
--- a/content/docs/sidebar.json
+++ b/content/docs/sidebar.json
@@ -631,6 +631,11 @@
"source": "install.md",
"label": "Installation"
},
+ {
+ "slug": "why-gto",
+ "source": "why-gto.md",
+ "label": "Why GTO"
+ },
{
"slug": "get-started",
"label": "Get Started",
From 0c0931e741775bace26649def467ea875ab18f0a Mon Sep 17 00:00:00 2001
From: rogermparent
diff --git a/content/docs/gto/user-guide.md b/content/docs/gto/user-guide.md
index c5a58f5b..941cfe72 100644
--- a/content/docs/gto/user-guide.md
+++ b/content/docs/gto/user-guide.md
@@ -55,7 +55,7 @@ use them in the next steps of the CI.
If you would like to set up CI/CD, but don't want to use GTO GH Action, check
out `gto show`, `gto check-ref` and `gto describe` commands.
-## Configuring types and stages
+## Configuring GTO
To configure GTO, use file `.gto` in the root of your repo:
From 6ddfb7054cc920b6b890556d46e9815c4ed36511 Mon Sep 17 00:00:00 2001
From: Alexander Guschin <1aguschin@gmail.com>
Date: Mon, 21 Nov 2022 19:38:01 +0600
Subject: [PATCH 30/38] fix lint
---
content/docs/gto/command-reference/stages.md | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/content/docs/gto/command-reference/stages.md b/content/docs/gto/command-reference/stages.md
index 0950f0a2..d9f4bd8e 100644
--- a/content/docs/gto/command-reference/stages.md
+++ b/content/docs/gto/command-reference/stages.md
@@ -12,8 +12,8 @@ usage: gto stages [-r ] [--allowed] [--used] [--json]
## Description
This command is used to get a list of stages used in the registry - whether
-directly from [the config file](/doc/gto/user-guide#configuring-gto), or from all GTO Git
-tags that exist.
+directly from [the config file](/doc/gto/user-guide#configuring-gto), or from
+all GTO Git tags that exist.
## Options
From 0f70d8269c5b4fcbf9a9078e894e8924e6f34b6b Mon Sep 17 00:00:00 2001
From: Alexander Guschin <1aguschin@gmail.com>
Date: Mon, 21 Nov 2022 21:33:48 +0600
Subject: [PATCH 31/38] remove some sections from sidebar
---
content/docs/sidebar.json | 15 ---------------
1 file changed, 15 deletions(-)
diff --git a/content/docs/sidebar.json b/content/docs/sidebar.json
index dd36d9e1..b1b0ba28 100644
--- a/content/docs/sidebar.json
+++ b/content/docs/sidebar.json
@@ -631,11 +631,6 @@
"source": "install.md",
"label": "Installation"
},
- {
- "slug": "why-gto",
- "source": "why-gto.md",
- "label": "Why GTO"
- },
{
"slug": "get-started",
"label": "Get Started",
@@ -646,16 +641,6 @@
"label": "User Guide",
"source": "user-guide.md"
},
- {
- "slug": "dvc",
- "source": "dvc.md",
- "label": "GTO with DVC"
- },
- {
- "slug": "mlem",
- "source": "mlem.md",
- "label": "GTO with MLEM"
- },
{
"slug": "command-reference",
"label": "Command Reference",
From 98a06c4885db6bde35ac525b1eeae118d4217dc8 Mon Sep 17 00:00:00 2001
From: Alexander Guschin <1aguschin@gmail.com>
Date: Mon, 21 Nov 2022 21:47:05 +0600
Subject: [PATCH 32/38] fix dvc link
---
content/docs/gto/user-guide.md | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/content/docs/gto/user-guide.md b/content/docs/gto/user-guide.md
index 941cfe72..eff8a03f 100644
--- a/content/docs/gto/user-guide.md
+++ b/content/docs/gto/user-guide.md
@@ -3,8 +3,8 @@
GTO helps you build a Artifact Registry out of your Git repository. It does so
by creating Git tags of special format and managing `artifacts.yaml` metafile.
Since committing large files to Git is not a good practice, you should consider
-not committing your artifacts to Git. Instead, use [DVC](/doc/gto/dvc), Git-lfs,
-or any method commit pointers to the binary files instead.
+not committing your artifacts to Git. Instead, use [DVC](https://dvc.org/doc),
+Git-lfs, or any method commit pointers to the binary files instead.
## Annotations in artifacts.yaml
From 2fb378ebffe965789f62b20fc3e846b5d302339c Mon Sep 17 00:00:00 2001
From: Alexander Guschin <1aguschin@gmail.com>
Date: Tue, 22 Nov 2022 11:24:45 +0600
Subject: [PATCH 33/38] reset package.json and yarn.lock
---
package.json | 2 +-
yarn.lock | 8 ++++----
2 files changed, 5 insertions(+), 5 deletions(-)
diff --git a/package.json b/package.json
index 912f1c99..2a4cc042 100644
--- a/package.json
+++ b/package.json
@@ -21,7 +21,7 @@
"prepare": "husky install"
},
"dependencies": {
- "@dvcorg/gatsby-theme-iterative": "0.1.24",
+ "@dvcorg/gatsby-theme-iterative": "0.1.23",
"@dvcorg/websites-server": "^0.0.10",
"@octokit/graphql": "5.0.1",
"@sentry/gatsby": "^7.13.0",
diff --git a/yarn.lock b/yarn.lock
index 70f34107..52201adb 100644
--- a/yarn.lock
+++ b/yarn.lock
@@ -1084,10 +1084,10 @@
resolved "https://registry.yarnpkg.com/@csstools/convert-colors/-/convert-colors-1.4.0.tgz#ad495dc41b12e75d588c6db8b9834f08fa131eb7"
integrity sha512-5a6wqoJV/xEdbRNKVo6I4hO3VjyDq//8q2f9I6PBAvMesJHFauXDorcNCsr9RzvsZnaWi5NYCcfyqP1QeFHFbw==
-"@dvcorg/gatsby-theme-iterative@0.1.24":
- version "0.1.24"
- resolved "https://registry.yarnpkg.com/@dvcorg/gatsby-theme-iterative/-/gatsby-theme-iterative-0.1.24.tgz#156264e5cb72e21c5dff1926306ff0121683c79e"
- integrity sha512-NFmiq+tJ7UkrkGY+36KC9XkIqtaJFxiz19RN9Zq14pHA8ZKjSxBG+qqFyR+kxo7KcF9CIJivoSHNQ10YZzhiOg==
+"@dvcorg/gatsby-theme-iterative@0.1.23":
+ version "0.1.23"
+ resolved "https://registry.yarnpkg.com/@dvcorg/gatsby-theme-iterative/-/gatsby-theme-iterative-0.1.23.tgz#8e13533008999fa8398a69525f920557a00bce45"
+ integrity sha512-LrEQvAHL0fPRsHw7Ymi4DNxw7H6CHJnIPQYbGbj1h8PI+IvBoTrnhpLoGtOPJSna/EUILrErWP+i1FSFcP1oAQ==
dependencies:
"@reach/portal" "^0.17.0"
"@reach/router" "^1.3.4"
From d8fec093634ac0fe24dfd0793497116da2a1d992 Mon Sep 17 00:00:00 2001
From: Alexander Guschin <1aguschin@gmail.com>
Date: Tue, 22 Nov 2022 12:38:50 +0600
Subject: [PATCH 34/38] fix some feedback about CI/CD in GS
---
content/docs/gto/get-started.md | 53 +++++++++++++++++++++++----------
1 file changed, 37 insertions(+), 16 deletions(-)
diff --git a/content/docs/gto/get-started.md b/content/docs/gto/get-started.md
index 89bb517c..dc478b65 100644
--- a/content/docs/gto/get-started.md
+++ b/content/docs/gto/get-started.md
@@ -83,7 +83,7 @@ $ gto show
We can push to Git repository to start a CI job!
-## Act on new versions and stage assignments in CI/CD
+## Act on registrations and assignments in CI/CD
GTO [uses Git tags] to register artifact versions and assign them to stages.
This means we can act upon these operations in any Git-based system such as many
@@ -108,8 +108,22 @@ start, you'll need to enable it on the "Actions" page of your fork.
-Let's do the same thing we did locally, but for your remote repo. Don't forget
-to replace the URL:
+Now let's push the Git tags we created locally to your repository (don't forget
+to replace `aguschin` with your username):
+
+```cli
+$ git remote update origin https://github.com/aguschin/example-gto
+$ git push origin cv-class@v0.1.14 cv-class#dev#1
+ * [new tag] cv-class@v0.1.14 -> cv-class@v0.1.14
+ * [new tag] cv-class#dev#1 -> cv-class#dev#1
+```
+
+
+
+### Or just repeat the assignment for your new repo
+
+We can do the same thing we did locally, but for your remote repo (don't forget
+to replace `aguschin` with your username):
```cli
# since we didn't register a version on remote, GTO will do that for us:
@@ -123,27 +137,34 @@ Running `git push origin cv-class#dev#1`
Successfully pushed git tag cv-class#dev#1 on remote.
```
-Now the CI/CD should start, and you should see that we found out: it was
-`cv-class` artifact, version `v0.1.14` that was assigned to `dev` stage. Using
-this information, the step `Deploy (act on assigning a new stage)` was executed
-(while `Publish (act on registering a new version)` was skipped):
-
-
-If you want to see more CI examples, check out
-[the example-repo](https://github.com/iterative/example-gto/actions).
+"GTO: figure out what was registered/promoted" step interprets the Git tag that
+triggered the workflow and passes the information to the next steps. The
+information is used later to trigger "Publish" or "Deploy" steps (since the CI
+on the screenshot was triggered by `cv-class#dev#1` Git tag, the "Publish" step
+was skipped).
-
+If you want to see CI examples without reproducing this Get Started, you can
+check out them in
+[the example-repo](https://github.com/iterative/example-gto/actions).
## Next steps
Thanks for completing this Get Started!
-
From 892c1f7663a1def2cbcf9054874fee1901a4a9f6 Mon Sep 17 00:00:00 2001
From: Alexander Guschin <1aguschin@gmail.com>
Date: Tue, 22 Nov 2022 12:39:18 +0600
Subject: [PATCH 35/38] fix some feedback about CI/CD in GS
---
content/docs/gto/get-started.md | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/content/docs/gto/get-started.md b/content/docs/gto/get-started.md
index dc478b65..8e5580c9 100644
--- a/content/docs/gto/get-started.md
+++ b/content/docs/gto/get-started.md
@@ -154,7 +154,7 @@ If you want to see CI examples without reproducing this Get Started, you can
check out them in
[the example-repo](https://github.com/iterative/example-gto/actions).
-## Next steps
+## What's next?
Thanks for completing this Get Started!
From 2ab8a9b697fc715da6d836e7c3df9e03f3b02bf6 Mon Sep 17 00:00:00 2001
From: Jorge Orpinel
+
-### CI/CD execution example
+Git tags pushed trigger the
+[CI workflow](https://github.com/iterative/example-gto/blob/main/.github/workflows/gto-act-on-tags.yml):
-If you want to see more CI examples, check out
-[the example-repo](https://github.com/iterative/example-gto/actions).
+"GTO: figure out what was registered/promoted" step interprets the Git tag that
+triggered the workflow and passes the information to the next steps. The
+information is used later to trigger "Publish" or "Deploy" steps (since the CI
+on the screenshot was triggered by `cv-class#dev#1` Git tag, the "Publish" step
+was skipped).
-
-### Click to set up a working repository to try this.
+### Click to set up a Git remote you can push to.
+
+
+
+You'll need a [GitHub account](https://github.com/signup)) for this.
+
+
+
+1. [Fork the example repo]. Make sure you uncheck "Copy the `main` branch only"
+ to preserve the repo's tags.
-Let's fork the [example-gto repo](https://github.com/iterative/example-gto) repo
-(you'll need a [GitHub](https://github.com/signup) account first). For CI/CD to
-start, you'll need to enable it on the "Actions" page of your fork.
+2. Enable the [workflows] in your fork's **Settings** -> **Actions** page. Now
+ its [preconfigured jobs] will trigger when Git tags are pushed.
-1. [Fork the repo](https://github.com/iterative/example-gto/fork). Make sure you
- uncheck "Copy the `main` branch only" to copy Git tags as well:
-
+[fork the example repo]: https://github.com/iterative/example-gto/fork
+[workflows]: https://docs.github.com/en/actions/using-workflows/about-workflows
+[preconfigured jobs]:
+ https://github.com/iterative/example-gto/blob/main/.github/workflows/gto-act-on-tags.yml
-2. Enable workflows in your repo for a new Git tag to trigger CI:
-
+3. Update your local repo's default remote (`origin`) with your fork (replace
+ `myuser` with your GitHub username):
+
+ ```cli
+ $ git remote update origin https://github.com/myuser/example-gto
+ ```
-Now let's push the Git tags we created locally to your repository (don't forget
-to replace `aguschin` with your username):
+To trigger your CI/CD workflows, you can push any of the Git tags created with
+GTO, for example the [latest model version](#registering-a-new-version):
```cli
-$ git remote update origin https://github.com/aguschin/example-gto
-$ git push origin cv-class@v0.1.14 cv-class#dev#1
- * [new tag] cv-class@v0.1.14 -> cv-class@v0.1.14
- * [new tag] cv-class#dev#1 -> cv-class#dev#1
+$ git push origin cv-class@v0.1.14
+* [new tag] cv-class@v0.1.14 -> cv-class@v0.1.14
```
-
+[fork the example repo]: https://github.com/iterative/example-gto/fork
+[workflows]: https://docs.github.com/en/actions/using-workflows/about-workflows
+[preconfigured jobs]:
+ https://github.com/iterative/example-gto/blob/main/.github/workflows/gto-act-on-tags.yml
-2. Enable workflows in your repo for a new Git tag to trigger CI:
-
+3. Update your local repo's default remote (`origin`) with your fork (replace
+ `myuser` with your GitHub username):
+
+ ```cli
+ $ git remote update origin https://github.com/myuser/example-gto
+ ```
-
-### Or just repeat the assignment for your new repo
-
-We can do the same thing we did locally, but for your remote repo (don't forget
-to replace `aguschin` with your username):
+Alternatively, GTO operations can target another `--repo` directly. Let's try
+the [stage assignment](#assigning-stages) again for example, but on your
+remote:
```cli
-# since we didn't register a version on remote, GTO will do that for us:
+# Replace myuser with your GitHub user below.
$ gto assign cv-class --stage dev \
- --repo https://github.com/aguschin/example-gto
-Created git tag 'cv-class@v0.1.14' that registers a version
-Running `git push origin cv-class@v0.1.14`
-Successfully pushed git tag cv-class@v0.1.14 on remote.
+ --repo https://github.com/myuser/example-gto
Created git tag 'cv-class#dev#1' that assigns stage to version 'v0.1.14'
Running `git push origin cv-class#dev#1`
Successfully pushed git tag cv-class#dev#1 on remote.
```
-
+Note that the tag is created locally first (if not present) and then pushed to
+the target repo.
-Git tags pushed trigger the
-[CI workflow](https://github.com/iterative/example-gto/blob/main/.github/workflows/gto-act-on-tags.yml):
+
+To see what the example repo's [CI/CD jobs] look like, see its [GitHub Actions] page.
-"GTO: figure out what was registered/promoted" step interprets the Git tag that
-triggered the workflow and passes the information to the next steps. The
-information is used later to trigger "Publish" or "Deploy" steps (since the CI
-on the screenshot was triggered by `cv-class#dev#1` Git tag, the "Publish" step
-was skipped).
+[ci/cd jobs]:
+ https://github.com/iterative/example-gto/blob/main/.github/workflows/gto-act-on-tags.yml
+[github actions]: https://github.com/iterative/example-gto/actions
-If you want to see CI examples without reproducing this Get Started, you can
-check out them in
-[the example-repo](https://github.com/iterative/example-gto/actions).
+