Skip to content

Commit

Permalink
Workflow update (NVIDIA#538)
Browse files Browse the repository at this point in the history 
* docs: update workflow

Signed-off-by: Kai Xu <[email protected]>

* docs: update rendered workflow

Signed-off-by: Kai Xu <[email protected]>

---------

Signed-off-by: Kai Xu <[email protected]>
  • Loading branch information
@xukai92
xukai92 authored Mar 12, 2024
1 parent db4d0eb commit efa83ba
Show file tree
Hide file tree
Showing 3 changed files with 83 additions and 72 deletions.
5 changes: 4 additions & 1 deletion docs/README.md
View file
Original file line number Diff line number Diff line change
@@ -1,7 +1,10 @@
Workflow figure is generated using [PlantUML](https://plantuml.com/ditaa) with the [ditaa](https://ditaa.sourceforge.net).
To generate it yourself, the easist way is to install the [PlantUML plugin in VS Code](https://marketplace.visualstudio.com/items?itemName=jebbs.plantuml) (with the pre-requiste installed), open the file and click preview.
You might need to use the following settings to make the preview work:

If you don't want to install the dependencies locally, you can use the following settings to make the preview work with a remote render:
```json
"plantuml.render": "PlantUMLServer",
"plantuml.server": "https://www.plantuml.com/plantuml",
```

[ASCIIFlow](https://asciiflow.com/#/) is a helpful tool to edit the source code.
Binary file modified docs/workflow.png
View file
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
150 changes: 79 additions & 71 deletions docs/workflow.puml
View file
Original file line number Diff line number Diff line change
@@ -1,73 +1,81 @@
@startditaa
+--------+
|lab init|-=-taxonomy repo cloned legend
+---+----+ +-------+
| |lab CLI|
v +-------+
+---+--------+ +-----------+
|lab download|-=-model weights downloaded |manual cPNK|
+---+--------+ +-----------+
| +----------+
v optional |colab cYEL|
+---+-----+ +----------+
|lab serve|-=-start model server
+---+-----+
|
v
+---+----+
|lab chat|-=-chat with the model
+---+----+
|
v
+---+----+
|lab list|-=-view the taxonomy
+---+----+
|
v
+---+-----------------------+
|contributor creates/updates|
|examples in taxonomy cPNK|<----------+
+---+-----------------------+ |
| |
v |
+---+--------+ |
|lab generate|--------------------------+ -=-synthesize data;
+---+-------++other hardware | iterate by modifying examples
|M chips| +-------+--------+ | if samples do not look good
v +->+colab train cYEL| |
+---+-----+ +-------+--------+ |
|lab train|-=-w/ MLX | |
+---+-----+ v |
| +-------+-------+ |
v |colab test cYEL+-+ |
+---+----+ +-------+-------+ | |
|lab test|-------------|---------+------+ -=-run unit/smoke tests;
+---+----+ v | iterate if some test failed
| +-------+----------+ |
v optional |download new model+ |
+---+-------+ +-------+---------++ |
|lab convert|-=-MLX to GGUF | |
+---+-------+ +----------------+ |
| | |
v optional v |
+---+-----------+---+ |
|kill lab serve cPNK|-=-kill old server |
+---+---------------+ |
| |
v optional |
+---+-----+ |
|lab serve|-=-start server with |
+---+-----+ new model |
| |
v optional |
+---+----+ |
|lab chat|------------------------------+ -=-chat with the model again;
+---+----+ iterate if not happy with outputs
|
v
+---+---------------------------+
|contributor uses Git to commit |
|changes to the taxonomy repo |
|and create a PR cPNK|
+---+---------------------------+
+--------+
|lab init|-=-taxonomy repo cloned legend
+---+----+ +-------+
| |lab CLI|
v +-------+
+---+--------+ +-----------+
|lab download|-=-model weights downloaded |manual cPNK|
+---+--------+ +-----------+
| +-------------+
v optional |notebook cYEL|
+---+-----+ +-------------+
|lab serve|-=-start model server
+---+-----+
|
v
+---+----+
|lab chat|-=-chat with the model
+---+----+
|
v
+---+----+
|lab list|-=-view the taxonomy
+---+----+
|
v
+---+-----------------------+
|contributor creates/updates|
|examples in taxonomy cPNK|<----------+
+---+-----------------------+ |
| |
v |
level 1 +---+-----+ |
only validated +---|lab check|-=-validate the YAML file |
the YAML file | +---+-----+ |
| | |
| v |
level 2 | +---+--------+ |
inspected the +---|lab generate|--------------------------+ -=-synthesize data;
generated samples | +---+-------++other hardware | iterate by modifying examples
| |mac or | +---------+---------+ | if samples do not look good
| vLinux +-->|notebook train cYEL| |
| +---+-----+ +---------+---------+ |
| |lab train|-=-w/ MLX or | |
| +---+-----+ PyTorch v |
| | +---------+--------+ |
| v |notebook test cYEL+--+ |
level 3 | +---+----+ +---------+--------+ | |
tested generated +---|lab test|----------------|-----------+-+ -=-run unit/smoke tests;
samples via LoRA | +---+----+ v | iterate if some test failed
| | +---------+--------+ |
| v optional |download new model| |
| +---+-------+ +-----------------++ |
| |lab convert|-=-MLX to GGUF | |
| +---+-------+ +-----------------+ |
| | | |
| v optional v |
| +---+-----------+---+ |
| |kill lab serve cPNK|-=-kill old server |
| +---+---------------+ |
| | |
| v optional |
| +---+-----+ |
| |lab serve|-=-start server with |
| +---+-----+ new model |
| | |
| v optional |
level 4 | +---+----+ |
performed the full | |lab chat|------------------------------+ -=-chat with the model again;
local smoke test | +---+----+ | iterate if not happy with outputs
| | |
| v |
| +---+-------------------------------+ |
| |contributor commits changes and | |
+-->|creates/updates their PR in the | |
|taxonomy repo cPNK| |
+-----------------+-----------------+ |
| PR reviewed by taxonomy reviwers;
v iterate if changes are requested

@endditaa

0 comments on commit efa83ba

Please sign in to comment.