Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement job run command #652

Merged
merged 25 commits into from
Apr 22, 2019
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.D/652.feature
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Implement *job run* command.
92 changes: 90 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@
* [Preface](#Preface)
* [neuro](#neuro)
* [neuro job](#neuro-job)
* [neuro job run](#neuro-job-run)
* [neuro job submit](#neuro-job-submit)
* [neuro job ls](#neuro-job-ls)
* [neuro job status](#neuro-job-status)
Expand Down Expand Up @@ -33,6 +34,7 @@
* [neuro completion generate](#neuro-completion-generate)
* [neuro completion patch](#neuro-completion-patch)
* [neuro help](#neuro-help)
* [neuro run](#neuro-run)
* [neuro submit](#neuro-submit)
* [neuro ps](#neuro-ps)
* [neuro status](#neuro-status)
Expand Down Expand Up @@ -102,6 +104,7 @@ Name | Description|
|Usage|Description|
|---|---|
| _[neuro help](#neuro-help)_| Get help on a command |
| _[neuro run](#neuro-run)_| Run an image with predefined configuration |
| _[neuro submit](#neuro-submit)_| Submit an image to run on the cluster |
| _[neuro ps](#neuro-ps)_| List all jobs |
| _[neuro status](#neuro-status)_| Display status of a job |
Expand Down Expand Up @@ -146,6 +149,7 @@ Name | Description|

|Usage|Description|
|---|---|
| _[neuro job run](#neuro-job-run)_| Run an image with predefined configuration |
| _[neuro job submit](#neuro-job-submit)_| Submit an image to run on the cluster |
| _[neuro job ls](#neuro-job-ls)_| List all jobs |
| _[neuro job status](#neuro-job-status)_| Display status of a job |
Expand All @@ -158,6 +162,48 @@ Name | Description|



### neuro job run

Run an image with predefined configuration.<br/><br/>IMAGE container image name.<br/><br/>CMD list will be passed as commands to model container.<br/>

**Usage:**

```bash
neuro job run [OPTIONS] IMAGE [CMD]...
```

**Examples:**

```bash

# Starts a container pytorch:latest with two paths mounted.
# Directory storage://<USERNAME> is mounted as /var/storage/home in read-write mode,
dalazx marked this conversation as resolved.
Show resolved Hide resolved
# storage://neuromation is mounted as :/var/storage/neuromation as read-only.
neuro run pytorch:latest --volume=HOME

```

**Options:**

Name | Description|
|----|------------|
|_\-s, --preset PRESET_|Predefined job profile \[default: gpu-small]|
|_\-x, --extshm / -X, --no-extshm_|Request extended '/dev/shm' space \[default: True]|
|_--http INTEGER_|Enable HTTP port forwarding to container \[default: 80]|
|_\--http-auth / --no-http-auth_|Enable HTTP authentication for forwarded HTTP port \[default: True]|
|_\-p, --preemptible / -P, --non-preemptible_|Run job on a lower-cost preemptible instance \[default: True]|
|_\-n, --name NAME_|Optional job name|
|_\-d, --description DESC_|Add optional description in free format|
|_\-q, --quiet_|Run command in quiet mode \(print only job id)|
|_\-v, --volume MOUNT_|Mounts directory from vault into container. Use multiple options to mount more than one volume. --volume=HOME is an alias for storage://~:/var/storage/home:rw and storage://neuromation:/var/storage/neuromation:ro|
|_\-e, --env VAR=VAL_|Set environment variable in container Use multiple options to define more than one variable|
|_\--env-file PATH_|File with environment variables to pass|
|_\--wait-start / --no-wait-start_|Wait for a job start or failure \[default: True]|
|_--help_|Show this message and exit.|




### neuro job submit

Submit an image to run on the cluster.<br/><br/>IMAGE container image name.<br/><br/>CMD list will be passed as commands to model container.<br/>
Expand Down Expand Up @@ -194,7 +240,7 @@ Name | Description|
|_\-n, --name NAME_|Optional job name|
|_\-d, --description DESC_|Optional job description in free format|
|_\-q, --quiet_|Run command in quiet mode \(print only job id)|
|_\-v, --volume MOUNT_|Mounts directory from vault into container. Use multiple options to mount more than one volume|
|_\-v, --volume MOUNT_|Mounts directory from vault into container. Use multiple options to mount more than one volume. --volume=HOME is an alias for storage://~:/var/storage/home:rw and storage://neuromation:/var/storage/neuromation:ro|
|_\-e, --env VAR=VAL_|Set environment variable in container Use multiple options to define more than one variable|
|_\--env-file PATH_|File with environment variables to pass|
|_\--wait-start / --no-wait-start_|Wait for a job start or failure \[default: True]|
Expand Down Expand Up @@ -859,6 +905,48 @@ Name | Description|



## neuro run

Run an image with predefined configuration.<br/><br/>IMAGE container image name.<br/><br/>CMD list will be passed as commands to model container.<br/>

**Usage:**

```bash
neuro run [OPTIONS] IMAGE [CMD]...
```

**Examples:**

```bash

# Starts a container pytorch:latest with two paths mounted.
# Directory storage://<USERNAME> is mounted as /var/storage/home in read-write mode,
# storage://neuromation is mounted as :/var/storage/neuromation as read-only.
neuro run pytorch:latest --volume=HOME

```

**Options:**

Name | Description|
|----|------------|
|_\-s, --preset PRESET_|Predefined job profile \[default: gpu-small]|
|_\-x, --extshm / -X, --no-extshm_|Request extended '/dev/shm' space \[default: True]|
|_--http INTEGER_|Enable HTTP port forwarding to container \[default: 80]|
|_\--http-auth / --no-http-auth_|Enable HTTP authentication for forwarded HTTP port \[default: True]|
|_\-p, --preemptible / -P, --non-preemptible_|Run job on a lower-cost preemptible instance \[default: True]|
|_\-n, --name NAME_|Optional job name|
|_\-d, --description DESC_|Add optional description in free format|
|_\-q, --quiet_|Run command in quiet mode \(print only job id)|
|_\-v, --volume MOUNT_|Mounts directory from vault into container. Use multiple options to mount more than one volume. --volume=HOME is an alias for storage://~:/var/storage/home:rw and storage://neuromation:/var/storage/neuromation:ro|
|_\-e, --env VAR=VAL_|Set environment variable in container Use multiple options to define more than one variable|
|_\--env-file PATH_|File with environment variables to pass|
|_\--wait-start / --no-wait-start_|Wait for a job start or failure \[default: True]|
|_--help_|Show this message and exit.|




## neuro submit

Submit an image to run on the cluster.<br/><br/>IMAGE container image name.<br/><br/>CMD list will be passed as commands to model container.<br/>
Expand Down Expand Up @@ -895,7 +983,7 @@ Name | Description|
|_\-n, --name NAME_|Optional job name|
|_\-d, --description DESC_|Optional job description in free format|
|_\-q, --quiet_|Run command in quiet mode \(print only job id)|
|_\-v, --volume MOUNT_|Mounts directory from vault into container. Use multiple options to mount more than one volume|
|_\-v, --volume MOUNT_|Mounts directory from vault into container. Use multiple options to mount more than one volume. --volume=HOME is an alias for storage://~:/var/storage/home:rw and storage://neuromation:/var/storage/neuromation:ro|
|_\-e, --env VAR=VAL_|Set environment variable in container Use multiple options to define more than one variable|
|_\--env-file PATH_|File with environment variables to pass|
|_\--wait-start / --no-wait-start_|Wait for a job start or failure \[default: True]|
Expand Down
8 changes: 0 additions & 8 deletions neuromation/api/jobs.py
Original file line number Diff line number Diff line change
Expand Up @@ -152,14 +152,6 @@ def from_cli(cls, username: str, volume: str) -> "Volume":
read_only=read_only,
)

@classmethod
def from_cli_list(
cls, username: str, lst: Sequence[str]
) -> Optional[List["Volume"]]:
if not lst:
return None
return [cls.from_cli(username, s) for s in lst]


@dataclass(frozen=True)
class HTTPPort:
Expand Down
Loading