Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] Gatsby migration #1025

Merged
merged 46 commits into from
Mar 17, 2020
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
46 commits
Select commit Hold shift + click to select a range
1750b88
Move static pages to gatsby
iAdramelk Feb 26, 2020
18ba74a
In-progress docs page migration
iAdramelk Feb 28, 2020
2f08a9a
Add syntax highlingting and dvc-syntax
iAdramelk Mar 2, 2020
50cb710
Add external link marker
iAdramelk Mar 2, 2020
fc4fe51
Rewrite LocalLink without deprecated methods
iAdramelk Mar 2, 2020
13b7445
Fix non-working links in sidebar
iAdramelk Mar 2, 2020
0105f47
Move remark-linker to backend
iAdramelk Mar 2, 2020
8234008
Make syntax highlight work
iAdramelk Mar 3, 2020
ebb91fa
Ignore testes in .cache in jest
iAdramelk Mar 3, 2020
9ed2054
Fix lint errors
iAdramelk Mar 3, 2020
a2153b6
Make custom react ccomponents work in markdown
iAdramelk Mar 5, 2020
eede61f
Remove old markdown component
iAdramelk Mar 6, 2020
337c0ee
Fix button colors
iAdramelk Mar 6, 2020
e846001
Fix paths in learn block
iAdramelk Mar 6, 2020
ba1b3f7
Add middleware for API
iAdramelk Mar 6, 2020
2109582
Add middleware for redirects
iAdramelk Mar 6, 2020
5c8a420
Add simple Express server with middlewares
iAdramelk Mar 6, 2020
d496c3e
Update path to docs in edit link
iAdramelk Mar 10, 2020
847b997
Add title to the doc pages
iAdramelk Mar 10, 2020
95d2289
Rename middleware folder
iAdramelk Mar 10, 2020
8efa638
Remove unneeded comment
iAdramelk Mar 10, 2020
68459d2
Removed unneded prop
iAdramelk Mar 10, 2020
d662217
Add public for jest ignore path
iAdramelk Mar 10, 2020
5a6f23c
Add res.end to redirects
iAdramelk Mar 10, 2020
e65b415
Add env to jest config
iAdramelk Mar 10, 2020
46beb32
Uncomment manifest and favicons generation
iAdramelk Mar 13, 2020
29e957d
Add redirects for non-existent pages
iAdramelk Mar 13, 2020
2756e82
Remove react-syntax-highlighter from package.json
iAdramelk Mar 13, 2020
6fb5fa8
Fix typo
iAdramelk Mar 13, 2020
24caaf3
Add new commants to dvc commands list
iAdramelk Mar 13, 2020
298e4d6
Add newline to .gitignore
iAdramelk Mar 13, 2020
a20558b
Add return to redirects middware to prevent calling next()
iAdramelk Mar 13, 2020
ce55f58
Fix next links
iAdramelk Mar 13, 2020
4eb952c
Add 307 status for docs’ redirects
iAdramelk Mar 13, 2020
6571197
Add dvc highlingter to snippet
iAdramelk Mar 16, 2020
abc5cfa
Add 404 page
psdcoder Mar 16, 2020
0b68d2c
Fix visibility calculation for community's sections on mobiles
psdcoder Mar 16, 2020
f7845fb
Add handler for anchor navigation
psdcoder Mar 16, 2020
71209eb
Fix hover issues for community dropdown in the main navigation
psdcoder Mar 16, 2020
19ed779
Fix path for SEO component in the 404 page
psdcoder Mar 16, 2020
36230b6
Fix paths list for 'lint-check' package script
psdcoder Mar 16, 2020
d03beca
Add missing api-reference docs
psdcoder Mar 17, 2020
d22140b
Make two layouts: one main and one for docs. Use these layouts in Ga…
psdcoder Mar 17, 2020
17ce028
Handle trailing slash in the doc's sidebar menu
psdcoder Mar 17, 2020
2835251
Fix incorrect layout for 404 page
psdcoder Mar 17, 2020
f89b450
Merge branch 'master' into gatsby-migration
shcheklein Mar 17, 2020
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
4 changes: 0 additions & 4 deletions .babelrc

This file was deleted.

5 changes: 4 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -56,8 +56,11 @@ typings/

# Next.js build output
.idea
.next/
*.log

# Mac finder artifacts
.DS_Store

# Gatsby cache
.cache
public
2 changes: 1 addition & 1 deletion .restyled.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -4,5 +4,5 @@ restylers:
include:
- './*.{js,md}'
- 'pages/**/*.js'
- 'public/static/docs/**/*.{js,md}'
- 'content/docs/**/*.{js,md}'
- 'src/**/*.js'
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
![owl logo](https://dvc.org/static/img/logo-github-readme.png)
![owl logo](https://dvc.org/img/logo-github-readme.png)
shcheklein marked this conversation as resolved.
Show resolved Hide resolved

[![Maintainability](https://api.codeclimate.com/v1/badges/5872e0a572ec8b74bd8d/maintainability)](https://codeclimate.com/github/iterative/dvc.org/maintainability)
[![CircleCI](https://circleci.com/gh/iterative/dvc.org.svg?style=svg)](https://circleci.com/gh/iterative/dvc.org)
Expand Down
52 changes: 52 additions & 0 deletions config/prismjs/dvc-commands.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
/* eslint-env node */

module.exports = [
'version',
'update',
'unprotect',
'unlock',
'tag',
'status',
'run',
'root',
'repro',
'remove',
'remote remove',
'remote modify',
'remote list',
'remote default',
'remote add',
'remote',
'push',
'pull',
'pkg',
'pipeline show',
'pipeline list',
'pipeline',
'move',
'metrics show',
shcheklein marked this conversation as resolved.
Show resolved Hide resolved
'metrics remove',
'metrics modify',
'metrics diff',
'metrics add',
'metrics',
'lock',
'list',
'install',
'init',
'import-url',
shcheklein marked this conversation as resolved.
Show resolved Hide resolved
'import',
'help',
'get-url',
'get',
'gc',
'fetch',
'diff',
'destroy',
'config',
'commit',
'checkout',
'cache dir',
'cache',
'add'
]
18 changes: 18 additions & 0 deletions config/prismjs/dvc-hook.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
/* eslint-env node */

const Prism = require('prismjs')

// Make sure the $ part of the command prompt in shell
// examples isn't copiable by making it an 'input' token.
Prism.hooks.add('after-tokenize', env => {
if (env.language !== 'dvc') {
return
}

for (const token of env.tokens) {
if (token.type === 'line' && /^\$\s+$/.test(token.content[0])) {
const old = token.content[0]
token.content[0] = new Prism.Token('input', old, null, old, false)
}
}
})
66 changes: 66 additions & 0 deletions config/prismjs/dvc.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
/* eslint-env node */

// we require prism and load its bash module so we have
// Prism.languages.bash to embed into our DVC language.

const Prism = require('prismjs')
require('prismjs/components/prism-bash')
require('./dvc-hook')
const { bash } = Prism.languages

const dvc = require('./dvc-commands')

// Command arrays are intentionally reverse sorted
// to prevent shorter matches before longer ones

const git = [
'tag',
'status',
'remote update',
'remote rename',
'remote remove',
'remote add',
'remote',
'push',
'pull',
'merge',
'init',
'fetch',
'commit',
'clone',
'checkout',
'add'
]

const beforeCommand = String.raw`(\$[\s(]+|;\s*)`

/* tslint:disable object-literal-sort-keys */

Prism.languages.dvc = {
line: {
pattern: /(?<=(^|\n))\$[\s\S]*?[^\\](:?\n|$)/,
inside: {
dvc: {
pattern: new RegExp(
String.raw`${beforeCommand}\b(?:dvc (?:${dvc.join('|')}))\b`
),
greedy: true,
lookbehind: true
},
git: {
pattern: new RegExp(
String.raw`${beforeCommand}\b(?:git (?:${git.join('|')}))\b`
),
greedy: true,
lookbehind: true
},
command: {
pattern: new RegExp(String.raw`${beforeCommand}\b[a-zA-Z0-9\-_]+\b`),
greedy: true,
lookbehind: true
},
...bash
}
},
comment: bash.comment
}
13 changes: 13 additions & 0 deletions config/prismjs/usage.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
/* eslint-env node */

const dvc = require('./dvc-commands')
const Prism = require('prismjs')

Prism.languages.usage = {
dvc: {
pattern: new RegExp(`dvc (?:${dvc.join('|')})`)
},
usage: {
pattern: /(^|\n)\s*(usage|positional arguments|optional arguments)/
}
}
110 changes: 110 additions & 0 deletions content/docs/api-reference/get_url.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,110 @@
# dvc.api.get_url()

Returns the URL to the storage location of a data file or directory tracked in a
<abbr>DVC project</abbr>.

```py
def get_url(path: str,
repo: str = None,
rev: str = None,
remote: str = None) -> str
```

#### Usage:

```py
import dvc.api

resource_url = dvc.api.get_url(
'get-started/data.xml',
repo='https://github.com/iterative/dataset-registry')

# resource_url is now "https://remote.dvc.org/dataset-registry/a3/04afb96060aad90176268345e10355"
```

## Description

Returns the URL string of the storage location (in a
[DVC remote](/doc/command-reference/remote)) where a target file or directory,
specified by its `path` in a `repo` (<abbr>DVC project</abbr>), is stored.

The URL is formed by reading the project's
[remote configuration](/doc/command-reference/config#remote) and the
[DVC-file](/doc/user-guide/dvc-file-format) where the given `path` is an
<abbr>output</abbr>. The URL schema returned depends on the
[type](/doc/command-reference/remote/add#supported-storage-types) of the
`remote` used (see the [Parameters](#parameters) section).

If the target is a directory, the returned URL will end in `.dir`. Refer to
[Structure of cache directory](/doc/user-guide/dvc-files-and-directories#structure-of-cache-directory)
and `dvc add` to learn more about how DVC handles data directories.

⚠️ This function does not check for the actual existence of the file or
directory in the remote storage.

💡 Having the resource's URL, it should be possible to download it directly with
an appropriate library, such as
[`boto3`](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html#S3.Object.download_fileobj)
or
[`paramiko`](https://docs.paramiko.org/en/stable/api/sftp.html#paramiko.sftp_client.SFTPClient.get).

## Parameters

- **`path`** - location and file name of the file or directory in `repo`,
relative to the project's root.

- `repo` - specifies the location of the DVC project. It can be a URL or a file
system path. Both HTTP and SSH protocols are supported for online Git repos
(e.g. `[user@]server:project.git`). _Default_: The current project is used
(the current working directory tree is walked up to find it).

- `rev` - Git commit (any [revision](https://git-scm.com/docs/revisions) such as
a branch or tag name, or a commit hash). If `repo` is not a Git repo, this
option is ignored. _Default_: `HEAD`.

- `remote` - name of the [DVC remote](/doc/command-reference/remote) to use to
form the returned URL string. _Default_: The
[default remote](/doc/command-reference/remote/default) of `repo` is used.

## Exceptions

- `dvc.api.UrlNotDvcRepoError` - `repo` is not a DVC project.

- `dvc.exceptions.NoRemoteError` - no `remote` is found.

## Example: Getting the URL to a DVC-tracked file

```py
import dvc.api

resource_url = dvc.api.get_url(
'get-started/data.xml',
repo='https://github.com/iterative/dataset-registry'
)

print(resource_url)
```

The script above prints

`https://remote.dvc.org/dataset-registry/a3/04afb96060aad90176268345e10355`

This URL represents the location where the data is stored, and is built by
reading the corresponding DVC-file
([`get-started/data.xml.dvc`](https://github.com/iterative/dataset-registry/blob/master/get-started/data.xml.dvc))
where the `md5` file hash is stored,

```yaml
outs:
- md5: a304afb96060aad90176268345e10355
path: get-started/data.xml
```

and the project configuration
([`.dvc/config`](https://github.com/iterative/dataset-registry/blob/master/.dvc/config))
where the remote URL is saved:

```ini
['remote "storage"']
url = https://remote.dvc.org/dataset-registry
```
16 changes: 16 additions & 0 deletions content/docs/api-reference/index.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
# Python API

DVC can be used as a Python library, simply [install](/doc/install) with `pip`
or `conda`. This reference provides the details about the functions in the API
module `dvc.api`, which can be imported any regular way, for example:

```py
import dvc.api
```

The purpose of this API is to provide programatic access to the data or models
[stored and versioned](/doc/use-cases/versioning-data-and-model-files) in
<abbr>DVC repositories</abbr> from Python code.

Please choose a function from the navigation sidebar to the left, or click the
`Next` button below to jump into the first one ↘
Loading