Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

guide: Basic Operations (Data Mgmt) #4053

Closed
wants to merge 39 commits into from
Closed
Changes from 1 commit
Commits
Show all changes
39 commits
Select commit Hold shift + click to select a range
3df841e
guide: add DM/ Basic Ops intro + struct
jorgeorpinel Oct 19, 2022
6e5450e
guide: Tracking data guide and
jorgeorpinel Oct 19, 2022
9a8ce22
guide: Traking updates, intros for Sync and Version (Data Mgmt)
jorgeorpinel Oct 19, 2022
afb38ee
guide: complete Sync inc. figure (Data Mgmt)
jorgeorpinel Oct 19, 2022
9fc77a9
Merge branch 'main' into guide/data-mgmt/basic-ops
jorgeorpinel Oct 20, 2022
314c953
guide" fixes and import/update flow to DM/Sync
jorgeorpinel Oct 20, 2022
a86325e
guide: More Data Versioning info. and
jorgeorpinel Oct 20, 2022
bc07653
guide: complete Versioning info (DM/ Basic Ops)
jorgeorpinel Oct 20, 2022
ac0a555
Merge branch 'guide/data-mgmt-flows' into guide/data-mgmt/basic-ops
jorgeorpinel Oct 20, 2022
baf97cc
guide: typo fix
jorgeorpinel Oct 20, 2022
766f329
Merge branch 'guide/data-mgmt-flows' into guide/data-mgmt/basic-ops
jorgeorpinel Oct 20, 2022
fe3208c
ref: simplify remote index (move to guide)
jorgeorpinel Oct 21, 2022
83834cd
ref: link from remote index to DM/ Ops/ Sync and
jorgeorpinel Oct 21, 2022
a45c9d6
Merge branch 'guide/data-mgmt-flows' into guide/data-mgmt/basic-ops
jorgeorpinel Oct 22, 2022
561ff06
Merge branch 'guide/data-mgmt-flows' into guide/data-mgmt/basic-ops
jorgeorpinel Oct 22, 2022
39e2965
Merge branch 'guide/data-mgmt-flows' into guide/data-mgmt/basic-ops
jorgeorpinel Oct 22, 2022
e77f5f1
Merge branch 'guide/data-mgmt-flows' into guide/data-mgmt/basic-ops
jorgeorpinel Oct 22, 2022
c1ed918
guide: rename DM/ TSV -> TSVD and
jorgeorpinel Oct 22, 2022
69f7de5
Merge branch 'guide/data-mgmt-flows' into guide/data-mgmt/basic-ops
jorgeorpinel Oct 22, 2022
ec7f2c1
guide: DM/ TSVD - data codification
jorgeorpinel Oct 22, 2022
032cfef
guide: remove comment
jorgeorpinel Oct 22, 2022
903632a
Merge branch 'guide/data-mgmt-flows' into guide/data-mgmt/basic-ops
jorgeorpinel Oct 22, 2022
5335be2
Merge branch 'guide/data-mgmt-flows' into guide/data-mgmt/basic-ops
jorgeorpinel Oct 22, 2022
1430354
Merge branch 'guide/data-mgmt-flows' into guide/data-mgmt/basic-ops
jorgeorpinel Oct 24, 2022
af6fec3
ref: roll back changes to remote index which
jorgeorpinel Oct 24, 2022
664f68a
Merge branch 'guide/data-mgmt-flows' into guide/data-mgmt/basic-ops
jorgeorpinel Oct 27, 2022
13ba7ae
guide: don't call `checkout` "plumbing"
jorgeorpinel Oct 27, 2022
dba203e
guide: don't call remote storage "additional"
jorgeorpinel Oct 27, 2022
e6a9ebf
Merge branch 'guide/data-mgmt-flows' into guide/data-mgmt/basic-ops
jorgeorpinel Feb 18, 2023
2916cf3
nav: remove unexistent page (per previous merge)
jorgeorpinel Feb 18, 2023
e2fdc09
typo
jorgeorpinel Feb 18, 2023
6115f72
guide: Basic Ops/ Tracking Data
jorgeorpinel Feb 18, 2023
315cf47
Remove unrelated changes...
jorgeorpinel Feb 18, 2023
272bc07
guide: Data Mgmt/ Basic Ops/ Sync
jorgeorpinel Feb 18, 2023
8fa58bc
fix link
jorgeorpinel Feb 18, 2023
d0ac5b9
guide: Data Mgmt/ Versioning + mention ML models more
jorgeorpinel Feb 20, 2023
5397f46
guide: link from Remote Storage to Basic Ops/ Sync
jorgeorpinel Feb 20, 2023
2c38ee4
Merge branch 'guide/data-mgmt-flows' into guide/data-mgmt/basic-ops
jorgeorpinel Feb 21, 2023
4a11e04
guide: more links to cache/remote sync
jorgeorpinel Feb 21, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
Prev Previous commit
Next Next commit
Merge branch 'guide/data-mgmt-flows' into guide/data-mgmt/basic-ops
jorgeorpinel committed Feb 18, 2023

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
commit e6a9ebf4bf82d3138ffc29a66f169916827592ab
81 changes: 0 additions & 81 deletions .circleci/config.yml

This file was deleted.

22 changes: 9 additions & 13 deletions .eslintrc.json
Original file line number Diff line number Diff line change
@@ -2,10 +2,10 @@
"extends": [
"plugin:react/recommended",
"plugin:@typescript-eslint/recommended",
"prettier",
"plugin:prettier/recommended",
"plugin:jsx-a11y/recommended",
"plugin:json/recommended"
"plugin:json/recommended",
"prettier"
],
"plugins": ["@typescript-eslint"],
"parser": "@typescript-eslint/parser",
@@ -37,22 +37,18 @@
"leadingUnderscore": "allow"
}
],
"no-tabs": "error",
"no-tabs": [
"error",
{
"allowIndentationTabs": true
}
],
"react/prop-types": "off",
"@typescript-eslint/no-unused-vars": "error"
},
"overrides": [
{
"files": [
"src/utils/shared/*.?(js|ts)",
"src/gatsby/**/*.js",
"scripts/**/*.js",
"config/**/*.js",
"src/server/**/*.js",
"plugins/**/*.js",
"gatsby-*.js",
"postcss.config.js"
],
"files": "**/*.js",
"rules": {
"@typescript-eslint/no-var-requires": "off",
"@typescript-eslint/explicit-function-return-type": "off"
16 changes: 16 additions & 0 deletions .github/workflows/check-deployment.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
name: Check Deployment
on:
workflow_dispatch:
inputs:
passci:
description: 'Pass CI'
required: true
type: boolean
pull_request:
types: [opened, synchronize, reopened, labeled]

jobs:
check-deployment:
uses: iterative/gatsby-theme-iterative/.github/workflows/check-deployment.yml@main
with:
passci: ${{ inputs.passci || false }}
2 changes: 1 addition & 1 deletion .github/workflows/download-link-check-deploy.yml
Original file line number Diff line number Diff line change
@@ -13,7 +13,7 @@ jobs:
steps:
- uses: actions/checkout@v3
- name: Download Link Checker
uses: lycheeverse/[email protected].1
uses: lycheeverse/[email protected].4
with:
args:
--verbose "${{ github.event.deployment.payload.web_url }}" --base
4 changes: 2 additions & 2 deletions .github/workflows/download-link-check-issue.yml
Original file line number Diff line number Diff line change
@@ -46,7 +46,7 @@ jobs:

- name: Run Lychee
id: lychee
uses: lycheeverse/[email protected].1
uses: lycheeverse/[email protected].4
with:
args: ${{ inputs.lychee_args }}
env:
@@ -85,7 +85,7 @@ jobs:
labels: ${{ inputs.labels }}

- name: Close if Check Passes
uses: peter-evans/close-issue@v2.1.0
uses: peter-evans/close-issue@v2.2.0
if: >-
steps.lychee.outputs.exit_code == 0 &&
steps.find-existing-comment.outputs.result
2 changes: 1 addition & 1 deletion .github/workflows/link-check-all.yml
Original file line number Diff line number Diff line change
@@ -5,4 +5,4 @@ on:
- cron: '0 0 * * *'
jobs:
run:
uses: iterative/link-check/.github/workflows/link-check-all.yml@v0.13.1
uses: iterative/link-check/.github/workflows/link-check-all.yml@v0.14.0
6 changes: 3 additions & 3 deletions .github/workflows/link-check-deploy.yml
Original file line number Diff line number Diff line change
@@ -4,6 +4,6 @@ on:
jobs:
run:
if:
github.event.deployment.ref != ${{ github.event.repository.default_branch }} &&
github.event.deployment_status.state == 'success'
uses: iterative/link-check/.github/workflows/link-check-deployment-status.yml@v0.13.1
github.event.deployment.ref != ${{ github.event.repository.default_branch
}} && github.event.deployment_status.state == 'success'
uses: iterative/link-check/.github/workflows/link-check-deployment-status.yml@v0.14.0
39 changes: 39 additions & 0 deletions .github/workflows/lint-and-test.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
name: 'Linting and Testing'

on: pull_request

concurrency:
group: lint-${{ github.head_ref }}
cancel-in-progress: true

jobs:
run-tests:
name: 'Run Tests'
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v3

- name: Install Node.js and use yarn
uses: actions/setup-node@v3
with:
node-version: '18.x'
cache: 'yarn'

- name: Install All Dependencies
run: yarn

- name: Check Formatting with Prettier
run: yarn format-check-all

- name: Lint JS with ESLint
run: yarn lint

- name: Lint CSS with Stylelint
run: yarn lint-css

- name: Check TypeScript compilation
run: yarn lint-ts

- name: Run tests
run: yarn test
16 changes: 9 additions & 7 deletions .github/workflows/update.yaml
Original file line number Diff line number Diff line change
@@ -4,9 +4,10 @@ on:
- cron: '0 18 * * *'
jobs:
update:
runs-on: ubuntu-18.04
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- uses: iterative/setup-cml@v1
- name: Update
id: update
shell: bash
@@ -15,11 +16,12 @@ jobs:
version=$(curl --silent $url | jq -r .tag_name)
path=src/components/DownloadButton/index.tsx
sed -i "s/^const VERSION = .*$/const VERSION = \`$version\`/g" $path
echo "::set-output name=changes::$(git diff)"
echo "::set-output name=version::$version"
echo "changes=$(git diff)" >> $GITHUB_OUTPUT
echo "version=$version" >> $GITHUB_OUTPUT
- name: Create PR
if: ${{ steps.update.outputs.changes != '' }}
uses: peter-evans/create-pull-request@v4
with:
commit-message: dvc ${{ steps.update.outputs.version }}
title: dvc ${{ steps.update.outputs.version }}
run: |
cml pr create \
--token="${{ secrets.GITHUB_TOKEN }}" \
--message="dvc ${{ steps.update.outputs.version }}" \
--title="dvc ${{ steps.update.outputs.version }}" .
5 changes: 4 additions & 1 deletion .husky/pre-commit
Original file line number Diff line number Diff line change
@@ -1 +1,4 @@
yarn format-staged && yarn lint-staged
#!/usr/bin/env sh
. "$(dirname -- "$0")/_/husky.sh"

yarn exec lint-staged
6 changes: 0 additions & 6 deletions .pre-commit-hooks.yaml

This file was deleted.

27 changes: 10 additions & 17 deletions .restyled.yaml
Original file line number Diff line number Diff line change
@@ -1,21 +1,14 @@
restylers:
- name: prettier
arguments:
[
'--write',
'--arrow-parens=avoid',
'--print-width=80',
'--prose-wrap=always',
'--no-semi',
'--single-quote=true',
'--tab-width=2',
'--trailing-comma=none',
'--use-tabs=false'
]
include: # matching '**/*.{js,jsx,md,tsx,ts,json}'
- '**/*.md'
- '**/*.jsx'
- '**/*.tsx'
arguments: ['--config=./.prettierrc', '--ignore-path=./.prettierignore']
include:
- '**/*.css'
- '**/*.js'
- '**/*.ts'
- '**/*.json'
- '**/*.jsx'
- '**/*.md'
- '**/*.mdx'
- '**/*.ts'
- '**/*.tsx'
- '**/*.yaml'
- '**/*.yml'
8 changes: 4 additions & 4 deletions .stylelintrc
Original file line number Diff line number Diff line change
@@ -1,10 +1,8 @@
{
"extends": "stylelint-config-standard",
"extends": ["stylelint-config-standard"],
"rules": {
"no-descending-specificity": null,
"no-duplicate-selectors": null,
"declaration-colon-newline-after": null,
"value-list-comma-newline-after": null,
"property-no-unknown": [
true,
{
@@ -20,9 +18,11 @@
"at-rule-no-unknown": [
true,
{
"ignoreAtRules": ["mixin", "tailwind"]
"ignoreAtRules": ["mixin", "tailwind", "screen"]
}
],
"custom-property-pattern": null,
"selector-id-pattern": null,
"selector-class-pattern": null,
"keyframes-name-pattern": null,
"function-no-unknown": [
3 changes: 3 additions & 0 deletions CODEOWNERS
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
* @iterative/websites

/content/docs/
16 changes: 5 additions & 11 deletions config/link-check/excluded-links.yml
Original file line number Diff line number Diff line change
@@ -1,20 +1,14 @@
- 'http://localhost:8000**'
- 'http://millionsongdataset.com/pages/getting-dataset/#subset'
- 'https://marketplace.visualstudio.com/items?itemName=stkb.rewrap'
- 'https://marketplace.visualstudio.com/items?itemName=redhat.vscode-yaml'
- 'https://remote.dvc.org/get-started'
- 'https://dvc.us10.list-manage.com/subscribe/post?u=a08bf93caae4063c4e6a351f6&id=24c0ecc49a'
- 'https://www.meetup.com/San-Francisco-Machine-Learning-Meetup/events/264846847/'
- 'https://www.reddit.com/r/MachineLearning/comments/bx0apm/d_how_do_you_manage_your_machine_learning/'
- 'https://www.amazon.com/DevOps-Handbook-World-Class-Reliability-Organizations-ebook/dp/B01M9ASFQ3'
- '**linkedin.com/in/**'
- '**linkedin.com/company/**'
- 'link'
- '/img/<filename>.gif'
- '/uploads/images/2020-02-10/image.png'
- 'https://portal.aws.amazon.com/gp/aws/developer/registration/index.html'
- 'https://github.com/iterative/dvc/releases/download/'
- 'https://www.kaggle.com/kurianbenoy/introduction-to-data-version-control-dvc'
- 'https://timheuer.com/blog/skipping-ci-github-actions-workflows/'
- 'https://www.zazzle.com/t_shirt-235920696568133954'
- 'https://medium.com**'
- 'src/*'
- 'data/*'
- 'models/*'
- 'mailto:*'
- 'https://machinelearningmastery.com/*'
39 changes: 39 additions & 0 deletions content/data/home.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
- title: Connect storage to repo
description:
Keep large data and model files alongside code and share via your cloud
storage.
terminal: |
$ dvc add cats-dogs

$ dvc remote add storage s3://bucket/dvc-cache

$ dvc push
5000 files pushed
- title: Configure steps as you go
description:
Declare dependencies and outputs at each step to build reproducible
end-to-end pipelines.
terminal: |
$ dvc exp run
"data/data.xml.dvc" didn't change, skipping^500
Stage "prepare" didn't change, skipping^500
Stage "featurize" didn't change, skipping^500
Running stage "train":
> python src/train.py data/features model.pkl
- title: Track experiments in Git
description:
Track experiments in your repo, compare results and restore entire
experiment states cross-team.
terminal: |
$ dvc exp show
───────────────────────────────────────────────────────────────────────────
<span class="text-black"><span class="bg-stone-100"> Experiment </span> <span class="bg-orange-200"> roc_auc.train </span> <span class="bg-orange-200"> roc_auc.test </span> <span class="bg-cyan-200"> train.min_split </span></span>
───────────────────────────────────────────────────────────────────────────
<b>workspace 0.96464 0.93989 0.04</b>
<b>main 0.98667 0.94602 0.01</b>
├── 0417ff0 [fuggy-zack] 0.98667 0.94602 0.01
├── cb5c648 [fishy-mene] 0.97652 0.94309 0.02
├── c0dca21 [atrip-teds] 0.96957 0.9407 0.03
├── 3d2c9c7 [brisk-aged] 0.96464 0.93989 0.04
└── 6e910c4 [chirk-main] 0.96193 0.93979 0.05
───────────────────────────────────────────────────────────────────────────
Loading
You are viewing a condensed version of this merge commit. You can view the full changes here.