Basic Concepts Section for the Documentation #2177

iesahin · 2021-02-11T08:38:04Z

You may disregard these recommendations if you used the Edit on GitHub button from dvc.org to improve a doc in place.

❗ Please read the guidelines in the Contributing to the Documentation list if you make any substantial changes to the documentation or JS engine.

🐛 Please make sure to mention Fix #issue (if applicable) in the description of the PR. This causes GitHub to close it automatically when the PR is merged.

Please choose to allow us to edit your branch when creating the PR.

Thank you for the contribution - we'll try to review it as soon as possible. 🙏

…or links. runs ok.

…s we add content now

…kage.json

…nd dependency.

…"frozen stage"

…b-21 Restyle ug-basic-concepts-feb-21

iesahin · 2021-02-11T14:36:59Z

I checked src/gatsby/models/glossary/index.js for failing codeclimate test and decided splitting the mentioned createSchemaCustomization function actually increases the complexity. Leaving it as it is.

shcheklein · 2021-02-11T19:49:21Z

@iesahin could we take the JS part as a separate PR please and describe what it does? (Overall a bit of preference here - it's easier for us to review and merge smaller and focused PRs)

jorgeorpinel · 2021-02-12T07:05:52Z

content/docs/sidebar.json

+        "label": "Basic Concepts",
+        "slug": "concepts",
+        "source": false,
+        "children": [
+          "workspace",
+          "dvc-project",
+          "dependency",
+          "external-dependency",
+          "data-pipelines",
+          "stage",
+          "dvc-cache",
+          "dvc-metafiles",
+          "remote-storage",
+          "import-stage",
+          "metrics-and-plots",
+          "parameters",
+          "run-cache"
+        ]
+      },


Too many changes in a single PR 🙂 Let's start with the js engine changes as Ivan suggested (for this PR)? Then (separate PR) the 1-3 most important concept pages — that in itself will be a big contribution.

That said, I also have a couple comments on (1) the concept naming, and (2) level of abstraction:

Let's use dvc- everywhere (where it can apply e.g. dvc-workspace?) or nowhere, I think.

Not everything that has a tooltip needs a concept page e.g. "external dependency" is a bit specific and falls within the "dependency" concept. Not sure which ones to keep exactly, we can decide later.

Thanks

Also on the level of abstraction: we could even consider putting related concepts together e.g. cache and remote, pipeline and stage (they could still each need separate tooltips though). WDYT? (also for later to worry about anyway)

I agree with this. I tried to make minimal changes to file naming as I think that was a decided matter but we can change all to non-dvc versions IMO.

You're also right on this but tooltips in the branch is using frontmatter and there can only be a single frontmatter for each page. We can either have a dependency tooltip/glossary and tell external dependency when the user visits the concept page, or all glosses should have separate pages. IMHO glossary/tooltips and concepts should be separated and we need to have 5-10 concept pages and put tooltips/glossary in a single file.

This also needs separation of glossary and basic concepts. Current structure requires single page, short documents for concepts which has tooltips defined at the top.

On 1. We're moving all the files to another directory I think so no problem in renaming them along the way, I think.
On 2. I think we can have empty concept md docs that only serve for tooltip purposes, but not link them in sidebar.js right?
On 3. True. For now we can pick which concept doc is more important and put other concepts in there (even if they have their own concept file that's only used for tooltips. Anyway, no need to worry about this for the first or 2nd PR probably...

p.s. the tooltip vs concept page separation part we can worry about when looking into #1395

package.json

jorgeorpinel · 2021-02-12T07:11:51Z

content/docs/user-guide/concepts/data-pipelines.md

+<!-- keywords: data pipeline, machine learning pipeline, devops for data science, devops for machine learning, "MLops", "what is a data pipeline?", "data pipeline examples", "machine learning (ML) workflow", "data science workflow", "data science pipeline workflow" -->
+


So FYI (next PR) these are desired terms we want to try to get into the concept text for SEO purposes.

Writing a document that puts all these words would be a challange 😄

Instead there can be a keywords: front matter that will be copied to meta tags of pages. Talking to search engine bots and talking to humans are different purposes IMO. These keywords can also be added to other pages.

Search engines don't really care about metadata keywords anymore, we need to incorporate the keywords to the content (in a meaningful/truthful way) but yeah no need to have them all strictly, especially in small concept pages that will be impossible.

jorgeorpinel · 2021-02-12T07:13:54Z

content/docs/user-guide/concepts/data-pipelines.md

+# Data Pipelines
+
+A data pipeline, in general, is a series of data processing <abbr>stages</abbr>
+like data gathering, data transformation, model training, model testing and
+reporting <abbr>metrics</abbr> and results. A pipeline may produce intermediate


As you may have realized, a lot of the text in this branch comes from existing content, in this case https://dvc.org/doc/command-reference/dag.

So the (next) PR should also remove most of those explanations from the cmd ref. — another reason to focus on just one concept at a time per PR probably — not to mention all the link/tooltip updates that some of these concepts will require throughout docs 😱

I'm not against using duplicate material as soon as it fits to the context. Concepts pages should be revised to contain less details but overall removing material from other parts of the documentation may not be necessary.

Users will be reading these sporadically and they probably will forget what they read in concepts when they are reading the command ref. Aesthetically it may not be pleasing to repeat ourselves but from POV of a user, these are separate instances of reading. Also we can't be sure the user will ever read the other if they read one of them.

removing material from other parts of the documentation may not be necessary

That's actually an important goal for this task (see #550 (comment)). Some (a little) redundancy is OK but there should be a single source of full/comprehensive explanation for these concepts. Keep in mind that maintaining duplicated content is hard and error-prone 😅 (just like code)

Users will be reading these sporadically and they probably will forget

TBH I don't expect users to navigate to concepts much it at all (from nav). Like you suggested without context they're pretty hard to care about or understand. They'll probably only open them from links we place in other docs, and from the tooltips of course.

It would be great though for this section to serve as an entry point. May be index page can give an overview of the most important ones? how do hey relate to each other.

Good idea. I agree about an guiding index page for those who do navigate in here. BUT probably the 1st PR (after this one) will only introduce 1 or a few concepts and the index may not make sense until we have most of them up.

iesahin · 2021-02-12T08:39:26Z

I fully agree that this should be split into smaller PRs but this is a breaking change. JS part is actually copied from what @rogermparent did in ug-basic-concepts here

If we don't include JS changes to this, Basic Concepts don't appear in the sidebar and tooltips become large documents. JS part involves showing tooltip frontmatter instead of the whole page and actually directly copied from @rogermparent 's work.

Because of this I had to bundle all pages together. If we apply only the JS part and an .md file under /concepts/ dir doesn't have tooltip: frontmatter, it breaks the existing glossary/tooltip functionality. The content is also extended in most of the pages.

I'll be merging master into this branch and update the documents as you review but splitting this into smaller pieces will probably do more harm than good.

Thanks. @shcheklein @jorgeorpinel

shcheklein · 2021-02-12T20:16:07Z

@iesahin

I fully agree that this should be split into smaller PRs but this is a breaking change. JS part is actually copied from what @rogermparent did in ug-basic-concepts here

makes sense! Can we make them as a separate PR still? May be just migrate existing tooltips to the new format? I'm happy to merge it with a content that just copies existing tooltips. Then we can proceed step by step improving it. WDYT?

jorgeorpinel · 2021-02-12T22:14:51Z

this is a breaking change
If we don't include JS changes to this, Basic Concepts don't appear in the sidebar and tooltips become large documents

@iesahin you can just leave all the json and js changes EXCEPT for sidebar.js AND move the existing md files (without changes) from basic-concepts/ to concepts/ for the first PR. That will preserve the existing tooltip functionality and enable editing concept pages and adding them to the nav in future PRs (I tried it locally and it seems to work). 👍

That's what I was referring to with "splitting the PR".

jorgeorpinel · 2021-02-12T22:16:17Z

move the existing md files (without changes

No sorry that's not accurate. The only change would be to leave the new frontmatter (with tooltip fields) but the md content of all those pages would be empty initially.

iesahin · 2021-02-13T07:46:48Z

It will be an interesting cherry-picking experience :)

I'll send you another PR by cherry picking the tooltip on the front matter functionality but without the concept pages in sidebar. That's the deal I think.

iesahin · 2021-02-13T07:47:48Z

~~Also I'll try to fix prettier by changing its config because it's good test.~~

This is in another PR. #2194

iesahin · 2021-02-13T13:44:04Z

I split JS parts, didn't touch current text in master but moved them to tooltip: front matter. Submitted #2192 for this. We can close this.

jorgeorpinel · 2021-02-14T23:57:12Z

Thanks @iesahin !

Emre Şahin and others added 30 commits February 8, 2021 10:19

merged older basic-concepts docs with new updates. modified sidebar f…

b5eaa63

…or links. runs ok.

prettifed.

6275f44

added title for the page

10fffb5

Merge branch 'master' into ug-basic-concepts-feb-21

0c62268

glossary model operates under docs/user-guide/concepts now

d38f1df

no need to skip docs/user-guide/concepts in page/sidebar generation a…

5f7b521

…s we add content now

applied changes for tooltips from ug-basic-concepts

0837cb5

added tooltip stubs for all concept pages and required package to pac…

7ea0feb

…kage.json

Updated run-cache description and tooltip

4a25d0c

Updated BC external dependency

8de0dee

prettified

ae92e91

added preliminary definitions for all missing tooltips

b19a999

husky changes

acb1d6e

Updated dvc-project concepts document

5957a3b

Revised and updated dvc-cache concepts document.

c57d3a4

Revised and updated dvc-metafiles document

4668f29

Merge branch 'master' into ug-basic-concepts-feb-21

e5959c0

updated and revised bc:dvc-metafiles document

a039900

added further reading section to metafiles

528e59a

Revised and added further reading to dvc-project

1d3140d

Added paragraph describing the conceptual difference between remote a…

fa62a6d

…nd dependency.

removed "frozen dependency" document as the concept should be called …

4cb9835

…"frozen stage"

sentence claiming "external dependencies are frozen by default" removed.

c392ddc

updated import-stage document

e78b5bc

Revised and updated metrics and plots BC document

d03efeb

Revised BC output document

8ef1833

prettified

2c3ac10

Revised and updated parameters BC doc

e650831

Updated remote-storage BC document

16650a1

Revised and updated run-cache BC document

122e8e5

Merge pull request #2178 from iterative/restyled/ug-basic-concepts-fe…

6fc2561

…b-21 Restyle ug-basic-concepts-feb-21

shcheklein temporarily deployed to dvc-landing-ug-basic-co-ronvhm February 11, 2021 08:54 Inactive

Reverting styling merged previously

7ab03e2

restyled-io bot mentioned this pull request Feb 11, 2021

Restyle [WIP] Basic Concepts Documents for Status Check #2180

Closed

shcheklein temporarily deployed to dvc-landing-ug-basic-co-ronvhm February 11, 2021 09:38 Inactive

iesahin added 2 commits February 11, 2021 17:28

Fixes broken links

570c434

styling

0b608f6

shcheklein temporarily deployed to dvc-landing-ug-basic-co-ronvhm February 11, 2021 14:29 Inactive

fix broken link

5d0bb8e

shcheklein temporarily deployed to dvc-landing-ug-basic-co-ronvhm February 11, 2021 14:41 Inactive

iesahin changed the title ~~[WIP] Basic Concepts Documents for Status Check~~ Basic Concepts Section for the Documentation Feb 11, 2021

adds [📖] links to the end of each tooltip

1bbe3a1

shcheklein temporarily deployed to dvc-landing-ug-basic-co-ronvhm February 11, 2021 14:57 Inactive

jorgeorpinel reviewed Feb 12, 2021

View reviewed changes

package.json Show resolved Hide resolved

jorgeorpinel reviewed Feb 12, 2021

View reviewed changes

iesahin mentioned this pull request Feb 12, 2021

concepts: links to related docs in concepts pages #2171

Closed

iesahin closed this Feb 13, 2021

jorgeorpinel deleted the ug-basic-concepts-feb-21 branch July 29, 2022 17:07

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Basic Concepts Section for the Documentation #2177

Basic Concepts Section for the Documentation #2177

iesahin commented Feb 11, 2021

iesahin commented Feb 11, 2021 •

edited

Loading

shcheklein commented Feb 11, 2021

jorgeorpinel Feb 12, 2021 •

edited

Loading

jorgeorpinel Feb 12, 2021 •

edited

Loading

iesahin Feb 12, 2021

jorgeorpinel Feb 12, 2021

jorgeorpinel Feb 12, 2021

jorgeorpinel Feb 12, 2021 •

edited

Loading

iesahin Feb 12, 2021

jorgeorpinel Feb 12, 2021 •

edited

Loading

jorgeorpinel Feb 12, 2021 •

edited

Loading

iesahin Feb 12, 2021

jorgeorpinel Feb 12, 2021

shcheklein Feb 12, 2021

jorgeorpinel Feb 12, 2021

iesahin commented Feb 12, 2021

shcheklein commented Feb 12, 2021

jorgeorpinel commented Feb 12, 2021 •

edited

Loading

jorgeorpinel commented Feb 12, 2021

iesahin commented Feb 13, 2021

iesahin commented Feb 13, 2021 •

edited

Loading

iesahin commented Feb 13, 2021

jorgeorpinel commented Feb 14, 2021

		<!-- keywords: data pipeline, machine learning pipeline, devops for data science, devops for machine learning, "MLops", "what is a data pipeline?", "data pipeline examples", "machine learning (ML) workflow", "data science workflow", "data science pipeline workflow" -->

Basic Concepts Section for the Documentation #2177

Basic Concepts Section for the Documentation #2177

Conversation

iesahin commented Feb 11, 2021

iesahin commented Feb 11, 2021 • edited Loading

shcheklein commented Feb 11, 2021

jorgeorpinel Feb 12, 2021 • edited Loading

Choose a reason for hiding this comment

jorgeorpinel Feb 12, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jorgeorpinel Feb 12, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jorgeorpinel Feb 12, 2021 • edited Loading

Choose a reason for hiding this comment

jorgeorpinel Feb 12, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

iesahin commented Feb 12, 2021

shcheklein commented Feb 12, 2021

jorgeorpinel commented Feb 12, 2021 • edited Loading

jorgeorpinel commented Feb 12, 2021

iesahin commented Feb 13, 2021

iesahin commented Feb 13, 2021 • edited Loading

iesahin commented Feb 13, 2021

jorgeorpinel commented Feb 14, 2021

iesahin commented Feb 11, 2021 •

edited

Loading

jorgeorpinel Feb 12, 2021 •

edited

Loading

jorgeorpinel Feb 12, 2021 •

edited

Loading

jorgeorpinel Feb 12, 2021 •

edited

Loading

jorgeorpinel Feb 12, 2021 •

edited

Loading

jorgeorpinel Feb 12, 2021 •

edited

Loading

jorgeorpinel commented Feb 12, 2021 •

edited

Loading

iesahin commented Feb 13, 2021 •

edited

Loading