Reorganize cuDF Python docs #10691

shwina · 2022-04-20T17:17:47Z

This PR is composed of two high-level changes:

Replaces the use of ReStructuredText with MyST Markdown. I used rst2myst for this and it worked pretty well. The rationale for this change is simple: we use myst-nb to render notebooks into documentation, and for consistency, it's nice to use myst-parser to parse the rest of our docs too. As a matter of opinion, I think Markdown is simpler and more familiar to most developers.
Reorganizes the docs (see below):

Prior to this PR, the cuDF documentation was divided into 3 sections:

A user guide
A "Basics" section
API reference

The distinction between the first two sections was never clear. I've gone ahead and merged those into a single section named "User Guide". This is also more consistent with Pandas.

This PR also makes a couple of other changes:

Renamed the "Basics" page under the previous "Basics" section to "Data Types", as that reflects its contents more accurately. I also modified the content here a bit.
Renamed the "10 minutes to CuPy and CuDF" notebook to "Interoperability between CuPy and CuDF" as that more accurately describes what that page is about.

Compare the TOC from this PR (below) with our currently published docs.

…reorg-docs

review-notebook-app · 2022-04-20T17:17:51Z

Check out this pull request on

See visual diffs & provide feedback on Jupyter Notebooks.

Powered by ReviewNB

codecov · 2022-04-20T18:34:26Z

Codecov Report

Merging #10691 (1857972) into branch-22.06 (84f88ce) will increase coverage by 0.02%.
The diff coverage is 100.00%.

❗ Current head 1857972 differs from pull request most recent head e68b86e. Consider uploading reports for the commit e68b86e to get more accurate results

@@               Coverage Diff                @@
##           branch-22.06   #10691      +/-   ##
================================================
+ Coverage         86.40%   86.43%   +0.02%     
================================================
  Files               143      143              
  Lines             22444    22448       +4     
================================================
+ Hits              19393    19402       +9     
+ Misses             3051     3046       -5

Impacted Files	Coverage Δ
python/cudf/cudf/_fuzz_testing/utils.py	`0.00% <ø> (ø)`
python/cudf/cudf/comm/gpuarrow.py	`79.76% <ø> (ø)`
python/cudf/cudf/core/column/string.py	`89.21% <ø> (+0.12%)`	⬆️
python/cudf/cudf/core/frame.py	`93.41% <ø> (ø)`
python/cudf/cudf/core/series.py	`95.16% <ø> (ø)`
python/custreamz/custreamz/kafka.py	`29.16% <ø> (ø)`
python/cudf/cudf/core/column/interval.py	`83.67% <100.00%> (ø)`
python/cudf/cudf/core/column/struct.py	`96.42% <100.00%> (ø)`
python/cudf/cudf/core/dataframe.py	`93.74% <100.00%> (+0.04%)`	⬆️
python/cudf/cudf/core/reshape.py	`90.11% <100.00%> (+0.11%)`	⬆️
... and 10 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 84f88ce...e68b86e. Read the comment docs.

…reorg-docs

shwina · 2022-04-26T15:03:03Z

docs/cudf/source/user_guide/guide-to-udfs.ipynb

@@ -2,15 +2,16 @@
 "cells": [


This looks to be an issue with the way ReviewNB diffs things. There's no ==================================== in the source. Ditto in many places below.

Reply via ReviewNB

shwina · 2022-04-26T15:05:30Z

docs/cudf/source/user_guide/missing-data.ipynb

@@ -2,46 +2,31 @@
 "cells": [


I removed the hardcoded table of contents below as Sphinx/myst-nb already generate one for us.

Reply via ReviewNB

galipremsagar

Overall looks great to me Ashwin. Except for a few rendering issues that I mentioned below.

docs/cudf/source/user_guide/data-types.md

docs/cudf/source/user_guide/groupby.md

galipremsagar · 2022-04-27T17:00:08Z

docs/cudf/source/user_guide/io.md

 .. table::
+    :class: io-supported-types-table special-table
    :widths: 15 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10

    +-----------------------+--------+--------+--------+--------+---------+--------+--------+--------+--------+-------------------+--------+--------+---------+---------+


I like how GitHub thinks you only moved this and 2 more files while you also made similar file movements in this PR 🤣

Yeah what the heck is that about...

…reorg-docs

galipremsagar

Changes LGTM @shwina and all the issues are now fixed. Thank you!

bdice

Really nice job @shwina. I haven't had a chance to render the docs locally, but I thought I would take a peek. I left a handful of comments, but only one (on index.md) is probably blocking. Feel free to skip/ignore any comments or deal with them in future PRs. I recognize this PR is huge so I don't want to block it. Hope the feedback is helpful! :)

docs/cudf/source/user_guide/dask-cudf.md

bdice · 2022-05-04T00:51:07Z

docs/cudf/source/user_guide/data-types.md

+also provide special data types for working with decimals, list-like,
+and dictionary-like data.
+
+All data types in cuDF are [nullable](/user_guide/missing-data).


This should use the MyST equivalent of Sphinx's :ref: rather than an absolute path. The absolute path won't work in rendered HTML unless it's being served at the root. That's prone to breakage. This piece of the documentation might help: https://myst-parser.readthedocs.io/en/latest/syntax/syntax.html#targets-and-cross-referencing

Used a relative path instead

A relative path is an improvement but not quite as good as ref syntax. If the page is moved or renamed, the ref will still pick it up as long as it's in the same Sphinx project. refs with missing referents will throw warnings, and refs are supported by non-HTML Sphinx outputs. An absolute/relative path won't be checked for correctness afaik.

With markdown syntax [nullable](mossing-data), I get:

cudf/docs/cudf/source/user_guide/data-types.md:8: WARNING: 'myst' reference target not found: mossing-data

The markdown syntax translates into Sphinx's any role, which seems pretty flexible in what it can do.

Nice to know! If it's translated to the any role, that's awesome. I thought it would be interpreted as a link, rather than a reference (any will match ref or any other role).

(also "mossing data" 😆 )

bdice · 2022-05-04T00:53:50Z

docs/cudf/source/user_guide/data-types.md

+## NumPy data types
+
+We use NumPy data types for integer, floating, datetime, timedelta,
+and string data types.  Thus, just like in NumPy,


Is it weird that this always uses two spaces between sentences? Personally I prefer one space between sentences.

This is just Emacs'Emacs' auto-fill at work. Personally, I think Semantic Linefeeds are the best convention for documentation source, but it's hard to incorporate them in anything but personal projects :(

Vim’s “gq” does the same thing with two spaces. I would be open to rewriting this with semantic lines (or the easier to enforce “one sentence per line”). I have had no issues in practice with using that style on other projects. This word wrap is pretty harsh right now - it’s wrapping at less than 80 chars (74?), so I think it is unlikely that this word wrapped formatting will survive more than one or two revisions anyway.

Perhaps a discussion to have offline/as part of the dev guide?

I would not oppose making that change in this PR, since all the files are being rewritten anyway. If you don't want to make that decision in this PR, this would be a good topic for tomorrow's cuDF Python meeting.

+1 for semantic linefeeds

bdice · 2022-05-04T00:59:36Z

docs/cudf/source/user_guide/data-types.md

+`np.dtype("float32")`, `np.float32`, and `"float32"` are all acceptable
+ways to specify the `float32` data type:
+
+```python


I discovered recently that code blocks using >>> are (depending on the renderer) supposed to be rendered as pycon: Python console, rather than python.

That applies to GitHub Flavored Markdown via Linguist and also content highlighted with Pygments. It appears that MyST uses Pygments.

TIL! I can confirm that it works, but are there any advantages to using pycon over python (either ways, I'd prefer to change this in a follow up)

Ah I see, things like keywords in output/tracebacks aren't syntax highlighted by mistake. That's really cool!

Yep! It was definitely a "TIL" thing for me too. I think a lot of projects just use python and don't know the difference.

docs/cudf/source/user_guide/data-types.md

docs/cudf/source/user_guide/internals.md

docs/cudf/source/index.rst

docs/cudf/source/user_guide/dask-cudf.md

Co-authored-by: Bradley Dice <[email protected]>

bdice

Thanks @shwina for your dedication to this!

shwina · 2022-05-04T18:53:22Z

Rerun tests

shwina · 2022-05-04T21:39:23Z

@gpucibot merge

mmccarty and others added 5 commits April 19, 2022 15:26

Migrated user guide notebooks to MyST-NB and added sphinx extension

4116d87

Reorg

bb807e1

Change CuPy notebook name

fa5c524

Merge branch 'branch-22.06' of https://github.com/rapidsai/cudf into …

23e8a6c

…reorg-docs

Remove duplicate

40e96e2

github-actions bot added the conda label Apr 20, 2022

Change title

1fad8a2

shwina mentioned this pull request Apr 20, 2022

Migrated user guide notebooks to MyST-NB and added sphinx extension #10685

Merged

3 tasks

mmccarty and others added 9 commits April 21, 2022 10:19

Using MyST-NB sphinx extension for Notebook execution and rendering

43ddc2d

Add Pandas comparison docs

5e70c13

Move back to ipynb

dd7c480

Deletee md

9828b01

Merge branch 'myst-nb' of github.com:mmccarty/cudf into reorg-docs

f8ff46e

Merge branch 'reorg-docs' into add-pandas-comparison

1f64e42

Add more Pandas comparison notes

bb87e73

Run notebooks

3f9fcf1

Merge branch 'add-pandas-comparison' into reorg-docs

867b8ca

mmccarty and others added 6 commits April 22, 2022 13:06

set overflow for notebook output

21bdce9

Update docs/cudf/source/_static/params.css

9e22eda

Move from .rst to .md

77dae5d

Merge branch 'myst-nb' of github.com:mmccarty/cudf into reorg-docs

09c9517

Cache notebook runs

784eab2

Improvements to pandas comparison docs

f2daba8

shwina mentioned this pull request Apr 25, 2022

Add a "comparison with Pandas" section to user guide #10708

Closed

shwina added 2 commits April 25, 2022 13:16

Remove Pandas comparison doc for now

c458622

Merge branch 'branch-22.06' of https://github.com/rapidsai/cudf into …

69d472c

…reorg-docs

github-actions bot removed the conda label Apr 26, 2022

shwina added 3 commits April 26, 2022 10:49

Use a wrapper to make special-table scrollable

da86dc0

Undo change to jupyter_execute_notebooks in conf.py

4afb2e3

Remove duplicate TOC in missing data notebook

b5444a9

shwina added doc Documentation non-breaking Non-breaking change labels Apr 26, 2022

shwina marked this pull request as ready for review April 26, 2022 14:58

shwina commented Apr 26, 2022

View reviewed changes

shwina requested a review from galipremsagar April 26, 2022 15:06

galipremsagar reviewed Apr 27, 2022

View reviewed changes

shwina added 5 commits April 29, 2022 06:40

Merge branch 'branch-22.06' of https://github.com/rapidsai/cudf into …

5b7809f

…reorg-docs

Improve data-types table

90fe4f5

[skip-ci] Add the result of gsr.dtype

9427197

[skip-ci] Fix warning render

7ef61af

[skip-ci] or -> and

f8bc555

galipremsagar approved these changes May 3, 2022

View reviewed changes

bdice reviewed May 4, 2022

View reviewed changes

mmccarty approved these changes May 4, 2022

View reviewed changes

shwina and others added 5 commits May 4, 2022 13:27

Changes after review

52fc1bf

Update docs/cudf/source/user_guide/internals.md

41fdb06

Co-authored-by: Bradley Dice <[email protected]>

[skip-ci] More suggestions from review

1687d32

gramar

1085d34

Merge branch 'reorg-docs' of github.com:shwina/cudf into reorg-docs

e68b86e

bdice approved these changes May 4, 2022

View reviewed changes

rapids-bot bot merged commit dd68db3 into rapidsai:branch-22.06 May 4, 2022

isVoid mentioned this pull request May 18, 2022

Update Documentation with Pydata Sphinx Theme, and more rapidsai/cuspatial#523

Merged

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Reorganize cuDF Python docs #10691

Reorganize cuDF Python docs #10691

shwina commented Apr 20, 2022 •

edited

Loading

review-notebook-app bot commented Apr 20, 2022

codecov bot commented Apr 20, 2022 •

edited

Loading

shwina Apr 26, 2022

shwina Apr 26, 2022

galipremsagar left a comment

galipremsagar Apr 27, 2022

shwina Apr 29, 2022

galipremsagar left a comment

bdice left a comment

bdice May 4, 2022

shwina May 4, 2022

bdice May 4, 2022

shwina May 4, 2022

bdice May 4, 2022 •

edited

Loading

bdice May 4, 2022

shwina May 4, 2022

bdice May 4, 2022

shwina May 4, 2022

bdice May 4, 2022

vyasr May 4, 2022

bdice May 4, 2022 •

edited

Loading

shwina May 4, 2022

shwina May 4, 2022

bdice May 4, 2022

bdice left a comment

shwina commented May 4, 2022

shwina commented May 4, 2022

Reorganize cuDF Python docs #10691

Reorganize cuDF Python docs #10691

Conversation

shwina commented Apr 20, 2022 • edited Loading

review-notebook-app bot commented Apr 20, 2022

codecov bot commented Apr 20, 2022 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

galipremsagar left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

galipremsagar left a comment

Choose a reason for hiding this comment

bdice left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bdice May 4, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bdice May 4, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bdice left a comment

Choose a reason for hiding this comment

shwina commented May 4, 2022

shwina commented May 4, 2022

shwina commented Apr 20, 2022 •

edited

Loading

codecov bot commented Apr 20, 2022 •

edited

Loading

bdice May 4, 2022 •

edited

Loading

bdice May 4, 2022 •

edited

Loading