DOC: Improving code quality of doc/make.py, PEP-8 and refactoring (#19631)

[datapythonista]

• closes DOC: Improve code quality of doc/make.py #19631
  closes DOC: enable parallel building of the docs #15591
  closes doc/make.py only works for development build #10340
• tests added / passed
• passes git diff upstream/master -u -- "*.py" | flake8 --diff
• whatsnew entry

Summary of the PR:

• doc/make.py in lint and respects PEP-8
• Making the structure of the code more organized
• Added command line arguments for the number of sphinx jobs (DOC: enable parallel building of the docs #15591), for the PYTHONPATH (doc/make.py only works for development build #10340)
• Should work in both Python 2 and 3
• Tracebacks are returned if a command fails (before just the output from the command was shown, but no traceback on where the exception happened).

[jreback]

a lot of these are not used so can be removed

@TomAugspurger @jorisvandenbossche

IOW we only need html, latex, clean, can remove all of the upload* ones

[jorisvandenbossche]

zip_html is also used.
I did use the upload commands when I uploaded the docs. Not sure how Tom currently does it (I know we changed some things in where we upload the versioned docs and how they are symlinked)

[jreback]

do we actually support this?

[datapythonista]

the current script does, both python make.py and python make.py all are equivalent to python make.py html. Would rather get rid of all and would show the usage if no command is provided, but I tried to keep compatibility with the old script.

[jreback]

i understand but let's remove what we dont need instead

[jreback]

when is this actually called?

[datapythonista]

Current script supports python make.py build_pandas, not sure if anyone is using it, happy to get rid of it.

[jorisvandenbossche]

never used it myself

[datapythonista]

Updated the PR, I left html, latex, clean and zip_html.

Just let me know if something else should be kept.

[jorisvandenbossche]

latex_forced is used as well I think

[TomAugspurger]

Yep latex_forced is user too. https://github.com/pandas-dev/pandas/wiki/Release-Checklist should have everything other than the symlinks, which I’ve done manually.

…

________________________________ From: Joris Van den Bossche <[email protected]> Sent: Sunday, February 11, 2018 4:08:19 PM To: pandas-dev/pandas Cc: Tom Augspurger; Mention Subject: Re: [pandas-dev/pandas] DOC: Improving code quality of doc/make.py, PEP-8 and refactoring (#19631) (#19634) latex_forced is used as well I think — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#19634 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/ABQHIsfvrD2eKVCJCu2MfzdcSrMbbGkFks5tT2TTgaJpZM4SBAvW>.

[datapythonista]

Updated the PR with latex_forced.

@TomAugspurger, we've got rid of make.py all, which was implemented as an alias of make.py html. But may be we can get it back to execute all the steps for the release: ./make.py clean; ./make.py; ./make.py zip_html; ./make.py latex_forced.

Just let me know if you think that's a good idea.

[jreback]

thanks @datapythonista

so this will prob need to have a followup with some instructions on how to use it? or is completely obvious from the --help? maybe a README.md?

[datapythonista]

@jreback a README never hurts, and we could surely have some more info, but I think the usage is quite covered in:

• The docstring at the beginning of the script
• The docstrings of each command
• The README.rst file in doc/
• The Contributing to documentation guide:https://pandas.pydata.org/pandas-docs/stable/contributing.html#id42
• And the release checklist: https://github.com/pandas-dev/pandas/wiki/Release-Checklist
• The help command

I'm working in implementing a way to build a single document from the api (and pep8fying doc/source/conf.py which I forgot in this PR). I'll review the documentation about doc/make.py once I finish that, and add documentation for anything not explained.

[jorisvandenbossche]

I'm working in implementing a way to build a single document from the api (

What do you already have? Because I updated yesterday my script I shared, and it is basically working (was planning to do a PR to add it this morning, but did not yet come to it)

[datapythonista]

@jorisvandenbossche I'll share what I've got in a branch later today. But my approach was to create a temporary directory, copy the files to have a minimal sphinx project there, run sphinx on it, and then copy the single api page it generates to the real project.

I see 2 advantages in this approach, first I found it tricky to build a single page from the api from sphinx, and second I prefer to avoid deleting all the .rst files to build the project. Doing a git checkout on them later is not a big deal if you know about it, but newcomers may find it confusing.

[jreback]

you know —single already exists yes?

[datapythonista]

@jreback thanks for double checking, but yes, I do. Actually I want to reuse it for simplicity of the interface. But now you can do something like ./make.py html --single contributing.rst but not something like ./make.py hml --single pandas.DataFrame.head which is what @jorisvandenbossche and myself were discussing, and which will be very convenient for the sprint.

[jorisvandenbossche]

I was planning to remove this "OK to delete those files" as well, as it is indeed very annoying (and I think not needed, if you specify to sphinx that those files can be ignored, that's the approach I was taking)

I will put up my approach as well as a PR, and then you can test it and we can see which approach is the easiest for the end user.

[datapythonista]

@jorisvandenbossche if you plan to remove the file deletion, which I think it's implemented in sphinx conf.py, can you please first review and merge #19839, so there are no conflicts with it later? It's a simple PR to make the file PEP-8 compatible.

If you can build a single api doc without deleting the rst, that's a surely much better approach than what I did. I didn't see how to do that. There are still couple of things missing, but this is my approach: datapythonista@5b4cc3b

[jorisvandenbossche]

@datapythonista my attempt is here: #19840. I used exclude_patterns to not have to delete the files. This now seems to work smoothly (try eg python make.py --docstring DataFrame.join), apart from a problem I have with subprocess.check_call (will comment on the PR).
I also still can be smarter (now hardcoded 'method', but could in principle detect this from the passed method/function name, instead of requiring the user to specify this).

#19839 will not give too much conflict, I only deleted some lines from conf.py