Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DOC: Change release and whatsnew #21599

Merged
merged 55 commits into from
Nov 14, 2018

Conversation

TomAugspurger
Copy link
Contributor

@TomAugspurger TomAugspurger commented Jun 22, 2018

Some cleanup & changes to facilitate release automation

  • We will include the correct (latest on master or maintenance branch)
    whatsnew directly in the index.rst toctree
  • Contributors are included in the whatsnew for each version (automatically)
  • Removed release.rst
  • Added new releases.rst which has toctrees for each release
    screen shot 2018-06-22 at 3 10 23 pm

Incidental changes

  • Updated style.ipynb. Writing the jinja template was confusing sphinx. We
    included it in the git source now.
  • Fixing some inconsitent header levels (will do more)
  • Refactored announce.py to support auto-generated contributors

TODO:

  • Finish up the rest of the whatsnews

cc @jorisvandenbossche, @datapythonista

Changes:

* We will include the correct (latest on master or maintenance branch)
  whatsnew directly in the index.rst toctree
* Contributors are included in the whatsnew for each version (automatically)
* Removed release.rst
* Added new releases.rst which has toctrees for each release

Incidental changes

* Updated style.ipynb. Writing the jinja template was confusing sphinx. We
  included it in the git source now.
* Fixing some inconsitent header levels (will do more)
* Refactored announce.py to support auto-generated contributors
source_suffix = '.rst'
source_suffix = [
'.rst',
'.txt',
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

required so that the whatsnew files can be included in the toctree.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there actually a reason they are not *.rst ?

@@ -298,8 +303,39 @@
for page in moved_api_pages
}


common_imports = """\
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Previously we had this at the top of whatsnew.rst. Now it has to be in every whatsnew/v0.x.x.txt, as they are no longer literally included below this. Usage is just {{ common_imports }}

'v0.23.2': announce.build_string('v0.23.1..HEAD'),
'v0.23.1': announce.build_string('v0.23.0..v0.23.1'),
'v0.23.0': announce.build_string('v0.22.0..v0.23.0'),
}
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is run every time we build the docs. It'll probably be run even for single page / docstring ones. I was concerned it would take a while, but it's surprisingly fast.

Once I finish the rest of the whatsnew, I'll time it again and look into ways of speeding it up if is a problem.

"""
# http://ericholscher.com/blog/2016/jul/25/integrating-jinja-rst-sphinx/
# Make sure we're outputting HTML
if app.builder.format != 'html':
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm I should look into what happens for PDF. We should be able to handle it just fine.

@@ -118,7 +118,7 @@ See the package overview for more detail about what's in the library.
{{ single_doc }}
{% endif -%}
{% if not single_doc -%}
whatsnew
What's New <whatsnew/v0.24.0>
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We'll update this as part of the post-release process.

@codecov
Copy link

codecov bot commented Jun 23, 2018

Codecov Report

Merging #21599 into master will not change coverage.
The diff coverage is n/a.

Impacted file tree graph

@@           Coverage Diff           @@
##           master   #21599   +/-   ##
=======================================
  Coverage   92.24%   92.24%           
=======================================
  Files         161      161           
  Lines       51340    51340           
=======================================
  Hits        47361    47361           
  Misses       3979     3979
Flag Coverage Δ
#multiple 90.64% <ø> (ø) ⬆️
#single 42.34% <ø> (ø) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update fcb8403...bc5d49e. Read the comment docs.

@datapythonista
Copy link
Member

Looks really good. There is just one thing that I'd consider changing. I don't think it's a good practice to import functions from scripts/. Whn in the future someone make changes to the scripts, I don't think we will feel that we need to keep compatibility on any function defined there, as they can be imported by other parts of the code like the doc.

What I'd do instead is creating somewhere a directory "dev utils", where we can not only have the code of build_string but also the generation of contributors (and anything that could potentially be reused by the docs, the release, and may be the web). And if possible, I'd also automatically generate there the list of releases (0.23.0, 0.23.1...) so releases.rst and anything that requires the list of releases can be generated with the info from there (releases.rst would be a template as index.rst in that case).

@pep8speaks
Copy link

pep8speaks commented Jun 25, 2018

Hello @TomAugspurger! Thanks for updating the PR.

Cheers ! There are no PEP8 issues in this Pull Request. 🍻

Comment last updated on June 26, 2018 at 12:00 Hours UTC

tags = [x.name for x in repo.tags if xpr.match(str(x))]
versions = sorted(zip(tags,
[tuple(map(int, xpr.match(x).groups())) for x in tags]),
key=lambda x: x[1])
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In case you don't know it, distutils provides a class to parse, sort... versions. You'd need to get rid of the heading v in v0.23.1. But then you can do this stuff automatically. For example:

>>> import distutils.version
>>> tags = ['0.22.0', '0.23.0', '0.23.1', '0.21.1', '0.21.0']
>>> versions = map(distutils.version.StrictVersion, tags)
>>> versions = sorted(versions)
>>> versions[-1].version
(0, 23, 1)

@TomAugspurger
Copy link
Contributor Author

Sorry I didn't mean to push those changes. Not sure whether it's worth auto-discovering.

@jorisvandenbossche
Copy link
Member

jorisvandenbossche commented Jun 25, 2018

Cool! Didn't have time to look into detail yet (will do that tomorrow), but some very quick feedback:

  • The docs are not yet building on travis: ModuleNotFoundError: No module named 'announce'
  • I get "/home/joris/scipy/pandas/doc/source/whatsnew/v0.20.3.txt: WARNING: document not readable. Ignored." for all whatsnew pages locally, but didn't further look into what is the cause (might be a problem on my side)
  • Big +1 on having the separate whatsnew files to separate html pages (already wanted to propose that for some time). However, the only problem is that this will break quite some links?
  • From a usability perspective of the "releases page", it would maybe be nice to include the highlights there, if this would be possible in some way.
  • Generating all the announce.build_string takes a non-negligible amount of time (when building the full docs probably not significant, but for a single page it might get annoying). Do we want to keep the script to generate it, but then actually include the content in the actual files (and use the script to generate it on release time) instead of generating it dynamically? (or otherwise skip it in certain cases, eg when building a single page that is not the releases pages)

@jorisvandenbossche
Copy link
Member

Looks really good. There is just one thing that I'd consider changing. I don't think it's a good practice to import functions from scripts/

+ 1. We can maybe also put things in or alongside doc/make.py ?

@TomAugspurger
Copy link
Contributor Author

TomAugspurger commented Jun 25, 2018

alongside doc/make.py

Sure.

this will break quite some links?

Which ones? Anything using the "canonical" format like /version/x.y.z/ should be fine, since we aren't changing those.

it would maybe be nice to include the highlights there, if this would be possible in some way.

Agreed. I think highlights.rst will be a stub or something that gets included in several places

Do we want to keep the script to generate it, but then actually include the content in the actual files (and use the script to generate it on release time) instead of generating it dynamically?

I would slightly prefer to generate it dynamically. My goal is to remove the "final doc update" commit that's currently part of the release process. In theory, it should be possible to make this a Jinja2 macro that is called inside the actual whatsnew, rather than ahead of time.

@TomAugspurger
Copy link
Contributor Author

TomAugspurger commented Jun 25, 2018 via email

@jorisvandenbossche
Copy link
Member

this will break quite some links?

Which ones? Anything using the "canonical" format like /version/x.y.z/ should be fine, since we aren't changing those.

Because it are now all separate pages, the url changes. For example, the first link in the highlights section of 0.23.0 points to http://pandas.pydata.org/pandas-docs/stable/whatsnew.html#whatsnew-0230-enhancements-round-trippable-json. However, with this change the link will end in .../whatsnew/v0.23.0.html#whatsnew-0230-enhancements-round-trippable-json

print(heading)
print(u"=" * len(heading))
print(author_msg % len(authors))
uline = '=' * len(heading)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this header does not always match the header levels in the whatsnew file

'v0.21.0': announce.build_string('v0.10.0..v0.21.0'),
'v0.21.1': announce.build_string('v0.10.0..v0.21.1'),
'v0.22.0': announce.build_string('v0.10.0..v0.22.0'),
'v0.22.0': announce.build_string('v0.22.0..v0.23.0'),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suppose you know you still have some typos here (a lot of "v0.10.0" above, and this line is duplicated)

@TomAugspurger
Copy link
Contributor Author

TomAugspurger commented Jun 26, 2018 via email

@TomAugspurger
Copy link
Contributor Author

Ok, cleaned up my mess from yesterday. Replaced with a custom directive so we write

.. contributors:: v0.23.0..v0.23.1

rather than building them up ahead of time.

I haven't updated all the old files yet.

@TomAugspurger
Copy link
Contributor Author

Will think about the links as well.

Writing the directive took a long time (I didn't realize I had to build the list of elements myself. I was hoping I could just pass it a string and have sphinx parse it), so I'm not sure I'll have time to finish this for 0.23.2.

@jorisvandenbossche
Copy link
Member

so I'm not sure I'll have time to finish this for 0.23.2.

no problem, we can still go the normal way for that

@TomAugspurger
Copy link
Contributor Author

Updated the headers.

Copy link
Contributor

@h-vetinari h-vetinari left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like you inadvertently edited the wrong file

v0.20.2 (June 4, 2017)
----------------------
What's new in v0.20.2 (June 4, 2017)
------------------------------------
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are you sure you wanted to do this in v.0.20.2 (which further increases the inconsistency) rather than v.0.24.0?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Alternatively, you could add "What's new in " to all versions (which would be even better than introducing it starting with 0.23.0)

@TomAugspurger
Copy link
Contributor Author

I think I needed to change the 0.23+ headers to get the levels consistent. Not super interested in changing the old ones.

Copy link
Contributor

@h-vetinari h-vetinari left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some tiny style comments, but nothing blocking. Also, you seem to have closed the PR by accident.

I think I needed to change the 0.23+ headers to get the levels consistent. Not super interested in changing the old ones.

Fair enough.

@@ -1,7 +1,9 @@
.. _whatsnew_0230:

v0.23.0 (May 15, 2018)
----------------------
What's new in 0.23.0 (May 15, 2018)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not a big deal, but "new" is capitalized for all the others. I'd go with lower-case for all.

upgrade instructions, see :ref:`install`.

0.24 release
------------
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it would look better (also looking at your render from the OP) to have

+ Version 0.24
- 0.24 release
  ------------

etc.

@TomAugspurger TomAugspurger reopened this Nov 13, 2018
@TomAugspurger
Copy link
Contributor Author

All green. Merging before we have more conflicts :)

I think this will require all outstanding PRs adding a release note to merge master. But contributors shouldn't have to manually fix any conflicts; git will handle things automatically.

@TomAugspurger TomAugspurger merged commit e413c49 into pandas-dev:master Nov 14, 2018
@TomAugspurger TomAugspurger deleted the doc-release-automation branch November 14, 2018 21:09
thoo added a commit to thoo/pandas that referenced this pull request Nov 15, 2018
* upstream/master: (25 commits)
  DOC: Delete trailing blank lines in docstrings. (pandas-dev#23651)
  DOC: Change release and whatsnew (pandas-dev#21599)
  DOC: Fix format of the See Also descriptions (pandas-dev#23654)
  DOC: update pandas.core.groupby.DataFrameGroupBy.resample docstring. (pandas-dev#20374)
  ENH: Allow export of mixed columns to Stata strl (pandas-dev#23692)
  CLN: Remove unnecessary code (pandas-dev#23696)
  Pin flake8-rst version (pandas-dev#23699)
  Implement _most_ of the EA interface for DTA/TDA (pandas-dev#23643)
  CI: raise clone depth limit on CI
  BUG: Fix Series/DataFrame.rank(pct=True) with more than 2**24 rows (pandas-dev#23688)
  REF: Move Excel names parameter handling to CSV (pandas-dev#23690)
  DOC: Accessing files from a S3 bucket. (pandas-dev#23639)
  Fix errorbar visualization (pandas-dev#23674)
  DOC: Surface / doc mangle_dupe_cols in read_excel (pandas-dev#23678)
  DOC: Update is_sparse docstring (pandas-dev#19983)
  BUG: Fix read_excel w/parse_cols & empty dataset (pandas-dev#23661)
  Add to_flat_index method to MultiIndex (pandas-dev#22866)
  CLN: Move to_excel to generic.py (pandas-dev#23656)
  TST: IntervalTree.get_loc_interval should return platform int (pandas-dev#23660)
  CI: Allow to compile docs with ipython 7.11 pandas-dev#22990 (pandas-dev#23655)
  ...
thoo added a commit to thoo/pandas that referenced this pull request Nov 15, 2018
…fixed

* upstream/master:
  DOC: Delete trailing blank lines in docstrings. (pandas-dev#23651)
  DOC: Change release and whatsnew (pandas-dev#21599)
  DOC: Fix format of the See Also descriptions (pandas-dev#23654)
  DOC: update pandas.core.groupby.DataFrameGroupBy.resample docstring. (pandas-dev#20374)
  ENH: Allow export of mixed columns to Stata strl (pandas-dev#23692)
  CLN: Remove unnecessary code (pandas-dev#23696)
  Pin flake8-rst version (pandas-dev#23699)
  Implement _most_ of the EA interface for DTA/TDA (pandas-dev#23643)
  CI: raise clone depth limit on CI
  BUG: Fix Series/DataFrame.rank(pct=True) with more than 2**24 rows (pandas-dev#23688)
  REF: Move Excel names parameter handling to CSV (pandas-dev#23690)
  DOC: Accessing files from a S3 bucket. (pandas-dev#23639)
  Fix errorbar visualization (pandas-dev#23674)
  DOC: Surface / doc mangle_dupe_cols in read_excel (pandas-dev#23678)
  DOC: Update is_sparse docstring (pandas-dev#19983)
  BUG: Fix read_excel w/parse_cols & empty dataset (pandas-dev#23661)
  Add to_flat_index method to MultiIndex (pandas-dev#22866)
  CLN: Move to_excel to generic.py (pandas-dev#23656)
  TST: IntervalTree.get_loc_interval should return platform int (pandas-dev#23660)
@jorisvandenbossche
Copy link
Member

@TomAugspurger the doc build is failing

Exception occurred:
  File "/home/travis/miniconda3/envs/pandas/lib/python3.6/site-packages/git/cmd.py", line 825, in execute
    raise GitCommandError(command, status, stderr_value, stdout_value)
git.exc.GitCommandError: Cmd('git') failed due to: exit code(128)
  cmdline: git shortlog -s v0.9.0..v0.10.0
  stderr: 'fatal: ambiguous argument 'v0.9.0..v0.10.0': unknown revision or path not in the working tree.

maybe because it has not a full clone?

@TomAugspurger
Copy link
Contributor Author

TomAugspurger commented Nov 15, 2018 via email

@jorisvandenbossche
Copy link
Member

@TomAugspurger Thanks!

It would still be nice to have it run at the CI as well, as I would imagine that at some point we would like to rely on some CI service to build the docs and upload that version, instead of having the release manager it to do locally?

@TomAugspurger
Copy link
Contributor Author

TomAugspurger commented Nov 15, 2018 via email

tm9k1 pushed a commit to tm9k1/pandas that referenced this pull request Nov 19, 2018
* DOC: Change release and whatsnew

Changes:

* We will include the correct (latest on master or maintenance branch)
  whatsnew directly in the index.rst toctree
* Contributors are included in the whatsnew for each version (automatically)
* Removed release.rst
* Added new releases.rst which has toctrees for each release

Incidental changes

* Updated style.ipynb. Writing the jinja template was confusing sphinx. We
  included it in the git source now.
* Fixing some inconsitent header levels (will do more)
* Refactored announce.py to support auto-generated contributors
Pingviinituutti pushed a commit to Pingviinituutti/pandas that referenced this pull request Feb 28, 2019
* DOC: Change release and whatsnew

Changes:

* We will include the correct (latest on master or maintenance branch)
  whatsnew directly in the index.rst toctree
* Contributors are included in the whatsnew for each version (automatically)
* Removed release.rst
* Added new releases.rst which has toctrees for each release

Incidental changes

* Updated style.ipynb. Writing the jinja template was confusing sphinx. We
  included it in the git source now.
* Fixing some inconsitent header levels (will do more)
* Refactored announce.py to support auto-generated contributors
Pingviinituutti pushed a commit to Pingviinituutti/pandas that referenced this pull request Feb 28, 2019
* DOC: Change release and whatsnew

Changes:

* We will include the correct (latest on master or maintenance branch)
  whatsnew directly in the index.rst toctree
* Contributors are included in the whatsnew for each version (automatically)
* Removed release.rst
* Added new releases.rst which has toctrees for each release

Incidental changes

* Updated style.ipynb. Writing the jinja template was confusing sphinx. We
  included it in the git source now.
* Fixing some inconsitent header levels (will do more)
* Refactored announce.py to support auto-generated contributors
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants