[WIP] adding a CLI to generate books #89

choldgraf · 2019-01-23T22:29:56Z

This is a CLI to generate books, riffing off of #87 here's the help section:

Creating new books:

jupyter-book create [--out-folder OUT_FOLDER] [--license LICENSE]
                    [--content-folder CONTENT_FOLDER] [--toc TOC]
                    [--config CONFIG] [--custom-css CUSTOM_CSS]
                    [--custom-js CUSTOM_JS] [--overwrite] [--demo]
                    name

### Create a new Jupyter Book

positional arguments:
  name                  The name of your Jupyter Book (your book template will
                        be placed in a folder of this name)

optional arguments:
  -h, --help            show this help message and exit
  --out-folder OUT_FOLDER
                        The location where your book will be placed
  --license LICENSE     A path to a LICENSE.md file if you have already
                        created one
  --content-folder CONTENT_FOLDER
                        A path to a folder that holds your book content
  --toc TOC             A path to a yaml file that contains a Table of
                        Contents for your Jupyter Book. This will overwrite
                        parts of the book template's default toc.yml
                        configuration
  --config CONFIG       A path to a configuration YAML file that contains
                        configuration for your Jupyter Book. This will
                        overwrite parts of the book template's default
                        _config.yml configuration
  --custom-css CUSTOM_CSS
                        A path to a CSS file that defines some custom CSS
                        rules for your book
  --custom-js CUSTOM_JS
                        A path to a JS file that defines some custom CSS rules
                        for your book
  --overwrite           Whether to overwrite a pre-existing book if it exists
  --demo                Whether to build the book with demo content instead of
                        your own content

Building book markdown:

jupyter-book build [--path-book PATH_BOOK]
                    [--path-template PATH_TEMPLATE]
                    [--path-config PATH_CONFIG] [--path-toc PATH_TOC]
                    [--overwrite] [--execute] [--local-build]

Convert a collection of Jupyter Notebooks into Jekyll markdown suitable for a
course textbook.

optional arguments:
  -h, --help            show this help message and exit
  --path-book PATH_BOOK
                        Path to the root of the textbook repository.
  --path-template PATH_TEMPLATE
                        Path to the template nbconvert uses to build markdown
                        files
  --path-config PATH_CONFIG
                        Path to the Jekyll configuration file
  --path-toc PATH_TOC   Path to the Table of Contents YAML file
  --overwrite           Overwrite md files if they already exist.
  --execute             Execute notebooks before converting to MD.
  --local-build         Specify you are building site locally for later
                        upload

If you don't give a --content-folder or --toc parameter, then it'll ask you to manually input the path to one per @arokem's suggestions.

Upgrade the book

This will create a copy of the book with the latest jupyter-book
template, then copy over the new files into your current book, preserving
the files that you've modified on your own.

jupyter-book upgrade path_book

Upgrade a book to the latest Jupyter Book version.

positional arguments:
  path_book   Path to the root of the book repository you'd like to upgrade.

To do

Write tests
Finalize CLI API
Finalize interactive inputs
Figure out the best way to auto-generate a toc.yml file if none is given (e.g. should it just be a flat list of pages, or should we try to infer page nesting etc)

To demo the CLI

I have a demo of the CLI running on Binder here:

https://mybinder.org/v2/gh/choldgraf/jupyter-book/cli?urlpath=terminals/1

that'll open a terminal - you should then be able to type

jupyter-book create <arbitraryname> --demo

and it'll create a jupyter book in <arbitraryname> with the "demo" content (which is the content from the website).

you should then be able to run:

jupyter-book build --path-book <arbitraryname>

and it'll build the markdown for the book.

Comments?

What do folks think about this general approach? Thoughts ideas etc would be appreciated! cc also @yuvipanda !

choldgraf · 2019-01-25T23:09:59Z

Anybody know what is the best way to ensure that a folder is copied over when you pip install something? I need to install the CLI which is in 'jupyter-book' but also have access to the template in 'book_template'

arokem · 2019-01-25T23:13:45Z

I think package_data would do that. See https://github.com/uwescience/shablona/blob/master/setup.py

…

On Fri, Jan 25, 2019 at 3:09 PM Chris Holdgraf ***@***.***> wrote: Anybody know what is the best way to ensure that a folder is copied over when you pip install something? I need to install the CLI which is in 'jupyter-book' but also have access to the template in 'book_template' — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#89 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAHPNulVlEPrBqck_7O43sXOTrb-yi0tks5vG47HgaJpZM4aPwlg> .

choldgraf · 2019-01-26T02:34:46Z

hmm, I'm not sure how this can include a whole folder of data files. Here's my conundrum:

The project structure in this PR looks like this:

./
├── book_template
├── jupyter_book
├── requirements.txt
└── setup.py

If I install with pip, then this goes to .../site-packages/jupyter_book/<contents-of-jupyter_book-folder>.

What I'd actually like to happen is that either:

.../site-packages/book_template/

would also exist, or

.../site-packages/jupyter_book/jupyter_book
.../site-packages/jupyter_book/book_template

would exist.

Any ideas on how that could happen?

arokem · 2019-01-26T03:19:11Z

glob(“dir/*”)

…

On Fri, Jan 25, 2019 at 6:34 PM Chris Holdgraf ***@***.***> wrote: hmm, I'm not sure how this can include a whole *folder* of data files. Here's my conundrum: The project structure in this PR looks like this: ./ ├── book_template ├── jupyter_book ├── requirements.txt └── setup.py If I install with pip, then this goes to .../site-packages/jupyter_book/<contents-of-jupyter_book-folder>. What I'd actually like to happen is that either: .../site-packages/book_template/ would also exist, or .../site-packages/jupyter_book/jupyter_book .../site-packages/jupyter_book/book_template would exist. Any ideas on how that could happen? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#89 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAHPNoD2-r7ZoikPKjpGFipo55DyopW-ks5vG77GgaJpZM4aPwlg> .

choldgraf · 2019-01-26T04:06:24Z

so that's basically what I'm doing:

https://github.com/jupyter/jupyter-book/pull/89/files#diff-2eeaed663bd0d25b7e608891384b7298R8

then

https://github.com/jupyter/jupyter-book/pull/89/files#diff-2eeaed663bd0d25b7e608891384b7298R32

Shouldn't running pip install . then deposit everything in book_template into the site-packages/jupyter-book folder?

arokem · 2019-01-26T10:18:40Z

setup.py

+import os.path as op
+from glob import glob
+
+template_files = glob(op.join('jupyter_book', 'book_template', '**', '*'), recursive=True)


Suggested change

template_files = glob(op.join('jupyter_book', 'book_template', '**', '*'), recursive=True)

template_files = glob(op.join('book_template', '**', '*'), recursive=True)

Following the shablona example, I think that this might need to be a path relative to the package.

Alternatively, if that doesn't work:

Suggested change

template_files = glob(op.join('jupyter_book', 'book_template', '**', '*'), recursive=True)

template_files = [op.join('book_template', '*'), op.join('book_template', '*', '*')]

And so on, for as many levels of recursion as you need.

choldgraf · 2019-01-26T17:16:17Z

ahh I think I figured it out - there was a gotcha in there. setup.py does indeed want paths relative to the package (in this case jupyter_book) however since I was using glob( to list those paths, I still needed to use the "correct" relative paths first. So I have to glob the files, then remove the top-level folder from each resulting path. Intuitive!

choldgraf · 2019-01-26T17:26:46Z

OK I have a demo of the CLI running on Binder here:

https://mybinder.org/v2/gh/choldgraf/jupyter-book/cli?urlpath=terminals/2

that'll open JupyterLab - you should be able to then open a terminal and type

jupyter-book create <arbitraryname> --demo

and it'll create a jupyter book in <arbitraryname> with the "demo" content (which is the content from the website).

you should then be able to run:

jupyter-book build --path-book <arbitraryname>

and it'll build the markdown for the book.

If anybody wants to give it a shot and give feedback on the API etc, that'd be helpful!

(and thanks @arokem for the help on the package_data!)

gnestor · 2019-01-29T05:42:29Z

I just tried it out and it works great!

I was going to suggest removing the --path-book argument from jupyter-book build and make the path the default argument but it looks like the API has changed and that is now the case 👍

The only issue is that I'm unable to serve the book on binder:

jovyan@jupyter-choldgraf-2djupyter-2dbook-2dmt96jhm3:~/my-book$ make serve
bundle exec guard
make: bundle: Command not found
Makefile:27: recipe for target 'serve' failed
make: *** [serve] Error 127

jovyan@jupyter-choldgraf-2djupyter-2dbook-2dmt96jhm3:~/my-book$ make install
gem install bundler
make: gem: Command not found
Makefile:14: recipe for target 'install' failed
make: *** [install] Error 127

choldgraf · 2019-01-29T21:50:04Z

Woot! thanks for the feedback - the latest push adds a binder/ folder that uses a conda environment with Ruby installed (turns out you can install Ruby with conda-forge on *nix). That should work (though I dunno if the port opening will work within Binder).

Any other thoughts on the CLI itself? Seems like a reasonable choice to create books / update old books / etc?

Edit: hmmm I take that back, it doesn't seem like Binder can build the gems needed because a file isn't there that it expects to be there:

/srv/conda/bin/ruby -r ./siteconf20190129-373-di8g14.rb extconf.rb
creating Makefile

current directory: /home/jovyan/.gem/ruby/2.4.0/gems/commonmarker-0.18.2/ext/commonmarker
make "DESTDIR=" clean

current directory: /home/jovyan/.gem/ruby/2.4.0/gems/commonmarker-0.18.2/ext/commonmarker
make "DESTDIR="
compiling arena.c
make: /home/conda/feedstock_root/build_artifacts/ruby_1537084490905/_build_env/bin/x86_64-conda_cos6-linux-gnu-cc: Commandnot found

gnestor · 2019-01-29T23:14:00Z

I think the interface is great 👍

kysolvik · 2019-02-01T21:14:07Z

I think your suggestion sounds good. If you wanted to keep the prompts, you could set the default for content as "None" or something similar, with a message that if they don't specify a content dir it creates an empty/simple one. But maybe more trouble than it's worth vs. just removing the prompts.

Originally I was thinking of an optional flag similar to --demo, like "--blank" or "--empty"

choldgraf · 2019-02-02T02:16:44Z

OK the latest push removes the interactive prompts for book content. If someone just types:

jupyter-book create mybook

then it'll simply add a bare-bones book w/ just a few pages of content to get you started (and adds this to the notice at the end).

If you add content with the --content flag, then it'll ask you if you want to create a table of contents file if you haven't specified one.

@kysolvik wanna give that a shot? seem more intuitive? less?

lheagy · 2019-02-07T17:16:52Z

Hi Chris, this is looking good! One question about consistency: in the book setup the flags for the path to the toc, config, etc are just --toc, --config, whereas for the build, these are now pre-pended with path, e.g. --path-toc, --path-config. Is this distinction intentional? or can we update to make them consistent? (I would vote for the less-verbose version --toc, --config unless others have strong opinions)

choldgraf · 2019-02-07T19:49:39Z

@lheagy I think that's a great point, will take a pass to standardize these flags

choldgraf · 2019-02-07T22:35:27Z

ok, CLI should now be more consistent. Thanks for all of your advice on this one @arokem @gnestor @kysolvik @yuvipanda and @lheagy - if you have any other thoughts please do share them! Otherwise I'll plan on merging this ~tomorrow after going through the docs once more

…ly empty minimal repo

choldgraf · 2019-02-09T00:16:49Z

ok! I took another pass at docs and updated the TOC part of the CLI, and magically the tests are still passing, and so I am merging this!! Let's goooooo 🎉

choldgraf mentioned this pull request Jan 23, 2019

Explore using a cookiecutter for the site #87

Closed

choldgraf force-pushed the cli branch 5 times, most recently from 1bc35ac to 355185f Compare January 25, 2019 19:02

choldgraf force-pushed the cli branch from 5072a56 to 5dcdc20 Compare January 26, 2019 04:06

arokem reviewed Jan 26, 2019

View reviewed changes

choldgraf force-pushed the cli branch 2 times, most recently from 3b7b031 to 0ad9a6e Compare January 26, 2019 16:50

choldgraf force-pushed the cli branch from 0ad9a6e to d22cd37 Compare January 26, 2019 17:22

choldgraf force-pushed the cli branch 4 times, most recently from 7467d89 to abda155 Compare January 29, 2019 00:59

choldgraf force-pushed the cli branch from d1a482e to 4f711be Compare January 29, 2019 21:51

choldgraf force-pushed the cli branch 3 times, most recently from c98b1a2 to e60eb8c Compare January 31, 2019 00:18

choldgraf mentioned this pull request Jan 31, 2019

Merged search functionality from jupyter/jupyter-book pyOpenSci/software-peer-review#5

Merged

choldgraf force-pushed the cli branch from 7db97e3 to a368f78 Compare February 2, 2019 02:15

choldgraf mentioned this pull request Feb 5, 2019

Adding Optional Buttons, Including Download Button #38

Closed

choldgraf force-pushed the cli branch from 3c65f5a to dabb6a9 Compare February 5, 2019 22:33

choldgraf mentioned this pull request Feb 8, 2019

Disappearing None #98

Closed

choldgraf added 17 commits February 8, 2019 14:46

adding a CLI

3fff5f8

adding tests and trying to get circle to work

80c7b4e

adding build to CLI and fixing circle

7bacfe5

moving template inside package and adding data files

eedbcb9

adding ability to run from the CLI

26b6a96

adding a binder folder

7356ed7

adding upgrade command and test

1821867

adding back in a test after rebse

affba5e

adding documentation for the CLI and updating CSS

273f141

copy button in all code cells except outputs

bc5c71a

simplifying toc

a476afa

scrollbar styles

1b17cb3

adding ability to add extra files and custom js and css tests

9625e51

removing content-based interactive input and instead using a relative…

8c1007f

…ly empty minimal repo

removing interactive prompts and improving ending messages

390d6c2

updating instructions

275f78d

updating CLI for build

0059ba9

choldgraf force-pushed the cli branch from cf1b384 to 0059ba9 Compare February 8, 2019 22:46

updating documentation and adding TOC building

f8a02a2

choldgraf merged commit 43874c9 into jupyter-book:master Feb 9, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] adding a CLI to generate books #89

[WIP] adding a CLI to generate books #89

choldgraf commented Jan 23, 2019 •

edited

Loading

choldgraf commented Jan 25, 2019

arokem commented Jan 25, 2019 via email

choldgraf commented Jan 26, 2019

arokem commented Jan 26, 2019 via email

choldgraf commented Jan 26, 2019

arokem Jan 26, 2019

arokem Jan 26, 2019

choldgraf commented Jan 26, 2019

choldgraf commented Jan 26, 2019 •

edited

Loading

gnestor commented Jan 29, 2019

choldgraf commented Jan 29, 2019 •

edited

Loading

gnestor commented Jan 29, 2019

kysolvik commented Feb 1, 2019

choldgraf commented Feb 2, 2019 •

edited

Loading

lheagy commented Feb 7, 2019

choldgraf commented Feb 7, 2019

choldgraf commented Feb 7, 2019

choldgraf commented Feb 9, 2019

	template_files = glob(op.join('jupyter_book', 'book_template', '*', ''), recursive=True)
	template_files = glob(op.join('book_template', '*', ''), recursive=True)

	template_files = glob(op.join('jupyter_book', 'book_template', '*', ''), recursive=True)
	template_files = [op.join('book_template', ''), op.join('book_template', '', '*')]

[WIP] adding a CLI to generate books #89

[WIP] adding a CLI to generate books #89

Conversation

choldgraf commented Jan 23, 2019 • edited Loading

Building book markdown:

Upgrade the book

To do

To demo the CLI

Comments?

choldgraf commented Jan 25, 2019

arokem commented Jan 25, 2019 via email

choldgraf commented Jan 26, 2019

arokem commented Jan 26, 2019 via email

choldgraf commented Jan 26, 2019

arokem Jan 26, 2019

Choose a reason for hiding this comment

arokem Jan 26, 2019

Choose a reason for hiding this comment

choldgraf commented Jan 26, 2019

choldgraf commented Jan 26, 2019 • edited Loading

gnestor commented Jan 29, 2019

choldgraf commented Jan 29, 2019 • edited Loading

gnestor commented Jan 29, 2019

kysolvik commented Feb 1, 2019

choldgraf commented Feb 2, 2019 • edited Loading

lheagy commented Feb 7, 2019

choldgraf commented Feb 7, 2019

choldgraf commented Feb 7, 2019

choldgraf commented Feb 9, 2019

choldgraf commented Jan 23, 2019 •

edited

Loading

choldgraf commented Jan 26, 2019 •

edited

Loading

choldgraf commented Jan 29, 2019 •

edited

Loading

choldgraf commented Feb 2, 2019 •

edited

Loading