Docpipeline tests by chapter and section

[mmatera]

This continues #1021, and starts to tackle the problem of accessing tests by chapter and section without interacting over the full documentation.

For chapters, the change is quite straightforward. On the other hand, for sections, I think we need to think a little bit what is the best way to do this.

One possibility would be to produce the documentation entry from the docstring of the target builtin, without passing through the documentation. The other way to go would be to keep a dictionary in the documentation, storing all the sections "by slug".

Thoughts?

[mmatera]

Now, this PR was updated with part of #1025, but produces the documentation following the pattern closer to Mathics 6.0

[mmatera]

This is how Graph looks in the Pymathics section part:

[mmatera]

Now the subsections have numbers:

What I still do not like about this is that Symbol references look different for symbols inside packages and symbols in shallow modules:

In any case, one possible fix would be to put the symbols in shallow modules inside a "default" guide.

[rocky]

This is how Graph looks in the Pymathics section part:

The numbers go back to 1. It would be much better instead to continue from the last previous chapter number. In particular the last chapter was 52. Units and Quantities. So Graphs should be chapter 53.

(It looks like some fragile pytests that refer to subsection* need to be adjusted to subsection without the star.)

[rocky]

For chapters, the change is quite straightforward. On the other hand, for sections, I think we need to think a little bit what is the best way to do this.

One possibility would be to produce the documentation entry from the docstring of the target builtin, without passing through the documentation. The other way to go would be to keep a dictionary in the documentation, storing all the sections "by slug".

Thoughts?

by slug seems fine.

[rocky]

What I still do not like about this is that Symbol references look different for symbols inside packages and symbols in shallow modules:

The images and looking at the PDF don't bother me in the slightest. Eventually, I imagine everything will be under some guide section of some sort. So this kind if thing is temporary.

[rocky]

@mmatera with this organization, can we now redo the code in Mathics3 Modules so that they are now organized more like the way builtins work? Or does more need to be done?

[mmatera]

@mmatera with this organization, can we now redo the code in Mathics3 Modules so that they are now organized more like the way builtins work? Or does more need to be done?

With these changes, docpipeline works in pymathics modules without making any changes. Also, each module is treated exactly as it were part of mathics.builtin (except that the results are associated to the Modules part)

[mmatera]

This is how Graph looks in the Pymathics section part:

The numbers go back to 1. It would be much better instead to continue from the last previous chapter number. In particular the last chapter was 52. Units and Quantities. So Graphs should be chapter 53.

The problem is that we put modules in a different part. There is a way in LaTeX to set the initial chapter number in a Part by hand, but at this point I guess would be a higher-order correction.

(It looks like some fragile pytests that refer to subsection* need to be adjusted to subsection without the star.)

Yes, I realized that after pushing it up.

[rocky]

Things are looking better. I am seeing that guide sections do not have labels associated with them, so I get

  Reference `reference-of-built-in-symbols/graphics-and-drawing/drawing-options-and-option-values' on page 231 undefined on input line 24606
  Reference `reference-of-built-in-symbols/graphics-and-drawing/drawing-options-and-option-values' on page 240 undefined on input line 41322

when I build the docs.

I suspect this is a two line change in latex_doc.py around line 800.

[mmatera]

Things are looking better. I am seeing that guide sections do not have labels associated with them, so I get

  Reference `reference-of-built-in-symbols/graphics-and-drawing/drawing-options-and-option-values' on page 231 undefined on input line 24606
  Reference `reference-of-built-in-symbols/graphics-and-drawing/drawing-options-and-option-values' on page 240 undefined on input line 41322

when I build the docs.

I suspect this is a two line change in latex_doc.py around line 800.

Yes, I see this kind of things related to how we handle keys for sections and subsections.

[rocky]

One other thing I noticed is that in the manual chapters are sorted and we don't want that. That's why in previous versions and drafts there was a parameter when gathering chapters. That should be put back in.

[rocky]

Things are looking better. I am seeing that guide sections do not have labels associated with them, so I get

  Reference `reference-of-built-in-symbols/graphics-and-drawing/drawing-options-and-option-values' on page 231 undefined on input line 24606
  Reference `reference-of-built-in-symbols/graphics-and-drawing/drawing-options-and-option-values' on page 240 undefined on input line 41322

when I build the docs.
I suspect this is a two line change in latex_doc.py around line 800.

Yes, I see this kind of things related to how we handle keys for sections and subsections.

Seems like an easy fix though.

[mmatera]

Things are looking better. I am seeing that guide sections do not have labels associated with them, so I get

  Reference `reference-of-built-in-symbols/graphics-and-drawing/drawing-options-and-option-values' on page 231 undefined on input line 24606
  Reference `reference-of-built-in-symbols/graphics-and-drawing/drawing-options-and-option-values' on page 240 undefined on input line 41322

when I build the docs.
I suspect this is a two line change in latex_doc.py around line 800.

Yes, I see this kind of things related to how we handle keys for sections and subsections.

Seems like an easy fix though.

This was just that GuideSections lost their labels. Now it seems to work.

[rocky]

Ok - everything looks good now. Thanks!

Some follow-on items (in a future PR?) are:

1. Don't reset numbers between parts.
2. Rewrite Mathics3 Modules graph and natural language to be organized the way builtins are. We should have done or should be doing this before we recommend that others use this to replace parts of the existing builtin. (There code may be additional scope issues in Mathics3 Modules, but that's another topic for another time.)

[rocky]

Something on my todo list before the release is to go over and read the entire doc more carefully. I see on page 29

there is a weird line break. And we have LaTeX format errors:

  Missing character: There is no ‐ ("2010) in font pplr8t!
  Missing character: There is no ‐ ("2010) in font pplr8t!

But more generally, there was some basic text that I had meant to go over. It was a while ago so I don't remember the details there and didn't write it down at the time.

[mmatera]

Ok - everything looks good now. Thanks!

Some follow-on items (in a future PR?) are:

1. Don't reset numbers between parts.

I found that commenting out a line in mathics.tex recovers this behavior. It is done now.

2. Rewrite Mathics3 Modules graph and natural language to be organized the way builtins are. We should have  done or should be doing this _before_ we recommend that _others_  use this to replace parts of the existing builtin. (There code may be additional scope issues in Mathics3 Modules, but that's another topic for another time.)

As far as I can see, pymathics.graphics has the same structure that mathics.builtin.arithmetic, and it is shown in a similar way. What is probably missing is more detailed content.

[mmatera]

Something on my todo list before the release is to go over and read the entire doc more carefully. I see on page 29

there is a weird line break.

I think this is an issue of the mathics.tex template.

And we have LaTeX format errors:

  Missing character: There is no ‐ ("2010) in font pplr8t!
  Missing character: There is no ‐ ("2010) in font pplr8t!

But more generally, there was some basic text that I had meant to go over. It was a while ago so I don't remember the details there and didn't write it down at the time.

In any case, these issues are not related to this PR, and can be corrected in another round.

[rocky]

Thanks for not resetting the chapter number between parts. Let's merge and pick up remaining things in future PRs.

[mmatera]

Thanks for not resetting the chapter number between parts. Let's merge and pick up remaining things in future PRs.

Thank you for the revision and feedback!
Actually the issue with the chapter numbers was easy to solve because it was something that we had added as a hack in the template.