Unclear how to maintain parameter documentation

[kslong]

We went through an extensive effort to update the rearrange the parameter names in 2019, and we switched from a yml-based approach to a Sphinx based approach. Sam wrote one program which scraped the source files for names of inputs and to create yaml files that represented these inputs. As the names changed or items were updated; the first program autogenerate_parameter_docs.py, provided a clean way to see whether files had changed. The second program autogenerate_rtd_pages.py was used to create proto-rst files from the yaml files. It is very unclear how to maintain this, though as we go forward.

Ideally there would be a single program that function much as the orginal yml based program worked, but worked directly on the rtd files. But these are less structured than the yml files and its less clear that they can be parsed to see if updates are need. We are as a result, back to the situation where it is quite difficult to maintain the documentation, because we have no automatic way to check the documentation against the code, which was one of the advanatages of the yml method.

[saultyevil]

To see where we are right now for documentation, I used the scripts which we currently have to try and generate the HTML documentation.

To me it looks like both Python scripts are required to create the parameter documentation and requires that we have the yaml documentation - as the script which which generates the RST files and HTML documentation requires the yaml files. I wasn't able to get the documentation to build, but what I think should happen to is,

1. Run autogenerate_parameter_docs.py to update the .yaml files in docs/parameters. This can print to screen, as well write out, parameters which require to be updated
2. To generate the HTML documentation, run autogenerate_rtd_pages.py which first creates the RST files in docs/rst/parameters
3. After this, autogenerate_rtd_pages.py uses Sphinx to build the HTML documentation for the parameters and places them in docs/html

We could combine both of these scripts into one script, if we wish, so one only has to run one script to update the documentation. But I understand why we would prefer to only work on one set of files, rather than having files containing the same information, but in a different format.

There were two issues I faced trying build the documentation. The first is that when I tried to run autogenerate_parameter_docs.py on my MacBook (and on Linux) directly from my $PATH, i.e.,

saultyevil@ASTROBOOK:~$ autogenerate_parameter_docs.py

I was greeted with some long output of problems and errors, with some errors ImageMagick(?!). Running it instead as,

saultyevil@ASTROBOOK:~$ python ~/python/auto_generate_parameter_docs.py

works though. I tracked this down to the shebang (the #!/usr/bin/env python bit) being in the wrong place. It should be at the top of the file, instead of after the file doc string. Moving it fixes that issue so now it can be called from $PATH.

The second problem is that after I had generated new .yaml files, when I tried to create the RST files I got this error,

FileNotFoundError: [Errno 2] No such file or directory: '/Users/saultyevil/python/docs/rst/parameters/top_level_parameters.rst'

So I have a missing file, which I can't find on the online repository either. I'll have to spend some more time looking at the script to figure out what's what.

[saultyevil]

So the FileNotFoundError was created because I didn't have a directory docs/rst/parameters/. We can easily modify the script to create this directory if it is missing.

I'm not sure if autogenerate_rtd_pages.py was finished? The script ran to completion on my machine without any errors, but I don't have the HTML verion of the documentation.

[saultyevil]

I've spoken to Sam about this now. For documentation purposes, I'll summarise here:

1. The YAML files which are still in the repository should be removed (or at least archived somewhere). It's confusing that they still exist, even though they have been deprecated.
2. The two scripts can be salvaged, as the function which scrapes the source code for changes returns a dict and both the YAML and RST generation code use dicts as well. So it should not be too hard to modify either of them.
3. We can scrape the code parameters and check if it has an RST file, and warn about the parameter being undocumented if an RST does not exist.
4. I think if we keep the files fairly structured with headings, we could probably create some method to check for basic differences, i.e. input type or file.

I don't mind having a go at implementing some of these changes.

[kslong]

It would be great if you could take this on @saultyevil I had spent some time looking at this, and a difficulty I thought was going to be the very hierarchical way the rst files were structured. Note that we need a way also to flag if an input parameter has gone away. The yaml version of the code did this nicely, stuffing extras into a different directory.

I don't know if this is a good idea or not, but one possibility regarding point 4 above, would be to have a simple way to display side by side, a basic rst file created by the program compared to the one that is in the directory. If one could do that, a visual step through of the parameters might work. If one could step through this in an hour or so, then we would have a way to check where improvements to the documentation were needed.

[kslong]

I have labelled this high priority, because I think we need to get it completed while Sam remembers what he did. It's definitely need prior to the release.