Add option to plot utilities to generate PNG output rather than inline plots #42
Conversation
First pass at generating image files (and a JSON file with associated metadata) rather than including images inline in the notebook. Additional commits on this branch will address different types of plots.
savefig() is part of the parent class, get_filename() is part of the child class. This current maintains the current API for summary_plots, but next commit will try to make that API more extensible. Still need to get casename into the SummaryMapClass to use in file name.
Use **plot_options for arguments beyond da and diag_metadata. Also updated get_filepaths() to return both the png path and the json path (this let me put output in images/summary_maps/CASENAME/... instead of requiring images/CASENAME to be the root)
Note that this doesn't update the notebooks yet, it's just the utils package
Note that I noticed a pretty big bug to fix in the next commit -- I'm not accounting for isel_dict in the file name, so plotting the same variable at two vertical levels will create a plot at the first level and then over-write it with the plot at the second level. I'll fix that tomorrow.
E.g. SiO3.0095-01-16_0104-12-16.z_t--500.00.png SiO3.0095-01-16_0104-12-16.z_t--35109.35.png
The issue only affected time series, and was due to a bad argument in self.get_timeseries_files() [wanted files for a specific variable, but was getting file names for ALL variables]
Added two new child classes of _PlotTypeBaseClass for trend maps and trend histograms; modified the two trend_maps notebooks to write files instead of generating plots.
|
I'm going to turn off draft status because at this point it looks
Still, I think there are several open questions we should discuss before merging:
|
|
@andersy005 -- I should have mentioned this in the previous comment, but png files are in |
|
Also on the to-do list: I need to add an argument to allow control over the figure size / resolution when writing to disk. |
|
I suggest that the That way, calls to these methods, that should never be called, will actually get to the As is, a user of a derived class that doesn't implement these methods will get errors about unexpected keyword arguments or too many positional arguments when the |
1. Address Keith's comment about *args and **kwargs 2. Use time_bounds to determine interval for histograms and time series 3. Reorganize output so that case name is the top-level directory rather than plot type -- this will get clunky when comparing multiple cases, but it lines up better with the directory structure of webext.cgd.ucar.edu. When we have plots comparing multiple cases, we can figure out a good naming scheme 4. generate_plot_catalog() can print relative paths (and the filepath in the json file is relative to the casename rather than the notebooks directory)
|
eefc145 introduces a few updates that are mostly related: The first bullet point is just some low-hanging fruit; the rest ties into making data available for the dashboard. I've started posting some PNG files (and associated CSV catalogs) on webext; so now we can inline images directly from that server: Also, @mgrover1 pointed out that we should use I'm waiting for one more notebook to finish running, then I'll make a commit with those updates. edit: I guess the dates don't actually appear in the plot; they're in the file name ( edit2: inline image is now a link to original (I think github cached a version when I originally posted) |
These notebooks put images in {casename}/{plottype}/ rather than
{plottype}/{casename}/; there is also a gen_csv.ipynb notebook that creates csv
catalogs for each case (the catalog is {casename}/png_catalog.csv)
|
I noticed that |
|
From chatting with @mgrover1, it looks like we pull useful information from |
For this commit, I only pass savefig_kwargs to summary_plot_histogram() and
summary_plot_global_ts(); for the plot_suite_{003,004,1deg}.ipynb files, I use
kwargs to set dpi=72 (which matches the default value). The next commit(s)
should update the rest of the plotting functions to also use
plot_options['savefig_kwargs'].
A few other small changes snuck in with this commit
1. run_notebooks.sh uses PBS rather than SLURM
2. the location of the zonally-averaged output was moved to campaign
2b. the old datasets were missing year 2 from 003, so I regenerated all the za
output files
-- I did this based on time series rather than history files because that
is what we have on disk
-- I used a slightly updated region mask that doesn't seem to have changed
the plots but did introduce O(1e-4) relative error in the Atlantic and
Pacific basins (I believe it moved some grid cells from the Atlantic to
the Pacific while leaving the global and Indian basins untouched)
I replaced metadata['isel_dict'] with metadata['reduced_dims'] and instead of including the index of the dim reduction, it lists the actual value (e.g. instead of basins=0, it contains basins="Global") I had also switched from ncar-jobqueue to dask-jobqueue in a previous commit, but with ncar-jobqueue now recognizing the PBS nodes on casper I was able to switch back.
I regenerated the csv files for each case, and this notebook reflects the change from isel_dict -> reduced_dims
|
@mgrover1 and @klindsay28 -- this is ready for review. @mgrover1 -- I've moved the latest output to webext, hopefully the dashboard reflects the change from @klindsay28 -- for now, I've left I believe all the other comments mentioned in meetings / this PR thread have been addressed. |
1. use "sel_dict" instead of "reduced_dims" for name of dictionary in metadata 2. no longer pass casename = ds.attrs["title"] to plotting functions; all functions receive ds, and can determine casename itself (updated trend_map notebooks to copy time series attrs to the zonally-averaged datasets) 3. use ds[ds["time"].attrs["bounds"]] instead of ds["time_bound"] or ds.time_bound 4. clean up gen_csv.ipynb: * use relative path for finding pngs / reading metadata * use relative path for writing csv file * don't explicitly skip trend_map and trend_hist for the 1 degree run; * generate_plot_catalog() can handle non-existent directories 5. Add png_catalog.csv to the repository
This PR will generate images and metadata for @andersy005's dashboard. The first commit produces a separate JSON file for each generated plot, but @andersy005 will follow behind with a tool to clean that all up :) Additional commits are needed to generate PNG files from other plot types (time series, etc).