Changes to @earth_day and @earth_night grids from GMT 6.4 to 6.5? #257
Comments
|
So here is what happened last September. We realised our spherical filtering was incorrect with a too-short filter radius. All data sets were reprocessed with the updated gmtserver-admin scripts. For the NASA images it is true they were indexed images (at least the full resolution 30s still is I think since no filtering done there). From memory (this is years ago) I think I just used photoshop to downsample the full file to get the lower resolutions using some semi-reasonable filter width. However, with the September work we had to do actual filtering and one cannot filter indices, hence we used grdmix to pull out red, green, and blue grids, filtered these, then grdmix put these filtered r,g,b back into one image, which now was RGB format. GMT has no ability to convert that to an indexed image, but probably GDAL does. We never did but we could discuss if that is important. Biggest problem with the images is we have not tiled them like we do for the grids. Hoping @joa-quim can provide some suggestions here. I think GMT 6.4 and earlier can plot RGB images as well as indexed images (since our GDAL bridge returns index images as RGB anyway). |
Just to clarify that to access these datasets, you need |
|
Paul, you used a color quantization option to go from RGB to indexed from a GDAL module that I forgot the name |
|
Thinking it must be gdal_translate but nothing obvious stand out. One argument for quantising to 256 unique colors via an index table is that our day/night images are not tiled so to use the earth_day_30s.tif means it is a singe file of 622 Mb. In absence of tiling we could probably get that down to 1/3 with indexing. But, what is the amount of work needed to enable gmt_gdalwrite.c learn a new trick to do this for us? I guess the quantising is tricky. How would we do it? Compute difference between all pixels in RGB space and pick the 256 points the have the highest sum of distances between them? So 255 244 239 will be selected as the final RGB index entry if other nearby colors are relatively near? Perhaps better to find what the GDAL option you refer to is.. Reading in index images is way more useful (and we do) than writing them out, which is encroaching on image processing stuff. |
|
Pretty sure I used PhotoShop who as quantification/index option. However, only lets me save as PNG, not TIFF, and I am pretty sure we expect the da/night images to be TIFFs from GMT's point of view. Perhaps need to convert the PNG index image to a TIFF index image with GDAL. |
|
Experiment: The 01m day tiff (RGB) is 171M but a photoshop -> index PNG is 53M and Preview can save this to index TIF at 47 Mb. I can plot the whole thing with gmt grdimage earth_day_01m_indexed.tiff -Dr -Rd -JQ20c -B -pdf map since the -180/180/-90/90 got lost probably in the commercial tools. Does reduce transmission size/time by 3 though. @joa-quim, if you can tell me how to add back in the reference (-Rd) to that indexed tiff file than we can revert the day/night filtered RGB images to indexed TIFF for once via PhotoShop->Preview->add-CRS-> indexes final file 1/3 the size and works in GMT. |
|
Looks like a fun research problem to apply the k-means statistics to determine the 256 best colors but how often is this needed. I think it is tolerable to do a bi manual labor on those images to shrink and speed them up via indices, no? What do @weiji14 say? |
|
I only find this now (a python script) I have this implemented in Mirone. Probably from an old MEX But I don't think it worth's spending time on this. |
Ah I see, thanks @seisman for clarifying! Somehow it worked without Personally I'd prefer the non-quantized version of the GeoTIFF right now (even though they are bigger) instead of the quantized/indexed version. Nowadays with GeoTIFFs, GDAL should be possible to read spatial subsets without downloading the whole file, as long as the data is arranged as square tiles internally. Could we convert the GeoTIFFs to Cloud-Optimized GeoTIFFs, e.g. using GDAL's COG driver? |
Downloading @earth_day_01d_p using GMT 6.4.0 actually works. The new earth_day_01d_p.tif file processed on 2023-09-29 doesn't have the quantized colormap, so no need for the extra handling. See GenericMappingTools/gmtserver-admin#257 for more info.
…2963) Rioxarray can now read the 3-bands from `@earth_day_01d_p` directly without needing to parse the colorinterp information from the GeoTIFF file. * Update baseline images for test_tilemap_* with GMT 6.5.0 and gs 10.02.1 * Trigger re-caching of earth_day_01d_p file Modify pygmt/helpers/caching.py file slightly to create new cache. * Remove elif block for manually creating RGB image Downloading @earth_day_01d_p using GMT 6.4.0 actually works. The new earth_day_01d_p.tif file processed on 2023-09-29 doesn't have the quantized colormap, so no need for the extra handling. See GenericMappingTools/gmtserver-admin#257 for more info.
Just checking something that I noticed at GenericMappingTools/pygmt#2963 (comment). Are the Blue Marble
@earth_day_01d_pand Black Marble@earth_night_01d_pdatasets only downloadable with GMT 6.5.0 (and not GMT 6.4.0 or older) now?E.g. doing
gmt which @earth_day_01d_p -Vdwith GMT 6.4.0 gives:But with GMT 6.5.0,
gmt which @earth_day_01d_pworks fine.Also, I noticed that the internal data structure of the Blue/Black Marble images have changed from a 1-band ColorInterp/Color Table structure to a 3-band structure:
Looking at https://oceania.generic-mapping-tools.org/gmt_data_server.txt, the modification date was 2023-09-29, and seems to be applied from 5bcc3d6.
Just checking if these changes are expected. Personally, I prefer the new 3-band structure since it will simplify things in PyGMT (GenericMappingTools/pygmt#1442 (comment)), but wanted to know if breaking the downloads for GMT 6.4.0 are ok.
The text was updated successfully, but these errors were encountered: