Titiler-cmr layer #1001

abarciauskas-bgse · 2024-06-12T20:33:08Z

Related Ticket: #1000

Description of Changes

Application code changes - necessary changes

renames ZarrPaintLayer to RasterPaintLayer, since it is not specific to Zarr
adds a new type titiler-cmr and renames the cmr type to cmr-stac for disambiguation (although it is sure to be a bit confusing, since cmr-stac is for the use case of using the CMR STAC catalog and titiler-cmr is for the use case of tiling data in CMR but those requests are setup via collection records catalogued in VEDA STAC)
adds a third hook useTitilerCMR. Because this third hook needs to return tiling parameters besides assetUrl, refactor the other hooks to also return a tileParams dictionary.

Application code - optional changes, simplifying some things

Merges/removes duplication across MapLayer*TimeseriesProps to just one interface MapLayerRasterTimeseriesProps since the only thing that seemed to separate these types was assetUrlReplacements. This change is not strictly necessary for this functionality to work but it seemed to simplify things IMO.
Simplify conditionals which set the datetime extent and periodicity. The VEDA STAC metadata for these CMR collections uses null as the second datetime value in the temporal extent to indicate ongoing collections. This is the declared way to do so in the STAC collection specification, so it seems valid to me to allow for all non-vector collections to set the end datetime to the current date if it is null in the metadata.

Dataset changes

Adds 2 datasets to demonstrate titiler-cmr is working for those datasets. We could go down to one since it seems cumbersome to manage multiple mock datasets for a single use case.
Removes the GPM IMERG cmr-stac collection, since this overlaps with the titiler-cmr example and we can already cover the cmr-stac example with the TRMM dataset
Adds the cmip6 kerchunk reference dataset, so there is a dataset to test with the "zarr" type

Notes & Questions About Changes

I would be happy to go over these changes with one or more of the core UI developers.

Validation / Testing

I tested these changes by opening up all of the mock datasets explore pages.

netlify · 2024-06-12T20:33:25Z

✅ Deploy Preview for veda-ui ready!

Name	Link
🔨 Latest commit	`f793fe5`
🔍 Latest deploy log	https://app.netlify.com/sites/veda-ui/deploys/666f7982e1e1150008e204c0
😎 Deploy Preview	https://deploy-preview-1001--veda-ui.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

abarciauskas-bgse · 2024-06-12T20:36:39Z

Will address linting and ts-check later today

abarciauskas-bgse · 2024-06-13T01:40:34Z

Checks are passing but unsure about the changes in style-generators and why it seems like there is duplication of files there. Would love to pair with UI team member(s) to review!

sandrahoang686 · 2024-06-13T12:39:48Z

@abarciauskas-bgse thanks for detailed description and for the explanation on #1000. I dont mind campfiring on this together for review! It will be more interactive though since I haven't worked with the style-generators much. I'm guessing the duplication of files may be related to this past PR #805 (specifically maybe this commit 0b2dece)

sandrahoang686 · 2024-06-14T18:27:33Z

@abarciauskas-bgse the veda dashboard team is also working on deprecating the /common/mapbox/ directory as we are are moving to consolidate our maps and mapping logic to a the new map provider and logic under dir /common/map/ (see OG ticket here). This is why you may be seeing duplicated logic in both these places. Work is still happening on migrating current maps and logic under the new path. I recommend if possible to try to minimize references to anything under /common/mapbox/ and if the logic doesn't already exist under the new dir /common/map/ if you would be comfortable moving this logic over, I can work with you to do that if as well

sandrahoang686 · 2024-06-14T18:35:41Z

app/scripts/components/common/mapbox/layers/raster-paint-layer.tsx

@@ -0,0 +1,114 @@
+import { useEffect, useMemo } from 'react';


🔧 : We are working to deprecate this path. The new path for this logic should go under /common/map/. I believe you would want to migrate this over to /common/map/style-generators/.

sandrahoang686 · 2024-06-14T18:49:18Z

app/scripts/components/common/map/style-generators/hooks.ts

+interface STACforCMRResponseData {
+  collection_concept_id: string;
+  renders: Record<string, any>;
+}


I wonder 🤔 if the backend team has json schemas we could generate into typescript interfaces to import, interested.. I'll follow up with them on this

sandrahoang686 · 2024-06-14T18:55:10Z

app/scripts/components/common/map/style-generators/hooks.ts

+        let tileParams = {
+          concept_id: data.collection_concept_id,
+          datetime: date,
+          ...sourceParams


🔧 : This matches the naming of local state variable, can we change the name so it isn't confusing.

sandrahoang686 · 2024-06-14T18:58:47Z

app/scripts/components/common/map/style-generators/hooks.ts

@@ -38,11 +43,19 @@ export function useZarr({ id, stacCol, stacApiEndpointToUse, date, onStatusChang
          controller
        });

-        setAssetUrl(data.assets.zarr.href);
+        const tileParams = {


🔧 : This matches the local state variable's name, can we change this so it isn't confusing? Or i'd just set state directly with the object like so...

if (data.assets.zarr.href) { setTileParams({ url: data.assets.zarr.href, time_slice: date, ...sourceParams }); }

abarciauskas-bgse · 2024-06-14T21:51:10Z

just a update that I am making some changes based on discussion with @sandrahoang686 (to use the new map component, to fix conflicts and to fix some variable naming). so will move this to draft until that is done and this is ready for review again.

hanbyul-here · 2024-06-21T19:49:28Z

Can you also provide details on how the temporal extent/summaries should be handled? @abarciauskas-bgse I remember we briefly talked about this before, but it's been a while. We are currently shimming the end date to the current day, and density to DAY if the data is missing the density & temporal extent's end date. Is this still the case? (https://github.com/NASA-IMPACT/veda-ui/blob/main/app/scripts/components/exploration/hooks/use-stac-metadata-datasets.ts)

abarciauskas-bgse · 2024-06-25T18:52:12Z

@hanbyul-here in a word, yes that is what we are still doing.

In more detail:

temporal density: In this line https://github.com/NASA-IMPACT/veda-ui/blob/ab/titiler-cmr/app/scripts/components/exploration/hooks/use-stac-metadata-datasets.ts#L85 we first pick whatever time density is defined in the dataset config, then the stac metadata dashboard:time_density, then fallback to DAY.
temporal extent: Here we prefer the temporal extent defined in the collection's summary but we fall back to the collections temporal extent. If the second value of the temporal extent is null we assume this is an ongoing dataset and set the end date to the current datetime here. I can now see one improvement where the most recent datetime which corresponds to the time density interval. Previously we were shimming the end date to the current date only when the dataset's "type" was "cmr", however this logic should apply to all STAC collections since it is valid to use null in temporal extent as a way to represent ongoing datasets.

I noticed the same logic is used in fetchLayerById.

abarciauskas-bgse · 2024-07-09T01:09:52Z

Closing to propose a new design for different layer types

abarciauskas-bgse added 11 commits June 10, 2024 12:45

Ongoing dataset support

973ce19

layer working (with bad changes)

a4f4087

Initial refactor of existing zarr/cmr types

4b659fd

Refactor source params into use* hooks

0242f19

Changes for titiler-cmr hook before testing

495c88b

Add titiler-cmr-timeseries file

8653e48

Minor tweaks to get layer working

0813bc7

Change layer type to cmr-stac and use jet colormap

c34b16a

Rename files

402bf2c

Add MUR; remove GPM IMERG via CMR STAC

f12bd6e

Simplify fetchStacDatasetById conditional

d705bf6

abarciauskas-bgse requested a review from hanbyul-here June 12, 2024 20:33

abarciauskas-bgse requested a review from sandrahoang686 June 12, 2024 20:33

abarciauskas-bgse added 3 commits June 12, 2024 18:20

Fix lint error

1b3bbd0

Revert changes to style-generators

eb0cb4d

Fix ts-check errors

1ae67d9

sandrahoang686 reviewed Jun 14, 2024

View reviewed changes

Refactoring to new map component

a95e25b

abarciauskas-bgse marked this pull request as draft June 14, 2024 21:51

abarciauskas-bgse added 2 commits June 15, 2024 10:40

Add new files

b96b6b2

Merge main

4791707

abarciauskas-bgse added 2 commits June 15, 2024 10:56

Don't use same name for local variable

528b5a2

Run ts check

722793e

abarciauskas-bgse changed the title ~~Ab/titiler cmr~~ Titiler-cmr layer Jun 15, 2024

Add hidden value

f793fe5

This was referenced Jun 25, 2024

Verify parameter configuration for titiler-cmr and veda-ui #965

Closed

Modify Zarr layers as needed to support sending requests to titiler-cmr #966

Closed

abarciauskas-bgse mentioned this pull request Jul 3, 2024

Ab/titiler cmr 2 #1032

Closed

abarciauskas-bgse closed this Jul 9, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Titiler-cmr layer #1001

Titiler-cmr layer #1001

abarciauskas-bgse commented Jun 12, 2024

netlify bot commented Jun 12, 2024 •

edited

Loading

abarciauskas-bgse commented Jun 12, 2024

abarciauskas-bgse commented Jun 13, 2024

sandrahoang686 commented Jun 13, 2024

sandrahoang686 commented Jun 14, 2024 •

edited

Loading

sandrahoang686 Jun 14, 2024

sandrahoang686 Jun 14, 2024

sandrahoang686 Jun 14, 2024

sandrahoang686 Jun 14, 2024

abarciauskas-bgse commented Jun 14, 2024

hanbyul-here commented Jun 21, 2024 •

edited

Loading

abarciauskas-bgse commented Jun 25, 2024 •

edited

Loading

abarciauskas-bgse commented Jul 9, 2024

		@@ -0,0 +1,114 @@
		import { useEffect, useMemo } from 'react';

Titiler-cmr layer #1001

Titiler-cmr layer #1001

Conversation

abarciauskas-bgse commented Jun 12, 2024

Description of Changes

Notes & Questions About Changes

Validation / Testing

netlify bot commented Jun 12, 2024 • edited Loading

✅ Deploy Preview for veda-ui ready!

abarciauskas-bgse commented Jun 12, 2024

abarciauskas-bgse commented Jun 13, 2024

sandrahoang686 commented Jun 13, 2024

sandrahoang686 commented Jun 14, 2024 • edited Loading

sandrahoang686 Jun 14, 2024

Choose a reason for hiding this comment

sandrahoang686 Jun 14, 2024

Choose a reason for hiding this comment

sandrahoang686 Jun 14, 2024

Choose a reason for hiding this comment

sandrahoang686 Jun 14, 2024

Choose a reason for hiding this comment

abarciauskas-bgse commented Jun 14, 2024

hanbyul-here commented Jun 21, 2024 • edited Loading

abarciauskas-bgse commented Jun 25, 2024 • edited Loading

abarciauskas-bgse commented Jul 9, 2024

netlify bot commented Jun 12, 2024 •

edited

Loading

sandrahoang686 commented Jun 14, 2024 •

edited

Loading

hanbyul-here commented Jun 21, 2024 •

edited

Loading

abarciauskas-bgse commented Jun 25, 2024 •

edited

Loading