Create minimal partial data layer scaffold starting off with Data Catalog for VEDA2 Refactor

[sandrahoang686]

Related Ticket:

• [VEDA2 Refactor] Exploration - Data layer scaffolding
• [VEDA2 Refactor] - Data Catalog Core Feature Breakout

Description of Changes

This is a very minimal partial implementation of the data layer scaffolding to start the conversation and work on breaking out the feature components which will involve moving data logic out of the components into the data layer. This PR moves the datasets dependency out from the Data Catalog component to help kick off breaking out this Feature Component to be data agnostic

This Data Layer scaffold will be a WIP throughout the refactor and scaffold WILL EVOLVE. This is just to define a current place where data logic can be migrated too to start core feature breakouts

Notes & Questions About Changes

@hanbyul-here thoughts?

Validation / Testing

• Data Catalog Page should work the same

[netlify]

✅ Deploy Preview for veda-ui ready!

Name	Link
🔨 Latest commit	7c80c2a
🔍 Latest deploy log	https://app.netlify.com/sites/veda-ui/deploys/6618540bd60c3c0008ed099a
😎 Deploy Preview	https://deploy-preview-893--veda-ui.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site configuration.

[oliverroick]

I looked over this but I don’t fully understand these changes and how it will evolve in the future.

I left two questions in the code but my main question is about the data-layer/datasets module: How will this be used in the future? I’m assuming there will be more complexity in the module at some point; what functionality are we planning to implement in the module in the future?

[oliverroick]

To keep the repo clean, could we move this into from the repository root to a subdirectory like src?

Suggested change

	"$data-layer/*": ["data-layer/*"]
	"$data-layer/*": ["src/data-layer/*"]

[hanbyul-here]

☝️ Can we put data-layer in app/scripts? That is where all the 'scripts' are right now.

[sandrahoang686]

@oliverroick @hanbyul-here should I move it under app/scripts/* or create a new directory to do src/data-layer/*? I will say I personally dont understand why we have a dir under app named /scripts and why it was named that but maybe we can revisit naming later.. lmk which you both prefer here

[oliverroick]

If we want to keep it all in the same place it should go to app/scripts. But I agree, we should revisit that at some point, this app/scripts convention we're using in some projects is odd.

[sandrahoang686]

Updated, can I get a re-review?

[sandrahoang686]

I looked over this but I don’t fully understand these changes and how it will evolve in the future.

I left two questions in the code but my main question is about the data-layer/datasets module: How will this be used in the future? I’m assuming there will be more complexity in the module at some point; what functionality are we planning to implement in the module in the future?

@oliverroick thanks for taking a look, so for the data-layer/datasets module, I was thinking we would ideally move that logic that parses the mdx files here. Right now that logic sits behind veda virtual modules which i'm actually not too familiar with but looks likes it is part of the Parcel Resolver... In the future, we may want to move that logic to live here directly instead so we don't have to use these virtual modules but I haven't explored that yet.

This PR is just moving the veda virtual modules datasets under the data-layer to establish a single source where our components would now call from. If we updated this logic or removed this virtual module, ideally it wouldn't affect the components because of this abstraction.

I was also thinking we would ideally have much of our fetching and hooks logic live here, doesn't have to be explicitly this file (file names and structure will change, I just created this file for now to just start).

Then the rest of the PR is just me creating a wrapper container for the DataCatalog Feature Component that interacts with the data layer to provide the data to this feature component. This is just to support the current architecture though. The template instance wouldn't need this wrapper container i'm creating as they would have their own page views that would interact with the data layer and pass directly to the feature component.

Does this help answer you Q? And any ideas from your end though on how to do this better though? Since this is the scaffold, right now is a great time to start talking about it! 🙏🏼

[oliverroick]

Thanks @sandrahoang686 that's clear now. As for the work you've put in the PR, that looks all good to me.

What we need to consider though is how the changes you plan in future relate to and affect to the discussion/spike for the parcel builder the @hanbyul-here is working on. From what Daniel is saying in the ticket, the parcel builder not only parses the content but also generates option lists that allow filtering for content. You mentioned data fetching in hooks, were you planning that this happens on-the-fly or are we still going to use a fair amount of pre-processing?

[sandrahoang686]

@oliverroick yeah I didn't realize how much the parcel resolver was doing... and I haven't thought about it. I'm thinking though and if I understood your question, seems like the parcel resolver is generating the list of taxonomies and datasets during build time but i'm thinking can we do something similar with something like NextJs's pre-rendering and static generation? So both? We initially fetch at build time with something like getStaticProps if using nextjs and this will not be managed by the hooks but the app can be pre-rendered and then fetch again on the client side when we need to?

What do you think?