Normalizing nodes from different sources into same shape

[bard]

Relevant Information

I'm building a toy blog with posts coming from two different sources, an RSS feed (using gatsby-source-rss-feed) and on-disk markdown files (using gatsby-source-filesystem and gatsby-transformer-remark).

The resulting MarkdownRemark nodes and FeedBlogPost nodes have different shapes, e.g. MarkdownRemark.frontmatter.title vs FeedBlogPost.title.

I figured I'd "intercept" both types of node in my project's gatsby-node.js/onCreateNode and create nodes of a Post type to normalize the data shape, so templates could later just query allPost { edges { node { title }}}:

// gatsby-node.js  
exports.onCreateNode = ({ node, actions, createContentDigest, createNodeId }) => {
  const { createNode } = actions

  if (node.internal.type === 'FeedBlogPost') {
    const nodeData = {
      title: node.title,
      html: node.content.encoded,
      slug: '/posts/' + node.link.split('/').slice(-2).shift()
    }

    createNode({
      ...nodeData,
      id: createNodeId(`${node.id} >>> Post`),
      children: [],
      parent: node.id,
      internal: {
        type: 'Post',
        contentDigest: createContentDigest(JSON.stringify(nodeData))
      }
    })
  }

  if (node.internal.type === 'MarkdownRemark') {
    const nodeData = {
      title: node.frontmatter.title,
      html: node.html,                    // => turns out to be undefined
      slug: '/posts/' + slugify(node.frontmatter.title)
    }

    createNode({
      ...nodeData,
      id: createNodeId(`${node.id} >>> Post`),
      children: [],
      parent: node.id,
      internal: {
        type: 'Post',
        contentDigest: createContentDigest(JSON.stringify(nodeData))
      }
    })
  }
}

This indeed works for normalizing the post's title, and for the content of FeedBlogPost, but fails for the content of MarkdownRemark (node.html above turns out to be undefined). As far as I can tell from gatsby-transformer-remark's source, the problem is that html isn't stored on the node, but rather added to GraphQL asynchronously via setFieldsOnGraphQLNodeType. Is that correct?

What is the gatsby-ish approach for this scenario?

Environment (if relevant)

System:
OS: Linux 4.20 Arch Linux undefined
CPU: (4) x64 Intel(R) Core(TM) i5-7200U CPU @ 2.50GHz
Shell: 5.0.0 - /bin/bash
Binaries:
Node: 10.15.1 - /tmp/yarn--1550500623588-0.9929318226449864/node
Yarn: 1.13.0 - /tmp/yarn--1550500623588-0.9929318226449864/yarn
npm: 6.8.0 - /usr/sbin/npm
Languages:
Python: 3.7.2 - /usr/sbin/python
Browsers:
Firefox: 65.0.1
npmPackages:
gatsby: ^2.1.4 => 2.1.4
gatsby-image: ^2.0.29 => 2.0.29
gatsby-plugin-manifest: ^2.0.17 => 2.0.17
gatsby-plugin-offline: ^2.0.23 => 2.0.23
gatsby-plugin-react-helmet: ^3.0.6 => 3.0.6
gatsby-plugin-sharp: ^2.0.20 => 2.0.20
gatsby-source-filesystem: ^2.0.20 => 2.0.20
gatsby-source-rss-feed: ^1.1.1 => 1.1.1
gatsby-transformer-remark: ^2.2.5 => 2.2.5
gatsby-transformer-sharp: ^2.1.13 => 2.1.13

[wardpeet]

I'm unsure how to answer this myself, I would create a React Component that does the query and normalisation.

Let's wait for others of @gatsbyjs/core to answer this

[KyleAMathews]

Yeah, combining data like this is tricky as like you found out, some fields aren't created until the query is run.

One thing you could do is query the parent MarkdownRemark field for html so:

{
  Post {
    html
    parent {
      ... on MarkdownRemark {
        html
      }
    }
  }
}

[bard]

@KyleAMathews yes, however that would force templates to become aware of MarkdownRemark's shape again, no? Unless there's some GraphQL incantation that allows on-the fly field mapping like:

// fantasy code
{
  Post {
    normalizedHtml: html || parent {
      ... on MarkdownRemark {
        html
      }
    }
  }
}

I guess a transformer plugin would stumble on the same block of the html field not being available at processing time, correct?

@wardpeet so that would mean scrapping the intermediate Post type, running a query like below (to gather data for e.g. an index with excerpts), and do data normalization in the index page plus in the post template. I thitn I'll give that a go.

{
  allFeedBlogPost {
    edges {
      node {
        id
        title
        content { encoded }
      }
    }
  }
  allMarkdownRemark {
    edges {
      node {
        id
        frontmatter { title }
        html
      }
    }
  }
}

[gatsbot]

Hiya!

This issue has gone quiet. Spooky quiet. 👻

We get a lot of issues, so we currently close issues after 30 days of inactivity. It’s been at least 20 days since the last update here.

If we missed this issue or if you want to keep it open, please reply here. You can also add the label "not stale" to keep this issue open!

Thanks for being a part of the Gatsby community! 💪💜

[gatsbot]

Hey again!

It’s been 30 days since anything happened on this issue, so our friendly neighborhood robot (that’s me!) is going to close it.

Please keep in mind that I’m only a robot, so if I’ve closed this issue in error, I’m HUMAN_EMOTION_SORRY. Please feel free to reopen this issue or create a new one if you need anything else.

Thanks again for being part of the Gatsby community!