Custom Node Types and big Gatsby refactor #1145

rogermparent · 2020-04-14T06:07:29Z

Hi everyone! I just discovered this project from an advertisement on the Gatsby jobs board on Discord. I have a lot of time with Gatsby under my belt, especially the schema, and this PR is an application of some best practices I've picked up along the way.

Blog Posts, Docs Pages, and Authors now have their own custom node types. Fields that used to be accessed through MarkdownRemark are all copied and accessible through the custom nodes with old nested fields like frontmatter and childImageSharp flattened to the Node's root level.
While not as initially simple as directly accessing remark nodes, these custom types pave the way for more complex logic and a more concise API in the future!
The old blog.js and docs.js helpers for createPage have now evolved into a dedicated pattern that spans the whole Gatsby lifecycle: "Models".
Models are basically tiny Gatsby plugins to the point where they respond to the same hooks, but Models are local to this repository and can respond to custom events like onCreateMarkdownContentNode.
Models are simply to organize code specific to this repo, and can easily be broken out into dedicated plugins later if need be.
I put a few little flags in page creation that skip most pages while keeping at least one of each kind when NODE_ENV is development. This means gatsby develop only has to remake a bit over 200 images after cache clear instead of 1000+. Developers can still build the full site with gatsby build, and production isn't affected. Still, this is a tad hacky so I'm open to removing it if necessary.
The new Author node has a posts field with a limit argument, which paves the way to fixing blog: make author name clickable, show description and posts by the author #1120 after a few design choices. My vote is for linked separate Author pages, but I'm not a designer.

I understand this is a massive PR, but there aren't many points outside of the existing commits that can be removed from the final product without breaking the build. Feel free to ask me anything, and I'm open to any changes that need to be made!

Models are a way to conceptualize and organize Gatsby API calls for separate content types like blog posts and doc pages. A Model is an object whose keys match either the Gatsby Node API or custom calls defined by other Models. At its core, a Model is a very barebones Gatsby plugin that can respond to events from other Models. Models are intended to be a little easier then Gatsby plugins to work with within a single repo, and they work so similarly to plugins that breaking them out down the line is no problem. Change list: - Swap gatsby-node API calls out for Model-based ones where applicable - onCreateNode and createPages are changed - createSchemaCustomization is added - Adapt Blog templates, fragments, and queries to new data shape - Most, if not all queries involving the markdown files are changed to go through the new custom nodes. - Custom nodes' fields are identical to the remark ones for the most part, other than the new node types and frontmatter/fields being hoisted up to the custom node's root. - Resolver-based Remark fields (html/Ast, timeToRead) are accessed by using a fragment on the node's parent. - Change various uses of "edges" to "nodes" - Convert src/gatsby page builders for posts and docs into Models - Move page path generation logic into Models and remove the original function - Define model list and imports in src/gatsby/models - Make MarkdownContent Model that bootstraps other Markdown-based Models - Merge IBlogPostFrontmatter into IBlogPostData - Access nodes in page queries by id instead of slug - Only generate one blog post in dev mode to cut down on image regeneration. (full site can still be built with `gatsby build`) - Change the feed plugin's query and serialization to use BlogPost nodes

- Added "gatsby-plugin-parent-resolvers", used functional-style and not imported to config. It's a pretty simple plugin, and can be replicated in the repo if need be. - Added resolver builder utils to help with making custom resolvers across similar node types. - `createSchemaCustomization` is now implemented on Authors, BlogPosts and DocsPages - MD-based custom nodes and ImageSharp nodes are now "id"ed with their relative path. - This is used extensively for the new node relations (I believe it's more performant than fetching the real ID or hopping through a few relations every time) - ImageSharps simply have this added through a Model that uses `createNodeField` - Custom node fields that used to depend on parent fragments now have built-in root level resolvers on their custom nodes. (html, htmlAst, timeToRead on Markdown-based nodes) - Markdown files that link to other files in their frontmatter now resolve to custom nodes. (blog post -> author) - BlogPost picture and Author avatar now link directly to ImageSharp nodes. - Adapted templates, interfaces, and queries to new data shape. Basically, anything that used to be in `parent`, `childMarkdownRemark`, or `childImageSharp` is now bumped up to root. - Some uses of Promise.all are made in the blog model in an attempt to better parallelize the page building process (These can be removed if need be)

By default, this field returns an object with all posts the author has made sorted by date descending as well as a totalCount. The field can also be supplied a "limit" argument to limit the amount of posts returned. This does not affect totalCount.

rogermparent · 2020-04-14T06:12:11Z

I should mention there's a few less-than-dry parts of the code on node generation, but I left them in there for easier adaptation to type-specific changes.

gatsby-config.js

…nd picture comments.

rogermparent · 2020-04-14T22:04:54Z

This PR can be adapted to #1073 by dropping a ternary in the docs model's path resolution

src/gatsby/utils/models/index.js

gatsby-node.js

src/gatsby/models/blog/index.js

src/gatsby/models/blog/createPages.js

src/gatsby/utils/resolvers/index.js

shcheklein · 2020-04-20T00:35:07Z

src/gatsby/utils/resolvers/index.js

+
+function resolveRelativeImage() {
+  return async function (source, args, context, info) {
+    const relativeChildPath = context.defaultFieldResolver(


need to ask @jorgeorpinel eventually to test this on Windows

Assuming this is about the thumbnail generation on dev env?

src/gatsby/models/docs/onCreateNode.js

src/gatsby/models/blog/createPages.js

src/gatsby/models/docs/onCreateNode.js

src/gatsby/models/markdown-content/index.js

src/gatsby/models/blog/onCreateNode.js

- Re-do docs/comments in a more understandable fashion. - Adapt to the new API runner and omit pluginOptions in the custom API. -

Since the functional Gatsby hook builder was received poorly, I flattened out the signature of `buildModelApi` and turned it into `runOnModels`, an async function with only one layer of signature and, by virtue of being async, can be cleanly used in `gatsby-node`. - Remove the options parameter from Model signatures, since we don't use it currently. This was to match Gatsby's `(api, options)` signature and this change makes Models deviate from plugins slightly more, but re-adding it in the case of making a Model into a Plugin would be nearly effortless so the change was made for aesthetics. - `onCreateMarkdownContentNode` hooks are now named after the custom hook instead of `onCreateNode`. - Completely rewrote docs for Model runner

- Initialize dotenv in `gatsby-node` to ensure it's loaded for all models. - Make the Blog Page limiter in the blog/createPages module trigger only when LIMIT_BLOG_PAGES is truthy.

rogermparent

I've made various improvements to this Model PR that address the points @shcheklein brought up.

Major changes include:

The model API builder is now a model API runner with a one layer signature.
The page limiter now runs when the environment variable LIMIT_BLOG_PAGES is truthy.
Rewritten and improved docs for the Model API and the MarkdownContent Model.

src/gatsby/utils/models/index.js

gatsby-node.js

src/gatsby/models/blog/createPages.js

src/gatsby/models/blog/onCreateNode.js

src/gatsby/models/docs/onCreateNode.js

src/gatsby/models/markdown-content/index.js

src/gatsby/utils/resolvers/index.js

src/gatsby/utils/models/index.js

gatsby-node.js

src/gatsby/models/blog/onCreateMarkdownContentNode.js

src/gatsby/models/docs/onCreateMarkdownContentNode.js

shcheklein · 2020-04-21T02:42:54Z

@jorgeorpinel could please review this on your dev environment?

…tter fields. The old resolvers were built around the MD content not changing, but given the greenlight to change the Markdown, these new resolvers can both simplify the resolver implementation logic as well as make linking images nicer for editors. For editors: - author avatar is now relative to "static/uploads/avatars" - authors no longer have the "path" frontmatter field - blog post picture is now relative to "static/uploads/images" - blog post author is now relative to "content/authors" and doesn't use an extension. Author frontmatters go from this: ``` --- path: ../authors/dmitry_petrov.md name: Dmitry Petrov avatar: ../../static/uploads/avatars/dmitry_petrov.png --- ``` to: ``` --- name: Dmitry Petrov avatar: dmitry_petrov.png --- ``` And for blog frontmatters, this: ``` author: ../authors/marija_ilic.md picture: ../../static/uploads/images/2017-07-24/post-image.png ``` to this: ``` author: marija_ilic picture: 2017-07-24/post-image.png ``` Beyond the immediate implications, these custom resolvers allow us to change how these fields are resolved behind the scenes to adapt to any future circumstances like content relocation.

This commit brought to you by Sed.

src/gatsby/utils/models/index.js

I replaced all instances of "runOnModels" and "asyncCallOnAll" to "callOnModels"

jorgeorpinel · 2020-04-21T06:55:27Z

I put a few little flags in page creation that skip most pages while keeping at least one of each kind when NODE_ENV is development. This means gatsby develop only has to remake a bit over 200 images after cache clear instead of 1000+

Yup! This dev build time is much faster than before. Under a minute for me (vs 10+).

It also solves parts of #1084! Specifically, renaming md files in the content dir no longer procude 404s (per #1084 (comment)) (not redirects though).

And it also allows for "hidden" docs )with no sidebar entry like we once wanted (see #731).

So pretty positive it seems 👍

shcheklein · 2020-04-21T07:25:50Z

Awesome stuff, @rogermparent !

rogermparent added 3 commits April 14, 2020 01:27

Add a "posts" field to Authors

2f31cf2

By default, this field returns an object with all posts the author has made sorted by date descending as well as a totalCount. The field can also be supplied a "limit" argument to limit the amount of posts returned. This does not affect totalCount.

rogermparent added 2 commits April 14, 2020 03:52

Merge branch 'master' into gatsby-models

8f440c3

Fix lint problems after merging with the web ui.

a7cf5fe

shcheklein reviewed Apr 14, 2020

View reviewed changes

gatsby-config.js Outdated Show resolved Hide resolved

shcheklein temporarily deployed to dvc-landing-gatsby-mode-1pv1fi April 14, 2020 14:49 Inactive

Remove accidental comment paste into gatsby-config

e7e7b19

shcheklein temporarily deployed to dvc-landing-gatsby-mode-v3mx3t April 14, 2020 16:48 Inactive

Remove the rest of the accidentally pasted link

8947c00

shcheklein temporarily deployed to dvc-landing-gatsby-mode-43mqwi April 14, 2020 16:58 Inactive

Re-introduce frontmatter markdown parsing for BlogPost descriptions a…

92b8efb

…nd picture comments.

shcheklein temporarily deployed to dvc-landing-gatsby-mode-43mqwi April 14, 2020 17:49 Inactive

weekly-digest bot mentioned this pull request Apr 19, 2020

Weekly Digest (12 April, 2020 - 19 April, 2020) #1168

Closed