createPages() does not re-run when files change

[andrewagain]

Recently switched to using createPages() in gatsby-node.js for some of my similar pages. Noticed that changes to my data do not trigger updates to my pages. createPages() seems to only be called once during startup. Would be nice if it was called again when my data changes.

Related issues:
#987
#1315

More details:
I'm using gatsby-transformer-json and I'm querying that data with graphql in createPages(). I'm creating one page for each object found in my JSON file.

Seems like if people are creating pages based on graphql data, it will be important that changes to that data trigger new pages to be created.

I've attached a simplified version of my use-case.

gatsby-json-update-test.zip

Versions:
Gatsby 1.2.0
Node v6.10.2
Mac 10.12.5

[chrisk2020]

I'm exploring the wordpress-source plugin - "Seems like if people are creating pages based on graphql data, it will be important that changes to that data trigger new pages to be created" pretty much sums it up: if creating pages from any graphql data source is it the expected behaviour to have to run the build process again on save/ data change?

[KyleAMathews]

Hey! Thanks for bringing this up. This is definitely the plan. Was actually hoping to get this done before v1 launch but it's one of the features that got edited out of the launch plan and postponed.

Everything is setup already to do this. I even documented things for this haha https://www.gatsbyjs.org/docs/node-apis/#createPagesStatefully

Anyways, the algorithm will be basically

1. On a data change, wait for data transformation to finish and call createPage for every implementing plugin.
2. Compare the array of pages returned by this call to the previous list of pages. If there were any pages not recreated, call deletePage to remove them.

Should work well.

Lemme know if you want to take this on and I can give you some more pointers!

[chrisk2020]

Would that be a (in my case [source-wordpress]) plugin level change or something 'deeper in the core'? Would love to crack this though just finding my feet with gatsby - so probably need some hand-holding.

[KyleAMathews]

@chrisk2020 it's a core issue but fairly tricky. Making this change requires understanding core well. Feel free to dive in though! Happy to help you along where can.

[andrewagain]

@KyleAMathews
I spent a little time looking into this but didn't get very far.

This is what I learned:

• I see that the CREATE_NODE action will be emitted when a new node is created. That's great.

• I can use apiRunnerNode to call createPages.

• I see my gatsby-node.js API method createPages being called, and I see it in turn calling boundActionCreators.createPage. Also good.

And these are my issues:

1. boundActionRunners.createPage does not seem to have any effect when called after bootstrap is complete. New pages are not created/updated.

2. I'm not sure how to determine what pages each plugin created so that I can delete the ones that were not created (as you suggested) - apiRunnerNode doesn't provide a list of which bound actions were called does it?

3. An invariant violation is thrown in query-watchers.js. It seems that this file expects to find a src/pages/<component name> file for each page. This is probably the least of my concerns right now.

My work so far (as little as it is) is in this commit: andrewagain@a9e32dd

Any light you can shed on these would be appreciated. Thanks!

[sebastienfi]

@KyleAMathews This is definitely something that will make its way into the gatsby-source-wordpress plugin

[KyleAMathews]

@sebastienfi this is a core Gatsby concern that needs to work for everyone.

@ahfarmer You need to wait until all transformer plugins have finished running. For example, you could add a new markdown file to your site which triggers the creation of a new File node. If you tried to create a page based on Markdown at that point, nothing would happen.

Instead you want to wait for the somewhat poorly named API_RUNNING_QUEUE_EMPTY event which is emitted whenever the API queue goes to zero or in other words, all transformer plugins have finished running.

You'll also want to listen for new nodes being created as API_RUNNING_QUEUE_EMPTY will get called in response to other internal changes not related to nodes. So in other words, when a node is created, mark the pages state as "dirty" so you know to regenerate pages in response to API_RUNNING_QUEUE_EMPTY.

It doesn't matter which plugins created the pages. You'll just take a snapshot of the pages data before running createPages and then again after and compare the two to see which pages weren't recreated and then delete those pages.

[KyleAMathews]

Some code links

• gatsby/packages/gatsby/src/utils/api-runner-node.js

  Line 159 in d68f7ab

  emitter.emit(`API_RUNNING_QUEUE_EMPTY`)

• gatsby/packages/gatsby/src/internal-plugins/query-runner/page-query-runner.js

  Line 54 in d68f7ab

  emitter.on(`API_RUNNING_QUEUE_EMPTY`, runQueuedActions)

[andrewagain]

@KyleAMathews AWESOME this clarifies everything. I'll take another stab at this tonight.

[andrewagain]

@KyleAMathews I'm getting really close here.

The last thing I have to do is the page deletion as you described. I'm not sure how to deal with createPagesStatefully. Pages created by that method will not have been re-created after I call createPages(), but they also should not be deleted, isn't that correct?

Is there a way to differentiate pages created statefully from those that were not?

Thank you!

BTW the changes I'm working with are here:
master...ahfarmer:topics/hot-create-pages
Any feedback on that would be great.

[leepowelldev]

Sorry to bring this up on such an old issue - but was there ever any solution to re-run createPages when source changes from something like gatsby-source-wordpress?