refactor(gatsby): enable running of static/page queries separately #12891

Moocar · 2019-03-27T04:29:13Z

Description

As part of my work to remove the global data.json, I need to run the build-javascript step after static queries have been run, but before page queries. This is because each page result file also includes the webpack build's compilation hash, so that the running app can be invalidated if a rebuild occurs while a user is navigating the page (more info in #11982).

While I was doing this refactor, I also took the opportunity to remove a bunch of global state/event emitters. The biggest is that the query queue is now instantiable. So now we can create a queue that processes the static queries, and then create a new one that processes the page queries. Then, we can start a fresh queue processor once gatsby develop has started. Hopefully it makes that part of the code easier to reason about.

I also took a stab at making the calculation of dirty queryIds a bit easier to read. It's not great, but hopefully these changes make it a little easier to understand.

page-query-runner has become index. Apologies for the lack of a good diff, but a lot of that file has changed. I have a follow up branch that moves src/internal-plugins/query-runner/ to src/query/

Related Issues

feat(gatsby): Move page component state & side effect handling to xstate #11897

Moocar · 2019-03-27T04:41:21Z

@KyleAMathews this fixes the

  // HACKY!!! TODO: REMOVE IN NEXT REFACTOR
  activity.start()
  emitter.emit(`START_QUERY_QUEUE`)
  await queryRunner.processQueries(
  // END HACKY

https://github.com/gatsbyjs/gatsby/pull/12891/files#diff-8c8b888e741f1c31e97d4bf05894f50dL476

pieh · 2019-03-27T21:36:50Z

packages/gatsby/src/internal-plugins/query-runner/query-queue.js

+      }
+    },
+    store: FastMemoryStore(),
+  }
 }


I think we want to keep those - those add extra overhead which is only needed in watch mode - but for production builds we don't really benefit from those

Oh wow, that was accidental. I didn't mean to delete that part. Great catch

Resolved in 60d1241. In my future branch, the websocket stuff is passed in by gatsby develop, so cleans up this file further.

pieh · 2019-03-27T21:38:45Z

Can you expand, why splitting query running is needed? Not including page queries result paths in file that gets webpacked seems like would achieve same result without touching so much code? What am I missing?

Moocar · 2019-03-27T22:01:53Z

Each page query result will now be in its own page-data.json file. Which looks like:

{
  "componentChunkName": "component---src-pages-index-js",
  "path": "/",
  "compilationHash": "6caec38909d092c4e3d3",
  "data": {
    "allFoo": {
      ...
    }
  },
  "pageContext": {
    "isCreatedByStatefulCreatePages": true
  }
}

The compilation hash references the latest webpack compilation. So each page-data.json depends on the build-javascript stage. So they need to be run in separate phases. I'm still working on the code that ends up using this, but you can see a preview at https://github.com/Moocar/gatsby/blob/per-page-manifest/packages/gatsby/src/commands/build.js#L65.

cleanup bootstrap moved page-query-runner to index use finishBootstrap() fix query-runner type annotation refactor query-runner/index doc sections queryjobs refactor make query refactor enqueueQueryID -> enqueueExtractedQueryId

Moocar · 2019-03-27T22:52:31Z

@pieh I'm trying to break up my huge data.json PR into multiple small ones to assist with reviewing. But recognize that it makes the context harder to understand. I think it's worth it, but let me know if you'd like the full PR instead (which isn't quite finished yet)

KyleAMathews · 2019-03-28T00:09:45Z

Couldn't we just append the compilationHash to the page-data.json file? I think we'll need to add that capability anyways to support my image prefetching RFC as we won't discover which images need prefetching until we're rendering HTML.

KyleAMathews · 2019-03-28T00:14:30Z

It's more disk I/O but should be relatively cheap. Tried googling how to append to a JSON file (which would be cheapest) but it doesn't look straightforward. We should be fine though since files will almost always be < 100kb.

KyleAMathews · 2019-03-28T00:18:53Z

packages/gatsby/src/internal-plugins/query-runner/query-queue.js

-  if (!isBootstrapping) {
-    queue.resume()
+/**
+ * Creates a queue that is optimized for running as a daemon during


curious about the terminology — a daemon is normally a separate process right?

Yeah, I'm not sold on daemon as terminology either. I'm trying to communicated that it's a long running process that won't end. Other ideas? startListener, startProcess?

@pieh and I and some others have talked about modelling more of Gatsby's internals after the Actor Model. It's a larger discussion to conclude if we want to do that but "startActor" would be a natural name then.

If it is actually an actor, then for sure :). But until then, we should probably stick to something generic so we don't confuse people (and yes, daemon is also a bit confusing).

If it believes it's an actor it is! Haha.

Service is a nice generic name :-)

Yep, I like service. Will change

Moocar · 2019-03-28T01:00:32Z

Couldn't we just append the compilationHash to the page-data.json file? I think we'll need to add that capability anyways to support my image prefetching RFC as we won't discover which images need prefetching until we're rendering HTML.

Just caught up with @KyleAMathews and we chatted about this. While we could figure out a way to load/rewrite the page-data.json after the build-javascript phase, there's technically no reason we need to finish running page queries before hand. Given that, it's simpler to write the page-data.json once after build javascript has finished.

In fact, it enables interesting possibilities like running build-javascript on another process :D

Moocar · 2019-03-31T20:36:24Z

I've made a few more changes downstream of this branch. I'm going to close this and reopen (or recreate) once those changes are settled.

Moocar requested review from a team as code owners March 27, 2019 04:29

pieh reviewed Mar 27, 2019

View reviewed changes

Moocar added 2 commits March 28, 2019 09:11

documenting

e85d723

Moocar force-pushed the query-runner-refactor2 branch from 498227e to e85d723 Compare March 27, 2019 22:12

Create queue configurations for build/develop

60d1241

KyleAMathews reviewed Mar 28, 2019

View reviewed changes

pieh mentioned this pull request Mar 29, 2019

Authentication support #1100

Closed

Moocar closed this Mar 31, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

refactor(gatsby): enable running of static/page queries separately #12891

refactor(gatsby): enable running of static/page queries separately #12891

Moocar commented Mar 27, 2019 •

edited

Loading

Moocar commented Mar 27, 2019

pieh Mar 27, 2019

Moocar Mar 27, 2019

Moocar Mar 27, 2019

pieh commented Mar 27, 2019

Moocar commented Mar 27, 2019

Moocar commented Mar 27, 2019 •

edited

Loading

KyleAMathews commented Mar 28, 2019

KyleAMathews commented Mar 28, 2019

KyleAMathews Mar 28, 2019

Moocar Mar 28, 2019

KyleAMathews Mar 28, 2019

Moocar Mar 28, 2019

KyleAMathews Mar 28, 2019

Moocar Mar 31, 2019

Moocar commented Mar 28, 2019

Moocar commented Mar 31, 2019

refactor(gatsby): enable running of static/page queries separately #12891

refactor(gatsby): enable running of static/page queries separately #12891

Conversation

Moocar commented Mar 27, 2019 • edited Loading

Description

Related Issues

Moocar commented Mar 27, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pieh commented Mar 27, 2019

Moocar commented Mar 27, 2019

Moocar commented Mar 27, 2019 • edited Loading

KyleAMathews commented Mar 28, 2019

KyleAMathews commented Mar 28, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Moocar commented Mar 28, 2019

Moocar commented Mar 31, 2019

Moocar commented Mar 27, 2019 •

edited

Loading

Moocar commented Mar 27, 2019 •

edited

Loading