Composing Gatsby Sites (#8917)

This is the first step towards gatsby themes. It is low level and defines the way multiple gatsby sites compose by defining the way in which gatsby-config's compose. Everything else will build on top of this composition model so it's important to make it extensible and maintainable for the future.

For those that are mathematically inclined, this defines a monoid for the gatsby-config data structure such that `(siteA <> siteB) <> siteC === siteA <> (siteB <> siteC)`. This makes it nice when thinking about sub-theming in the future (imagine a complex `ThemeA <> subthemeA <> ThemeB <> subthemeB <> user-site` situation)

This method of composition opens the door to themes and sub-themes and allows us to get more user input into how to deal with potentially conflicting artifacts (such as two singleton plugins being defined), test out approaches to generic overriding the rendering of components in user-land, and more.

## Themes

A theme is defined as a parameterizable gatsby site. This means that gatsby-config can be a function that accepts configuration from the end user or a subtheme. This is important because in the current state of the world when setting up plugins like `gatsby-source-filesystem`, we need them to be configured with a `__dirname` from the user's site (we could have a special `__inTheCurrentSite` value in the future instead).

In the end-user's site, we declare a "theme" using the `__experimentalThemes` keyword in gatsby-config. We use this keyword so that people are aware this functionality is experimental and may change without warning. A theme can be configured in the same way plugins are configured (TODO: change `[theme, config]` syntax to match plugin `{resolve:,options}` form) so that the userland APIs match up.

```js
// gatsby-config.js
module.exports = {
  __experimentalThemes: [[`blog-theme`, { dir: __dirname }]],
}
```

The theme then includes a gatsby-config.js which allows it to defined all of the expected fields, such as plugins, and also configure them based on user input. (TODO: looks like gatsby-config.js is ignored in the .gitignore file for the theme package in commit #2)

```js
// blog-theme gatsby-config.js
module.exports = ({ dir }) => ({
  siteMetadata: {},
  plugins: [
    `gatsby-mdx`,
    {
      resolve: `gatsby-source-filesystem`,
      options: {
        name: "blog-posts",
        path: `${dir}/blog-posts/`,
      },
    },
  ],
})
```

### Composing themes

Multiple themes can be used, although there is (intentionally) nothing included in this PR to stop or resolve potential conflicts (for example if a gatsby plugin needs a singleton instance for some reason).

```js
// gatsby-config.js
module.exports = {
  __experimentalThemes: [`blog-theme`, `store-theme`],
}
```

### Themes as plugins

Themes are also included in the plugin list, so they can take advantage of using files such as `gatsby-node`. When being used as plugins, themes receive the full themeConfig as the options object. As an example, a blog theme could be instantiated multiple times on a site, once for blog posts and once for product reviews.

```js
// gatsby-config.js
module.exports = {
  __experimentalThemes: [
    [`blog-theme`, { baseUrl: '/posts' }},
    [`blog-theme`, { baseUrl: '/reviews' }]
  ],
}
```

# etc

##### Commits

This PR contains two commits. The first is the actual functionality, the second is a set of examples (a theme defined as an npm package and an example site using said theme). I expect to remove the second commit before merging, but am open to other approaches to keep an example, etc around and develop it further as we progress.

##### This PR intentionally does not cover:

- Defining data types in any way different than the current sourcing patterns
- Any official sub-theming support for overriding components, etc
  * the only way to "override" things right now is to use gatsby lifecycles (ex: on-create-page hooks) to replace the full page component.
  * still technically possible in user-land, planned but not included in core yet

packages/gatsby/src/bootstrap/index.js

@@ -8,8 +8,10 @@ const crypto = require(`crypto`)
 const del = require(`del`)
 const path = require(`path`)
 const convertHrtime = require(`convert-hrtime`)
+const Promise = require(`bluebird`)
 
 const apiRunnerNode = require(`../utils/api-runner-node`)
+const mergeGatsbyConfig = require(`../utils/merge-gatsby-config`)
 const { graphql } = require(`graphql`)
 const { store, emitter } = require(`../redux`)
 const loadPlugins = require(`./load-plugins`)
@@ -62,14 +64,42 @@ module.exports = async (args: BootstrapArgs) => {
   })
 
   // Try opening the site's gatsby-config.js file.
-  let activity = report.activityTimer(`open and validate gatsby-config`, {
+  let activity = report.activityTimer(`open and validate gatsby-configs`, {
     parentSpan: bootstrapSpan,
   })
   activity.start()
-  const config = await preferDefault(
+  let config = await preferDefault(
     getConfigFile(program.directory, `gatsby-config`)
   )
 
+  // theme gatsby configs can be functions or objects
+  if (config.__experimentalThemes) {
+    const themesConfig = await Promise.mapSeries(
+      config.__experimentalThemes,
+      async ([themeName, themeConfig]) => {
+        const theme = await preferDefault(
+          getConfigFile(themeName, `gatsby-config`)
+        )
+        // if theme is a function, call it with the themeConfig
+        let themeConfigObj = theme
+        if (_.isFunction(theme)) {
+          themeConfigObj = theme(themeConfig)
+        }
+        // themes function as plugins too (gatsby-node, etc)
+        return {
+          ...themeConfigObj,
+          plugins: [
+            ...(themeConfigObj.plugins || []),
+            // theme plugin is last so it's gatsby-node, etc can override it's declared plugins, like a normal site.
+            { resolve: themeName, options: themeConfig },
+          ],
+        }
+      }
+    ).reduce(mergeGatsbyConfig, {})
+
+    config = mergeGatsbyConfig(themesConfig, config)
+  }
+
   if (config && config.polyfill) {
     report.warn(
       `Support for custom Promise polyfills has been removed in Gatsby v2. We only support Babel 7's new automatic polyfilling behavior.`

packages/gatsby/src/joi-schemas/joi.js

@@ -1,6 +1,7 @@
 const Joi = require(`joi`)
 
 export const gatsbyConfigSchema = Joi.object().keys({
+  __experimentalThemes: Joi.array(),
   polyfill: Joi.boolean(),
   siteMetadata: Joi.object(),
   pathPrefix: Joi.string(),

packages/gatsby/src/utils/__tests__/merge-gatsby-config.js

@@ -0,0 +1,113 @@
+const mergeGatsbyConfig = require(`../merge-gatsby-config`)
+
+describe(`Merge gatsby config`, () => {
+  it(`Merging empty config is an identity operation`, () => {
+    const emptyConfig = {}
+    const basicConfig = {
+      plugins: [`gatsby-mdx`],
+    }
+
+    expect(mergeGatsbyConfig(basicConfig, emptyConfig)).toEqual(basicConfig)
+    expect(mergeGatsbyConfig(emptyConfig, basicConfig)).toEqual(basicConfig)
+  })
+
+  it(`Merging plugins concatenates them`, () => {
+    const basicConfig = {
+      plugins: [`gatsby-mdx`],
+    }
+    const morePlugins = {
+      plugins: [`a-plugin`, `b-plugin`, { resolve: `c-plugin`, options: {} }],
+    }
+    expect(mergeGatsbyConfig(basicConfig, morePlugins)).toEqual({
+      plugins: [
+        `gatsby-mdx`,
+        `a-plugin`,
+        `b-plugin`,
+        { resolve: `c-plugin`, options: {} },
+      ],
+    })
+    expect(mergeGatsbyConfig(morePlugins, basicConfig)).toEqual({
+      plugins: [
+        `a-plugin`,
+        `b-plugin`,
+        { resolve: `c-plugin`, options: {} },
+        `gatsby-mdx`,
+      ],
+    })
+  })
+
+  it(`Merging plugins uniqs them, keeping the first occurrence`, () => {
+    const basicConfig = {
+      plugins: [`gatsby-mdx`],
+    }
+    const morePlugins = {
+      plugins: [
+        `a-plugin`,
+        `gatsby-mdx`,
+        `b-plugin`,
+        { resolve: `c-plugin`, options: {} },
+      ],
+    }
+    expect(mergeGatsbyConfig(basicConfig, morePlugins)).toEqual({
+      plugins: [
+        `gatsby-mdx`,
+        `a-plugin`,
+        `b-plugin`,
+        { resolve: `c-plugin`, options: {} },
+      ],
+    })
+    expect(mergeGatsbyConfig(morePlugins, basicConfig)).toEqual({
+      plugins: [
+        `a-plugin`,
+        `gatsby-mdx`,
+        `b-plugin`,
+        { resolve: `c-plugin`, options: {} },
+      ],
+    })
+  })
+
+  it(`Merging siteMetadata is recursive`, () => {
+    const a = {
+      siteMetadata: {
+        title: `my site`,
+        something: { else: 1 },
+      },
+    }
+
+    const b = {
+      siteMetadata: {
+        something: { nested: 2 },
+      },
+    }
+
+    expect(mergeGatsbyConfig(a, b)).toEqual({
+      siteMetadata: {
+        title: `my site`,
+        something: { else: 1, nested: 2 },
+      },
+    })
+  })
+
+  it(`Merging proxy is overriden`, () => {
+    const a = {
+      proxy: {
+        prefix: `/something-not/api`,
+        url: `http://examplesite.com/api/`,
+      },
+    }
+
+    const b = {
+      proxy: {
+        prefix: `/api`,
+        url: `http://examplesite.com/api/`,
+      },
+    }
+
+    expect(mergeGatsbyConfig(a, b)).toEqual({
+      proxy: {
+        prefix: `/api`,
+        url: `http://examplesite.com/api/`,
+      },
+    })
+  })
+})

packages/gatsby/src/utils/merge-gatsby-config.js

@@ -0,0 +1,39 @@
+const _ = require(`lodash`)
+/**
+ * Defines how a theme object is merged with the user's config
+ */
+module.exports = (a, b) => {
+  // a and b are gatsby configs, If they have keys, that means there are values to merge
+  const allGatsbyConfigKeysWithAValue = _.uniq(
+    Object.keys(a).concat(Object.keys(b))
+  )
+
+  // reduce the array of mergable keys into a single gatsby config object
+  const mergedConfig = allGatsbyConfigKeysWithAValue.reduce(
+    (config, gatsbyConfigKey) => {
+      // choose a merge function for the config key if there's one defined,
+      // otherwise use the default value merge function
+      const mergeFn = howToMerge[gatsbyConfigKey] || howToMerge.byDefault
+      return {
+        ...config,
+        [gatsbyConfigKey]: mergeFn(a[gatsbyConfigKey], b[gatsbyConfigKey]),
+      }
+    },
+    {}
+  )
+
+  // return the fully merged config
+  return mergedConfig
+}
+const howToMerge = {
+  /**
+   * pick a truthy value by default.
+   * This makes sure that if a single value is defined, that one it used.
+   * We prefer the "right" value, because the user's config will be "on the right"
+   */
+  byDefault: (a, b) => b || a,
+  siteMetadata: (objA, objB) => _.merge({}, objA, objB),
+  // plugins are concatenated and uniq'd, so we don't get two of the same plugin value
+  plugins: (a = [], b = []) => _.uniqWith(a.concat(b), _.isEqual),
+  mapping: (objA, objB) => _.merge({}, objA, objB),
+}