feat(gatsby-source-wordpress): Add custom routes with parameters

[ghost]

This is my first contribution that adds a feature to a core Gatsby package.
All kind of suggestions are very welcome!

Description

gatsby-source-wordpress fetches the main API endpoint to identify available routes and compare them to whitelisted/blacklisted ones.

But some routes are only available with parameters and do not appear on the WordPress REST API index (wp-json). Example: WPML REST API allows language switching by appending ?lang=LANG or ?wpml_lang=LANG to the request url.

Usage

gatsby-config.js

// Add to plugin options.
allRoutes: [
  { // Example with WPML.
    namespace: `wp/v2`,
    endpoint: `/pages/?lang=en`,
  },
],
includedRoutes: [
  // ... other whitelisted routes ...
  `/*/*/pages/?lang=*`,
],

Issues

I was able to fetch the route and handle parameters that are appended to the entity type. In this example: /pages/?lang=en becomes wordpress__wp_pages_lang_en, but there is no GraphQL query like allWordPressPageLangEn. I thought Gatsby would create that out of the new entity.

So how do I fetch wordpress__wp_pages_lang_en in GraphQL?

Related Issues

Fixes #10915.
Fixes #3263.
Fixes #1648.

Edit: Just discovered this PR comment, that mentions adding custom routes.

[ghost]

Found out that I missed the fetching part. Now custom routes can be queried in GraphQL (in my example with allWordpressWpPagesLangEn) – if the results are not a duplicate of another GraphQL __type.

[pieh]

Found out that I missed the fetching part. Now custom routes can be queried in GraphQL (in my example with allWordpressWpPagesLangEn) – if the results are not a duplicate of another GraphQL __type.

Ah, when you add /pages/?lang=en endpoint it probably create nodes with same IDs as nodes from default /pages endpoint and they get overwritten by them

[ghost]

Before I added the fetching, allWordPressWpPagesLangEn had the same data (one WordPress page with wordpress_id: 2) as allWordPressPages and in GraphQL, only the latter was available. Since the GraphQL data was the same, allWordPressWpPagesLangEn was removed, I suspect.

But now, allWordPressWpPagesLangEn fetches the correct data (one page with wordpress_id: 8) – the translated one.

I think this is ready now, but additional feedback is very welcome!

[pieh]

One thing that I feel is problematic is adding new routes. Maybe way to define query params used for all routes might better API for this use case?

With this you would need to repeat ?lang=en for all routes I think?

[ghost]

I think it is more flexible and would allow for hidden routes that should not appear on the REST API index – publicly available for all.

Yes, we'd need to repeat that for every route we want translations for. For me, this is mostly pages. For a complex website, two or three more. I have more of a bad feeling about adding parameter to every route that gets fetched. That could be problematic.

Example: Adding ?lang=en to every route by default prevents from fetching other languages.
Reverse example: Adding query params that do not result in a new route with new entries, but rather filter or sort some things, would be applied to all other routes we want to fetch without manipulating the response.

[ghost]

Could someone review this as I have a project in the pipeline that needs WPML and custom routes.

[wardpeet]

@cardiv is it possible to share an api with us we can fool around with?

[ghost]

@wardpeet I tested this with Local by Flywheel – a free (community edition) app for one-click local WordPress installs.

You can easily import local sites exported with Local by Flywheel. I could send you such a site with WPML and the WPML REST plugin activated. Note: The main language of this local site is set to german, but the user I created for you is set to english.

[ghost]

I count 7 issues now, that are related to this feature. We should find a solution for this very soon.

If the proposed code/API in this PR is not suitable, we could rewrite the base. But it should be possible to fetch full WPML translations and working with custom query params, no matter what. @gatsbyjs/inkteam

[ghost]

After testing this in a current project, I realized that custom routes which are identical to another type can be cumbersome.

You would have to repeat yourself by adding the same queries and createPages logic over and over for every post.

The new options look like this:

// Add to plugin options.
allRoutes: [
  { // Example with WPML.
    namespace: `wp/v2`,
    endpoint: `/pages/?lang=en`,
    type: `pages`,
  },
],
includedRoutes: [
  // ... other whitelisted routes ...
  `**/*/*/pages/?lang=*`,
],

The optional type key allows to override the default type generated by Gatsby. In this example: wordpress__wp_pages_lang_en would become wordpress__PAGE and the created nodes will be merged with all other nodes under wordpress__PAGE.

Need to fix those errors though ...

[KyleAMathews]

Do we want to do this? @pieh? Seems we'd want to fetch all languages by default and add the language as a field to nodes. And then filter when writing graphql queries in Gatsby. Is that possible?

[pieh]

Do we want to do this? @pieh? Seems we'd want to fetch all languages by default and add the language as a field to nodes. And then filter when writing graphql queries in Gatsby. Is that possible?

This is problematic because multilingual support isn't baked directly into wordpress and relies on 3rd party plugins. When I was working with WP, there was at least 2 popular multilingual plugins (WPML and Polylang) and both have its quirks. At this point it's also hard for me to research that:

• WPML is paid plugin and I didn't renew my license in quite some time so version I have is severly outdated
• Polylang free version doesn't have REST support, so again - need paid version to do any testing

But generally I think we should not create different types for different languages, those should be fields on nodes that users can filter on

[KyleAMathews]

Have people tried adding this functionality to wpgraphql? That might be a better route to move forward with this.

[firxworx]

I have a relevant project too where it'd be great to use WP as a headless CMS w/ WPML and also use Gatsby.

Note -- depending on how WPML is configured, accessing another language can also work like the following (note: no query var):

Response includes only pages in the DEFAULT language (e.g. "en"):
https://wordpress.example.com/wp-json/wp/v2/pages

Response includes only pages in a translated language ("fr" in this case):
https://wordpress.example.com/fr/wp-json/wp/v2/pages

It's worth noting that depending on configuration a behaviour of WPML is that adding the language in the URL as above for the primary language will result in a 404. For example, the following would result in a 404 if English was set as the default language:

https://wordpress.example.com/en/wp-json/wp/v2/pages

Perhaps this is another way to configure things. In any case, I tested just now on a WP with WPML configured as above by adding ?lang=en and ?lang=fr to my GET request to the WP-REST API and can confirm that the query string method still works. In other words, this approach is valid even if WPML is configured to specify the alternate language in the URL itself.

I have a developer license to WPML. If anyone needs a test WP with WPML installed please let me know.

[firxworx]

Wondering aloud, what if the WP REST API response was hacked via a WP plugin instead?

For example, if every post response had a property "translations" containing an array of complete post response objects (id, title, content, etc) plus a "locale" and/or "lang" property for each translation of the requested page, wouldn't that provide the necessary data?

For example, in gatsby-node.js something like allWordpressPost.edges.node.translations.forEach(...createPage(...)...) could create the pages for each required translation with the appropriate path set for each language.

[pieh]

@firxworx Tackling this on Wordpress side of things would probably also work, but might not be what users want - especially if Gatsby isn't the only consumer of REST api - then forcing that extra data in responses that only Gatsby would benefit from, would potentially negatively impact other consumers.

[firxworx]

@pieh that's fair enough. I suppose one could get fancier with different responses for different users, so a hypothetical gatsby user would receive the tailored responses. I do think more can be done on the Gatsby side and agree with what you're doing here. If I'm to use Gatsby in my current project, I need a good reliable solution soon, or will have to use a different approach.

[henrikwirth]

What is the status of this? I really would need this to work for my next project. Thanks for working on it. Is there anything one can help?

[firxworx]

FWIW I managed to get gatsby to generate translated pages using the approach that I suggested: using a plugin to expose each translation belonging to a given post/page in an array. I created a native language version and an 'i18n' version of each template.

In gatsby-node.js I use createPage() to generate the translated pages with the language code in the path, and pass lang to the page via context so the query in the template accesses the desired language from the available translations.

I used the WPML REST API plugin by Shawn Hooper (https://github.com/shawnhooper/wpml-rest-api) as the basis (it exposes wpml_current_locale and an array wpml_translations that identifies the locale + ID of each translation ) and added in the slug, title, and content.

I fully agree with @pieh above that tackling this on the WordPress side of things might not be the most ideal situation for many Gatsby users. It would be great if Gatsby had more "out of the box" support for WordPress and leading approaches to WP + i18n such as WPML.

[maru3l]

We shouldn't only think of it for the translation. We also can pass the status of the post type as a params, for example /page?status=publish,draft. I've made a pull request for something similar this winter but I've been refused because this pull request was already started. #13147

But we should be careful and specify which route we when to add the params because some endpoint doesn't have the same params and if we send something it doesn't support, it just breaks. For example, we can add ?status=publish,draft on page and post but not on media. If so, the media endpoint will send a error and no media will be downloaded.

[willpaige]

Any update on this? I'm currently blocked on a project.

@firxworx I'm interested in how you rendered the translated pages in gatsby-node. Could you share an example?

[maru3l]

@willpaige I've you tried with polylang and the plugin I've made wp-rest-polylang.

The way polylang (and most of the wordpress multilang plugin like WPML) is working it's making a post for each version. For example, the French version is the post id 1 and the English version the 2.

When you fetch the API with the wp-rest-polylang plugin, you receive each post with the polylang_current_lang value as a string and an array polylang_translations will all the post with a different language linked to the actual post.

So in my gatsby-node.js I'm fetching the polylang_current_lang and in my loop where I'm creating my pages, I'M passing the langKey in the page context and I've made a switch to create my path in the good language.

singles.forEach(({
  node
}) => {
  const {
    id,
    slug,
    polylang_current_lang
  } = node;

  switch (polylang_current_lang) {
    case 'en_CA':
      const path = `/en/${slug}/`;
      break;

    default:
      const path = `/${slug}/`;
      break;
  }

  createPage({
    path,
    component: path.resolve('src/templates/single.jsx'),
    context: {
      id,
      langKey: polylang_current_lang,
    },
  });
})

I'm also using the gatsby plugin gatsby-plugin-i18n for creating pages who are not from wordpress and react-intl for translating the hard-coded sting all around the website.

A good example is https://silverwax.ca/. I've made it with the stack I've just explained. For the commerce section, I've used Magento2 and for the page like training and the upcoming blog, I've used Wordpress.

[firxworx]

@willpaige heh I pivoted from this approach due to the "unknown hours" trap and went with trusty WordPress (which can be fast with caching!), however I would like to come back to using Gatsby in a future project. Per your request:

In gatsby-node.js I have the following (including an example of a Custom Post Type with the example name 'Custom'):

  ...
  allWordpressPage {
    edges {
      node {
        id
        wordpress_id
        slug
        status
        template
      }
    }
  }

  allWordpressPost(
    filter: { status: { eq: "publish" } }
    sort: { fields: [date], order: DESC }
  ) {
    edges {
      node {
        id
        wordpress_id
        slug
        status
        template
        format
        excerpt
        title
      }
    }
  }

  allWordpressWpCustom(
    filter: { status: { eq: "publish" } }
  ) {
    edges {
      node {
        id
        wordpress_id
        slug
        status
        translations {
          wordpress_id
          lang
          slug
        }
      }
    }
  }
  ...

Using the Custom post type as an example when creating pages:

const customTemplate = path.resolve(`./src/templates/custom.js`)
const customTemplateI18N = path.resolve(`./src/templates/custom-i18n.js`)

allWordpressWpCustom.edges.forEach(edge => {
  createPage({
    path: `/custom/${edge.node.slug}/`,
    component: slash(customTemplate),
    context: {
      id: edge.node.id,
      lang: 'en',
      postId: edge.node.wordpress_id
    }
  })

  edge.node.translations.forEach(translation => {
    createPage({
      path: `${translation.lang}/custom/${translation.slug}/`,
      component: slash(customTemplateI18N),
      context: {
        id: edge.node.id,
        lang: translation.lang,
        postId: translation.wordpress_id,
      }
    })
  })
})

The above would create /custom/slug pages for EN and /fr/custom/slug pages for FR (for example). This is the URL structure that I required.

Note my August 3 comment that I created a plugin that exposes the translations for each post, which was straightforward to implement: I simply added/exposed additional fields to the WPML REST API plugin code so when querying a given post in EN, all translations are included in the response. Since this would add extra overhead to the WP REST API, you might want to secure an endpoint like that such that these fields are only exposed for certain users related to gatsby (or similar).

[willpaige]

Thanks for the reply @firxworx and @maru3l and apologies for my delay in replying!

@maru3l I'm working on a large scale WPML translated site and switching to Polylang isn't an option.

@firxworx It looks as if we tackled the gatsby side of things the same, the issue I ran into was that only the base translations are being parsed. I'm not sure on the best solution for exposing all translations, seems I might copy you and hack the WPML rest api.

[maru3l]

@willpaige My way to do it work exactly the same with WPML. It just needs wpml-rest-api plugin instead.

[pieh]

Hey folks, is anyone able to provide access to site with either WPML or Polylang PRO (I think free version doesn't have REST support) to test run few other ideas I have for this?

[maru3l]

Hey folks, is anyone able to provide access to site with either WPML or Polylang PRO (I think free version doesn't have REST support) to test run few other ideas I have for this?

I've made a plugin for the free version of Polylang wp-rest-polylang. I've tried to mimic the way the pro version doing it.

[willpaige]

Oh nice @maru3l i'll give it a go!

@pieh sorry I can't help you with that!

[maru3l]

Hey folks, is anyone able to provide access to site with either WPML or Polylang PRO (I think free version doesn't have REST support) to test run few other ideas I have for this?

You can use this API Silverwax rest API. It's using polylang.

the pages are a good exemple for the basic wordpress post type https://wp-api.silverwax.ca/wp-json/wp/v2/pages

We also have custom post type news who also use polylang https://wp-api.silverwax.ca/wp-json/wp/v2/news

We use ACF for custom fields.

[pieh]

@maru3l if Silverwax site is using your plugin, I'll probably spin up local site and install free polylang and your plugin, so there's less content and will be quicker to iterate

[pieh]

Hmm, with polylang (and your plugin to add REST fields) it doesn't seem like we need changes to wordpress plugin (as REST return all posts by default so query params are not needed - it least in my test)

[maru3l]

Hmm, with polylang (and your plugin to add REST fields) it doesn't seem like we need changes to wordpress plugin (as REST return all posts by default so query params are not needed - it least in my test)

@pieh I know it's not the reason I need this feature. I was only helping for translating their website with an easier way.

Using the queryParams for translating is another way to do it, but it is much easier and gatsbyer way to use the already existing plugin like the way I've used it because we can link post with different languages altogether.

For me, using queryParans is more useful for getting Post is a draft status. By default, the rest API of wordpress only send the published post. We can pass the params status with the list of status we need /page?status=publish,draft for getting the published and draft pages.

I've made a pull request in April for this reason but it’s been closed and referred to this one. #13147

You can find the documentation about query rest API in https://developer.wordpress.org/rest-api/reference/posts/

[pieh]

Yeah, I'm trying to work out best way to support those endpoints that make use query params and trying to find something I can use to simulate WPML usage here. I think status is perfect for it (after I make wordpress to let unauthenticated users to show them as I don't want another complexity in my test setup to authenticate requests :P)

[maru3l]

Yeah, I'm trying to work out best way to support those endpoints that make use query params and trying to find something I can use to simulate WPML usage here. I think status is perfect for it (after I make wordpress to let unauthenticated users to show them as I don't want another complexity in my test setup to authenticate requests :P)

@pied https://mont-sainte-anne.com/wp-json is using WPML.

By default, the content is in french, but you can use ?lang=en https://mont-sainte-anne.com/wp-json/wp/v2/pages?lang=en

[pieh]

Ok, so few notes:

I think ultimate goal would be to, so people can create subplugins for gatsby-source-wordpress (so refresh gatsbyjs/rfcs#23 and actually implement this), so support for WPML or pulling draft posts/pages can be packaged as reusable plugin and people don't have to fish for whatever custom options they would need to apply to their gatsby-source-wordpress config.

But to do above we need to come up with set of extension points that make sense.

I think this PR identifies at least one of those pretty well: there is need to be able to add/modify/delete endpoints that we hit.

I would propose we split

gatsby/packages/gatsby-source-wordpress/src/fetch.js

Lines 557 to 650 in 4af3f91

	for (let key of Object.keys(allRoutes.data.routes)) {
	if (_verbose) console.log(`Route discovered :`, key)
	let route = allRoutes.data.routes[key]
	// A valid route exposes its _links (for now)
	if (route._links) {
	const entityType = getRawEntityType(key)
	// Excluding the "technical" API Routes
	const excludedTypes = [
	`/v2/**`,
	`/v3/**`,
	`**/1.0`,
	`**/2.0`,
	`**/embed`,
	`**/proxy`,
	`/`,
	`/jwt-auth/**`,
	]
	const routePath = getRoutePath(url, key)
	const whiteList = _includedRoutes
	const blackList = [...excludedTypes, ..._excludedRoutes]
	// Check whitelist first
	const inWhiteList = checkRouteList(routePath, whiteList)
	// Then blacklist
	const inBlackList = checkRouteList(routePath, blackList)
	const validRoute = inWhiteList && !inBlackList
	if (validRoute) {
	if (_verbose)
	console.log(
	colorized.out(
	`Valid route found. Will try to fetch.`,
	colorized.color.Font.FgGreen
	)
	)
	const manufacturer = getManufacturer(route)
	let rawType = ``
	if (manufacturer === `wp`) {
	rawType = `${typePrefix}${entityType}`
	}
	let validType
	switch (rawType) {
	case `${typePrefix}posts`:
	validType = refactoredEntityTypes.post
	break
	case `${typePrefix}pages`:
	validType = refactoredEntityTypes.page
	break
	case `${typePrefix}tags`:
	validType = refactoredEntityTypes.tag
	break
	case `${typePrefix}categories`:
	validType = refactoredEntityTypes.category
	break
	default:
	validType = `${typePrefix}${manufacturer.replace(
	/-/g,
	`_`
	)}_${entityType.replace(/-/g, `_`)}`
	break
	}
	validRoutes.push({
	url: buildFullUrl(url, key, _hostingWPCOM),
	type: validType,
	})
	} else {
	if (_verbose) {
	const invalidType = inBlackList ? `blacklisted` : `not whitelisted`
	console.log(
	colorized.out(
	`Excluded route: ${invalidType}`,
	colorized.color.Font.FgYellow
	)
	)
	}
	}
	} else {
	if (_verbose)
	console.log(
	colorized.out(
	`Invalid route: detail route`,
	colorized.color.Font.FgRed
	)
	)
	}
	}

a bit:

• Have first part just create those valid routes (but not filter them out with whitelist / blacklist)
• Then we could have callback to modify our validRoutes list. Currently items in the list have url (endpoint that we will hit) and type (used to determine node type). I think we should add few more fields there:
  • queryParams (IMO it makes sense to structure it more than adding ?x=y to urls)
  • normalize - it seems like there would be need for some postprocessing of fetched entities (at least in WPML case), so we could attach fields like lang, as this doesn't come back in entities when we add something like ?lang=fr / ?lang=en to posts endpoint, and we want to be able to target entities from specific endpoint
• Then apply whitelist/blacklist

Then in user config it would work pretty similar to what's proposed in this PR, just in slight different form:

routesFilter: currentRoutes => {
  // remove initial posts, as it doesn't include lang filter and it will not have `lang` field
  currentRoutes = currentRoutes.filter(route => !route.url.includes(`/wp-json/wp/v2/posts`))

  currentRoutes.push({
    url: `/wp-json/wp/v2/posts`,
    type: `wordpress__POST`,
    queryParams: {
      lang: `en`
    },
    normalize: entities => {
      return entities.map(e => { e.lang = `en`; return e })
    }
  })

  currentRoutes.push({
    url: `/wp-json/wp/v2/posts`,
    type: `wordpress__POST`,
    queryParams: {
      lang: `fr`
    },
    normalize: entities => {
      return entities.map(e => { e.lang = `fr`; return e })
    }
  })
  return currentRoutes
}

pretty verbose, but there's opportunity for some helper function to make it nicer to read

Ultimate goal would be to have something even more automatic that can be used for any WPML site, so user can paste snippet in without any config

routesFilter: async (currentRoutes) => {
  // programatically get list of all languages
  const langs = await getAllLangs()
  const post_types = await getPostTypes()

  const post_routes = []

  // temporarily remove post routes
  currentRoutes = currentRoutes.filter(route => {
    if (thisIsRouteForPostType(route, postTypes)) {
      post_routes.push(route)
	  return false
    }
    return true
  })

  // re-add post routes for each language
  post_routes.forEach(postRoute => {
    langs.forEach(lang => {
      currentRoutes.push({
		url: postRoute.url,
		type: postRoute.type,
		queryParams: {
		  lang: lang
		},
		normalize: entities => {
		  return entities.map(e => { e.lang = lang; return e })
		}
	  })
    })
  })
  return currentRoutes
}

Which hopefully finally would be just packed into subplugin to avoid snippet pasting completely

[willpaige]

Sounds good @pieh I'd love to see that fix released soon. I've got a 're-focus' week next week in which I'll be re-visiting WPML localisation and Gatsby.

[pieh]

I created detailed issue about implementing this feature ( #17943 ) and will be closing this PR for now.