[gatsby-source-wordpress] add config option to manipulate list of endpoint we fetch

[pieh]

There is need for users to be able to manipulate list of endpoints we get from wp rest introspection endpoint. This includes setting query params on routes and adding/removing routes.

Use cases for it are:

• adding support for WPML translations (see gatsby-source-wordpress create pages programmatically for translated pages with wpml #10915, [gatsby-source-wordpress] switching languages using WPML #3263, [gatsby-source-wordpress] multilingual support (WPML) #1648 ) (setting lang query param on post types endpoints + hitting them for each language)
• being able to pull in draft / private posts ( feat(gatsby-source-wordpress): Add custom routes with parameters #10942 (comment) )

There were few attempts to implement this ( #10942 , #10335 ), but those implementations didn't allow for general purpose, robust system (they were mostly geared toward fixing particular use case)

I'd like to propose solution that is similar to wordpress filters, where user will be able to programatically manipulate list of endpoints with config option (and is similar to how normalizer config option works), where it's optional function that get list of endpoints, do some processing on it and return modified list to wordpress plugin:

{
      resolve: "gatsby-source-wordpress",
      options: {
        // [...] rest of config
        routesFilter: async ({ routes }) => {
         // do some work on routes array
          return routes
        }
      }

routesFilter is not final name of the config option - please comment on better name for it.

routes would be array of objects with following fields:

{
  // required fields
  "url", "http://example.com/wp-json/wp/v2/pages", // url to endpoint - possibly this wouldn't be full absolute url and just "/wp/v2/pages" so users don't have to have special cases for wordpress.com hosting which have a bit different endpoints than user hosted wordpress instance
  "type": "wordpress__PAGE", // type used to generated gatsby data nodes
  // optional fields
  "queryParams": {
    "lang": "es"
  }, // optional object with params that will be used when fetching
  "normalizer": ({ entities }) => {
    // manipulate entites that were created after fetching this route - useful to set additional meta fields (like lang)
    return entities
  }, // optional function
}

Ultimate goal here is to create API that later on can be packaged into reusable gatsby-source-wordpress subplugins (see gatsbyjs/rfcs#23 ). Creating subplugin system wouldn't be in scope for initial implementation. This features is just first step on path to be able to do this.

Implementation:

Split

gatsby/packages/gatsby-source-wordpress/src/fetch.js

Lines 557 to 650 in 4af3f91

	for (let key of Object.keys(allRoutes.data.routes)) {
	if (_verbose) console.log(`Route discovered :`, key)
	let route = allRoutes.data.routes[key]
	// A valid route exposes its _links (for now)
	if (route._links) {
	const entityType = getRawEntityType(key)
	// Excluding the "technical" API Routes
	const excludedTypes = [
	`/v2/**`,
	`/v3/**`,
	`**/1.0`,
	`**/2.0`,
	`**/embed`,
	`**/proxy`,
	`/`,
	`/jwt-auth/**`,
	]
	const routePath = getRoutePath(url, key)
	const whiteList = _includedRoutes
	const blackList = [...excludedTypes, ..._excludedRoutes]
	// Check whitelist first
	const inWhiteList = checkRouteList(routePath, whiteList)
	// Then blacklist
	const inBlackList = checkRouteList(routePath, blackList)
	const validRoute = inWhiteList && !inBlackList
	if (validRoute) {
	if (_verbose)
	console.log(
	colorized.out(
	`Valid route found. Will try to fetch.`,
	colorized.color.Font.FgGreen
	)
	)
	const manufacturer = getManufacturer(route)
	let rawType = ``
	if (manufacturer === `wp`) {
	rawType = `${typePrefix}${entityType}`
	}
	let validType
	switch (rawType) {
	case `${typePrefix}posts`:
	validType = refactoredEntityTypes.post
	break
	case `${typePrefix}pages`:
	validType = refactoredEntityTypes.page
	break
	case `${typePrefix}tags`:
	validType = refactoredEntityTypes.tag
	break
	case `${typePrefix}categories`:
	validType = refactoredEntityTypes.category
	break
	default:
	validType = `${typePrefix}${manufacturer.replace(
	/-/g,
	`_`
	)}_${entityType.replace(/-/g, `_`)}`
	break
	}
	validRoutes.push({
	url: buildFullUrl(url, key, _hostingWPCOM),
	type: validType,
	})
	} else {
	if (_verbose) {
	const invalidType = inBlackList ? `blacklisted` : `not whitelisted`
	console.log(
	colorized.out(
	`Excluded route: ${invalidType}`,
	colorized.color.Font.FgYellow
	)
	)
	}
	}
	} else {
	if (_verbose)
	console.log(
	colorized.out(
	`Invalid route: detail route`,
	colorized.color.Font.FgRed
	)
	)
	}
	}

into few stages:

• fist part that creates routes list (but doesn't do whitelist/blacklist filtering)
• execute filterRoutes function (if it is defined)
• apply whitelist/blacklist filters

Adjust

gatsby/packages/gatsby-source-wordpress/src/fetch.js

Lines 396 to 399 in 4af3f91

	url: `${url}?${querystring.stringify({
	per_page: _perPage,
	page: page,
	})}`,

to use optional queryParams object in route object

Figure out where in

gatsby/packages/gatsby-source-wordpress/src/fetch.js

Lines 252 to 369 in 4af3f91

	async function fetchData({
	route,
	apiUrl,
	_verbose,
	_perPage,
	_auth,
	_cookies,
	_accessToken,
	_concurrentRequests,
	}) {
	const { type, url, optionPageId } = route
	if (_verbose) {
	console.log(
	colorized.out(
	`=== [ Fetching ${type} ] ===`,
	colorized.color.Font.FgBlue
	),
	url
	)
	console.time(`Fetching the ${type} took`)
	}
	let routeResponse = await getPages({
	url,
	_perPage,
	_auth,
	_cookies,
	_accessToken,
	_verbose,
	_concurrentRequests,
	})
	let entities = []
	if (routeResponse) {
	if (type.indexOf(`wordpress__menus_menus`) !== -1) {
	routeResponse = routeResponse.map(r => {
	return { ...r, ID: r.term_id }
	})
	}
	// Process entities to creating GraphQL Nodes.
	if (Array.isArray(routeResponse)) {
	routeResponse = routeResponse.map(r => {
	return {
	...r,
	...(optionPageId ? { __acfOptionPageId: optionPageId } : {}),
	__type: type,
	}
	})
	entities = entities.concat(routeResponse)
	} else {
	routeResponse.__type = type
	if (optionPageId) {
	routeResponse.__acfOptionPageId = optionPageId
	}
	entities.push(routeResponse)
	}
	// WordPress exposes the menu items in meta links.
	if (type === `wordpress__wp_api_menus_menus`) {
	for (let menu of routeResponse) {
	if (menu.meta && menu.meta.links && menu.meta.links.self) {
	entities = entities.concat(
	await fetchData({
	route: {
	url: useApiUrl(apiUrl, menu.meta.links.self),
	type: `${type}_items`,
	},
	apiUrl,
	_verbose,
	_perPage,
	_auth,
	_cookies,
	_accessToken,
	})
	)
	}
	}
	}
	// Menu nodes for WP-REST-API V2 Menus ( https://wordpress.org/plugins/wp-rest-api-v2-menus/ )
	if (type === `wordpress__menus_menus`) {
	for (let menu of routeResponse) {
	entities = entities.concat(
	await fetchData({
	route: { url: `${url}/${menu.term_id}`, type: `${type}_items` },
	_verbose,
	_perPage,
	_auth,
	_accessToken,
	_cookies,
	})
	)
	}
	}
	// TODO : Get the number of created nodes using the nodes in state.
	let length
	if (routeResponse && Array.isArray(routeResponse)) {
	length = routeResponse.length
	} else if (routeResponse && !Array.isArray(routeResponse)) {
	length = Object.keys(routeResponse).length
	}
	console.log(
	colorized.out(
	` -> ${type} fetched : ${length}`,
	colorized.color.Font.FgGreen
	)
	)
	}
	if (_verbose) {
	console.timeEnd(`Fetching the ${type} took`)
	}
	return entities
	}

to apply optional normalizer function for given route (this is a bit tricky, because gatsby-source-wordpress already have special support for some wordpress plugins - do we want to apply normalization before or after those special cases?)

References:
See #10942 (comment) comment (that contains some of information in this issue) about motivation and "roadmap" to be able to create wordpress subplugins in smaller incremental steps

[andresgutgon]

@pieh I did a PR following your recommendations 😄 #19144

Please let me know how I can improve the code in order to merge it into master.

Thanks!

[gatsbot]

Hiya!

This issue has gone quiet. Spooky quiet. 👻

We get a lot of issues, so we currently close issues after 30 days of inactivity. It’s been at least 20 days since the last update here.

If we missed this issue or if you want to keep it open, please reply here. You can also add the label "not stale" to keep this issue open!

As a friendly reminder: the best way to see this issue, or any other, fixed is to open a Pull Request. Check out gatsby.dev/contribute for more information about opening PRs, triaging issues, and contributing!

Thanks for being a part of the Gatsby community! 💪💜

[github-actions]

Hey again!

It’s been 30 days since anything happened on this issue, so our friendly neighborhood robot (that’s me!) is going to close it.
Please keep in mind that I’m only a robot, so if I’ve closed this issue in error, I’m HUMAN_EMOTION_SORRY. Please feel free to reopen this issue or create a new one if you need anything else.
As a friendly reminder: the best way to see this issue, or any other, fixed is to open a Pull Request. Check out gatsby.dev/contribute for more information about opening PRs, triaging issues, and contributing!

Thanks again for being part of the Gatsby community! 💪💜

[LekoArts]

With v4 of gatsby-source-wordpress now being powered by WPGraphQL and not the REST API anymore we'll close outdated issues that we won't fix anymore. Please use the new version and check if your issue is fixed. Thanks!