Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Search and replace the hostname in URLs. #3498

Merged
merged 9 commits into from
Jan 14, 2018
Merged

Search and replace the hostname in URLs. #3498

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
e8b1300
Search and replace the hostname in URLs.
Jan 12, 2018
9ee6877
Fixed code styling issues
Jan 12, 2018
4985550
Missed backticks for the timers requirement
Jan 12, 2018
cfe0d9d
Updated readme with search and replace options
Jan 12, 2018
98d7dae
Removed async, wrapped json parse in a try catch and updated blacklis…
Jan 13, 2018
01b1a5b
Updated search and replace api and updated documentation
Jan 13, 2018
4d2e510
Added logging for invalid search and replace option
Jan 13, 2018
829b1f2
make option more explicit, updated default value & removed redundant …
Jan 13, 2018
4b94bbf
Updated the variable name making it clearer
Jan 13, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions packages/gatsby-source-wordpress/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -83,6 +83,8 @@ plugins: [
// Set verboseOutput to true to display a verbose output on `npm run develop` or `npm run build`
// It can help you debug specific API Endpoints problems
verboseOutput: false,
// Search and Replace Urls across WordPress content
searchReplace: ["https://source-url.com", "https://replacement-url.com"]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we change this to use named fields e.g.

searchReplace: {
  sourceUrl: `https://source-url.com`,
  replacementUrl: `https://replacement-url.com`,
},

I find array-based APIs really ambiguous as the position of an item in an array isn't much of a cue to how you should use it or later, what it even means.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Search & Replace option has been updated to accept an object rather than an array

},
},
];
Expand Down
7 changes: 7 additions & 0 deletions packages/gatsby-source-wordpress/src/gatsby-node.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@ exports.sourceNodes = async (
auth = {},
verboseOutput,
perPage = 100,
searchReplace = [],
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should be {}

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oops! missed this 👎

}
) => {
const { createNode } = boundActionCreators
Expand Down Expand Up @@ -92,6 +93,12 @@ exports.sourceNodes = async (
createNode,
})

// Search and replace Content Urls
entities = await normalize.searchReplaceContentUrls({
entities,
searchReplace,
})

// creates nodes for each entry
normalize.createNodesFromEntities({ entities, createNode })

Expand Down
35 changes: 35 additions & 0 deletions packages/gatsby-source-wordpress/src/normalize.js
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
const { setTimeout } = require(`timers`)
const crypto = require(`crypto`)
const deepMapKeys = require(`deep-map-keys`)
const _ = require(`lodash`)
Expand Down Expand Up @@ -234,6 +235,40 @@ exports.mapTagsCategoriesToTaxonomies = entities =>
return e
})

exports.searchReplaceContentUrls = async function ({ entities, searchReplace }) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

these functions don't need to be async — unless you're doing file or network IO this should be a normal synchronous function.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Your right, I am normally used to writing non blocking code for node.. I'll make this change 👍


if (!Array.isArray(searchReplace) && searchReplace.length !== 2) {
return entities
}

const [search, replace] = searchReplace

const _blacklist = [
`_links`,
`__type`,
]

const blacklistProperties = function (obj = {}, blacklist = {}) {
for (var i = 0; i < blacklist.length; i++) {
eval(`delete obj.${blacklist[i]}`)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you could just do delete obj[blacklist[i]

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I used eval here because I wanted to account for a scenario where the blacklist goes deeper than one level for example: 'content.rendered'.

If you think we will never have to blacklist data in the object deeper than level one I can make this change :)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we extend the blacklist we could change the code.

}

return obj
}

const final = entities.map(async (entity) => new Promise((resolve, reject) => {
setTimeout(() => {
var whiteListedEntities = JSON.stringify(blacklistProperties(entity, _blacklist))
var replacedString = whiteListedEntities.replace(new RegExp(search, `g`), replace)
var parsed = JSON.parse(replacedString)
resolve(_.defaultsDeep(parsed, entity))
}, 0)
})
)

return await Promise.all(final)
}

exports.mapEntitiesToMedia = entities => {
const media = entities.filter(e => e.__type === `wordpress__wp_media`)

Expand Down