Automate synchronization across repos that use this

[mcmire]

When we make changes to this repo, we need to go into each of our other repos and add those changes to them as well. This is time-consuming and error-prone work, as it must be done manually. Automating such changes would ensure that all of our repos match the template.

In addition, our repos were created at different times and may be in various states of change. It is entirely possible that some repos do not match the template 100%, and currently, we have no way of knowing.

To address this, we should add a script to this repo which lints a project for consistency between that project and this template. If there are any files which are missing, or if there are any major differences in known files that are missing, then the script should report an error. Additionally there should be a --fix option similar to ESLint which will apply any changes which are auto-fixable. Then we can run this script across all of our projects (locally).

Note that we can make the script somewhat intelligent when it comes to certain kinds of files:

• For configuration files which contain JSON or JSON-compatible objects (package.json, GitHub workflow files, etc.), we can determine a difference between objects rather than difference between lines.
• For configuration files which export objects (.eslintrc.js, etc.) we may have to write a codemod to parse the AST of the file, read the object being exported, determine a difference between objects, update the AST, then generate a new file.
• For text-based files we could do a diff. If the diff applies cleanly then the file is autofixable, if not then the developer would have to apply that diff manually.

[mcmire]

I began work on a tool within this repo which will look at a project and recursively merge configuration options within package.json, tsconfig.json, etc. into that project. It's completely WIP and there are no tests but here's what I have so far: af5979e

However we would need a way to run that script across multiple repos. Really we need a way to run any arbitrary command across multiple repos, because I would love to have another script that doesn't make any changes to the project, it just spits out some statistics as compared with the module template. For instance you could run the tool and it would say:

Uses TypeScript? Yes ✅
TypeScript version: 4.6.0 (💡 should upgrade to: 4.7.4)
Node version: 14.x (💡 should upgrade to: 16.x)
Yarn version: 1.x (💡 should upgrade to: 3.x)
`@metamask/eslint-config` version: 9.0.0 (should upgrade to: 10.0.0)
......

Here are some things I found for running commands across multiple repos:

• https://blog.gruntwork.io/introducing-git-xargs-an-open-source-tool-to-update-multiple-github-repos-753f9f3675ec
• https://stackoverflow.com/questions/816619/managing-many-git-repositories
• https://github.com/nosarthur/gita
• https://inside.getyourguide.com/blog/2021/6/14/run-updates-across-multiple-github-repositories-with-our-new-auto-pr-tool (and https://github.com/getyourguide/auto-pr/)

Another idea is to create some sort of GitHub bot that we can install on each repo and it would monitor changes to the module template and run the standardization script as I mentioned above automatically, apply the changes, and automatically create a PR.

There are many ways to skin this cat but those are just some ideas.

[legobeat]

When it comes to repo settings, branches, actions, permissions, etc (basically everything except code), this sounds like a great use-case for Terraform. Anyone aware of if there's been existing conversations on managing GH repos through IaaC?

e.g.

https://registry.terraform.io/providers/integrations/github/latest/docs/resources/repository

Should be possible to combine that with custom tooling like the one you started on.

[mcmire]

@legobeat No I don't believe using Terraform or a similar tool has been mentioned before. I definitely like the declarative nature of these kinds of tools. I'd be curious to see how easy it would be to set up, how it would work in practice, etc., though.

One thing I haven't mentioned which makes synchronizing configuration across repos challenging is allowing for exceptions. For any of our repos that were created using the module template there always seems to be a case where we want to intentionally stray from the module template. A common example of this is in ESLint config files, where we might have a special case such as disabling the node/no-shebang rule in JavaScript files. Another example is that we might want to stick with Node v14 even if the module template supports v16. We should try to move things to the module template where we can to eliminate exceptions, of course, but we likely won't be able to cover all of them. Allowing for exceptions might be seen as an advantage actually, since we can test things out in repos before moving them to the module template.

Given this I'd be curious to know whether Terraform would be able to handle the ability to specify exceptions/overrides for specific pieces of configuration or whether it just overwrites configuration files entirely.

[mcmire]

Had an idea over the weekend for this. In private conversations with Mark we'd discussed the possibility of building similar to Dependabot, that is, creating a bot that runs the checks and posts a summary of the violations it's found in a PR. We could still do that I suppose, but we don't actually need a bot — we just need a GitHub action. This action could run the same checks that I laid out in this comment, then — crucially — it would spit the results to a JSON file and store it in the action artifacts. We then build a dashboard app that uses the GitHub REST API to grab this JSON file through the artifacts of the latest successful run of this action on the main branch across all of the repos where this action is installed. This would give us a 10,000 foot view of all of our repos and where each one lies in terms of synchronization status.

[mcmire]

We now have https://github.com/MetaMask/template-sync, which we can use today to synchronize repos.

It must be noted, however, that there are certain limitations. Namely, it requires manual intervention in the case of conflicts. I think we can do better here; ideally, a synchronization tool should accept that there will be deviations from the module template and allow the user to specify them so that they will get kept when new changes are brought in from the module template.

The other thing to note is that the tool that I've been building (available here: https://github.com/MetaMask/metamask-module-template/tree/automate-standardization-redux) is a little more exhaustive in the sense that it follows a linter-like approach, where smaller pieces can be checked and fixed. And (although this is not built yet) a project may choose to disable any checks it likes.