Switch to a custom Github-Api module

[carterm]

The github-api module is causing many exceptions related to "Cannot read property 'status' of undefined" when there is a fetch issue. Now that we are using tree updates only it would be easy to switch back to a custom module that doesn't need all the other github-api features.

This would incorporate all the features in the current gitTreeCommon module, that is used by multiple projects.

New module features

• Take content map (filename + content) and compare it with a GitHub project. Create a commit that updates the project.
• Option to include the commits in a PR
• Option to add assignees to the PR
• Uses the GitHub hashing algorithm to improve the ability to determine binary file changes and prevent unnecessary content uploads.

@cagov\github-sync ?

For custom REST examples, use our old GitHub API connector here

[aaronhans]

Makes sense, glad we tried the github module, shame it isn't well supported

[carterm]

Latest issue...
github-tools/github#439

[carterm]

The new github module will focus on tree updates, and only support the github api functions that we use on a regular basis. Considering names for a new repo. cagov/github-connector

[chachasikes]

Any updates on this?

[chachasikes]

@carterm - I have a couple of questions:
I see the thread on github-tools.

I've handled a similar issue before, and introduced a fetch timeout (on our end).
The issue from github-tools is requesting that they handle network detection, which would be nice - but it's also something we can do.

Q: can you share the lines of code you are proposing to replace (so we can all see them?)

I know you were just asking about what to name the module. It's helpful to see the code. Not sure it needs to be a module yet.

[carterm]

This would be a replacement for all references to github-api.

[chachasikes]

Ah.

[carterm]

This would incorporate all the features in the current gitTreeCommon module, that is used by multiple projects.
https://github.com/cagov/wordpress-to-github/blob/main/wordpress-to-github/gitTreeCommon/index.js

[chachasikes]

(context: Carter is asking us in Slack about what to name a module.)

https://github.com/github-tools/github - "provides a minimal higher-level wrapper around Github's API."

So you are talking about parsing data from the github-api for this module for wordpress-to-github.

To name it, I would look through the code you wrote, & share with us a bullet pointed list of the main features that are going to go into this new module. Then we can name it better.

For example:

• Get github tree data and check for recent commits
• Process sha to see if we need to create a new pull request.

What else?

[carterm]

• Push commits
• Push commits as Pull request
• Add issue assignees
• get commit reports from actions that can be used for logging

[carterm]

Take a list of content -> compare content to GitHub -> make updates where needed -> list what was changed

[chachasikes]

Great - so trying to come up with some names for the features you have described:

(brainstorming)

• github-content-api-pr
• github-pr-from-content-api
• github-headless-diffs
• github-content-diff
• github-content-api-diff

PR's are an important part of this right?
Also - would this module be useful to the data pipelines?

[carterm]

PRs are an option, but the main thing is synchronizing files. It could just be a commit.

[chachasikes]

• github-sync-content-api
• github-api-sync

I'm assuming the source is an API. It definitely is for what we are doing & that might be okay as a specialty.

Does this include the media handling, or can that be a wordpress specific add on that's part of this repo?

[carterm]

API seems overused. Everything uses an API. All types of files, binaries too.

[carterm]

It's also for pushing content syncs into GitHub, not for syncing a file system download like a pull would do.

[chachasikes]

If the interface is an api, it's okay to overuse it - if it's designed to work with APIs. (Which, as you saw in my presentation, is overused but not understood - also "content APIs" is underutilized and we might use that term more if it's more clear to people we are working with - and that's one of our most important features.

More questions:

• Is it (in State context) only for github with WordPress API (or also Snowflake or other API commonly used by the State?)
• Can it read a binary folder directory, or does it take an array of paths and then downloads them, diffing binaries as needed?

[carterm]

• The only dependency would be GitHub as a service (via using its API).
• The current gitTreeCommon has the ability to generate hashes from binary files that match the GitHub hashes, which greatly improve the update process. Determining the hashes of the source files still requires downloading them from the content source, unless there was a way to push the GitHub hashes back to the source files (like WordPress).

[carterm]

We had a custom GitHub connector before using the NPM module. Here is the last version...
https://github.com/cagov/wordpress-integration-api/blob/dc4fe93267079ccae1ff5a77b0149ceacea50fd2/WordPressService/gitHub.js

[chachasikes]

Another thought question: What are the inputs/interfaces to this module - a bunch of arrays?

I found this:

    const newTree = await createTreeFromFileMap(
      gitRepo,
      gitHubTarget.Branch,
      fileMap,
      filePath
    );

And this interface:

* @param {*} gitRepo from github-api
* @param {string} masterBranch usually "master" or "main"
* @param {Map<string,any>} filesMap contains the data to push
* @param {string} outputPath the root path for all files
* @param {boolean} [cleanoutputPath] true to delete all unmatched files in outputPath

This is helpful

[chachasikes]

More terrible options
(github-sync is close but it feels like one more specifier & a little more context awareness will help)

More brainstorming:

• github-sync-map
• github-push-data-map - it's a one way sync (pushing into github), right?
• github-data-push
• github-map-push - (easy to confuse with maps)
• github-api-push
• github-content-push
• github-content-api-push - it is only for content / designed for content API's right?

More use cases to think about:

• non-wordpress APIs (headless CMSs that are not wordpress)
• data APIs yes or no?

[carterm]

One important key component is that all the updates will be performed using Git Trees, which are much more transactional for updates and reduce the amount of conflicts for large updates. They work fine for single file updates too. So maybe tree can be in the name? But, that's transparent to the user of this module. So maybe not.

I'm not a fan of putting API in the name because it's just a way that computers talk, and unless understanding how the API works is a pre-requisite for use, I'd suggest we leave it out.

This is CMS agnostic. Downloading content from the CMS (custom data API or not) is outside of the scope of this module.

My favorites from your list...

• github-data-push
• github-content-push

I'm thinking for keywords, for 3rd parties looking to use this, tree might be good to have in there. So possibly...

• github-tree-push
• github-tree-planter

[carterm]

Deepwork start here...
https://github.com/cagov/wordpress-to-github/blob/starting-github-tree-push/github-tree-push/index.js

[carterm]

Going to start creating the repo with github-tree-push as the name. Will hold off on publishing to NPM.

[carterm]

Work has moved to new repo...
https://github.com/cagov/github-tree-push

[carterm]

New module deployed!
https://www.npmjs.com/package/@cagov/github-tree-push