Add support for GitHub app authentication #269

timja · 2020-01-25T16:07:22Z

JENKINS issue(s):
- JENKINS-57351
Description:
- Adds support for authenticating as a GitHub app (see the jira for more information)
Documentation changes:
- I've added a guide to this repo internally, can easily be ported elsewhere if needed
Users/aliases to notify:
- @bitwiseman

ojacques · 2020-01-26T07:28:43Z

Great! If we wanted to try - would generating a locally built github-api-plugin version 1.102 work?

timja · 2020-01-26T08:22:39Z

You’ll need to compile locally:
GitHub-api
GitHub-api-plugin
This PR

It’s based off 1.101 but version shouldn’t matter too much as long as it’s recent

bitwiseman · 2020-01-28T21:18:50Z

@ojacques @timja You can now switch to just a local build of this PR. The other two items have been merged.

bitwiseman

Looking good. A few questions. Probably a few more once you've added that test coverage you want to.

src/main/java/org/jenkinsci/plugins/github_branch_source/Connector.java

timja · 2020-01-29T07:08:34Z

@bitwiseman I might need some guidance on the tests you would like to see here, I commented out the existing code around auth and all the tests passed so i don’t think there’s any existing coverage

timja · 2020-01-30T18:04:54Z

src/main/java/org/jenkinsci/plugins/github_branch_source/JwtHelper.java

+    private static PrivateKey getPrivateKeyFromString(final String key) throws GeneralSecurityException {
+        if (key.contains("RSA")) {
+            throw new InvalidPrivateKeyException(
+                    "Private key must be a PKCS#8 formatted string, to convert it from PKCS#1 use: "


we could add a dependency to bouncy castle then we could handle this? or ask users to convert it?

Let's start with asking user to do this. Smoothing can be a later release.

There is a Bouncy Castle API Plugin that was split from Jenkins core in the past and has a wide install base that you can depend on for this kind of thing, but I agree that it seems best to save that for later.

timja · 2020-02-01T15:13:42Z

Great! If we wanted to try - would generating a locally built github-api-plugin version 1.102 work?

@ojacques did you get a chance to try this? I've just pushed documentation for it
https://github.com/timja/github-branch-source-plugin/blob/github-app-support/docs/github-app.adoc

ojacques · 2020-02-02T19:16:12Z

@ojacques did you get a chance to try this? I've just pushed documentation for it
https://github.com/timja/github-branch-source-plugin/blob/github-app-support/docs/github-app.adoc

@timja, thanks for everything, and for the doc (I will suggest some modifications about it later).
Unfortunately, I was not able to get this going. Note that this is with a GitHub enterprise instance, but that should not matter.

I'm getting this:

It looks like this is failing in Connector.java, at GHApp app = gitHubApp.getApp();.

145 is my github app ID (which is not the installation id).
I "installed" the app in the github organization I want to get access to, but that did not make a difference.

My test environment is: github enterprise, the github-branch-source-plugin compiled fresh from this PR and Jenkins 2.219 running in a container.

Any idea?

timja · 2020-02-02T19:18:02Z

Do you know what version of GHE you are on?

I've pushed more logging, which might help debug it

ojacques · 2020-02-02T19:18:19Z

Do you know what version of GHE you are on?

2.19.6

timja · 2020-02-02T19:24:15Z

Could you try re-build a fresh version, (or wait for CI which will give you one too if you just want to download it)

I've added additional logging to help debug it

ojacques · 2020-02-02T19:33:31Z

It looks like it tries to talk to api.github.com, not the github enterprise URL that is configured:

Exception validating credentials 145
java.io.FileNotFoundException: https://api.github.com/app
	at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1896)
	at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1498)
	at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
	at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:352)
	at org.kohsuke.github.Requester._fetchOrRetry(Requester.java:523)
Caused: java.io.FileNotFoundException: https://api.github.com/app

The end point should be https://github.xyz.com/api/v3, which is the one that is selected as the API end point, just above "credentials" drop down.

timja · 2020-02-02T21:11:53Z

Should be fixed now, please re test

ojacques · 2020-02-03T09:35:51Z

Thanks @timja, update from my testing:

✔ I can scan a github org (GitHub enterprise) with Jenkins as a GitHub app! 🎉 🎆
For the organization scan to fully work, I had to give the GitHub app more "repository" type permissions:
- Contents: read-only (to read the Jenkinsfile, and I was expecting also to be able to get the code from the repo)
- Metadata: Read-only (was set automatically)
- Pull requests: read-only (without this one, the scan was not fully completing as it looks through PRs / branches to create all the jobs)
- Commit status: read-write (this one is as per your doc)
❌ Yet, the actual jobs are failing: they can't seem to get repo content from GitHub:

For that git phase, the credential used seems to be an ssh key (using GIT_SSH to set credentials) - probably the one set for the GitHub app. Shouldn't this be a token instead (which is what we get from a GitHub app)?

Great progress in any case!

timja · 2020-02-04T13:55:46Z

@ojacques I've pushed a temporary fix, not sure how this was working for me locally, but it's not anymore.

docs/github-app.adoc

ojacques · 2020-02-04T20:35:40Z

Fetching repo / running jobs still does not work:

I could be available on slack or gitter if needed.

After some discussions and more tests:

This PR does work "as is" with github.com and public repos
It needs more work to support private repos on github.com or GitHub enterprise in "private mode"

src/main/java/org/jenkinsci/plugins/github_branch_source/Connector.java

timja · 2020-02-04T21:56:24Z

We've had a chat on gitter, the GHE issue can be re-produced on Github.com by trying to checkout a private repo, for some reason the creds aren't being used on checkout

oleg-nenashev · 2020-03-04T20:34:43Z

I am fine. We can always generalize it later if needed

oleg-nenashev · 2020-03-04T23:29:19Z

FYI @sladyn98 @uhafner, the functionality created here could be reused in the GitHub Checks API integration project. https://jenkins.io/projects/gsoc/2020/project-ideas/github-checks/

sladyn98 · 2020-03-05T15:15:37Z

@oleg-nenashev Yes this is definitely reusable, I used a part of this code in my POC for the Github Checks API Plugin.https://github.com/sladyn98/JenkinsChecksPOC

ojacques · 2020-03-05T18:45:50Z

FYI @sladyn98 @uhafner, the functionality created here could be reused in the GitHub Checks API integration project. https://jenkins.io/projects/gsoc/2020/project-ideas/github-checks/

Yes. Here is a PoC which posts a checks run using a valid authentication token issued from Jenkins as GitHub App, all with Jenkinsfile:

pipeline {
  agent any

  stages{
    stage('Check run PoC') { 
      steps {
        withCredentials([usernamePassword(credentialsId: 'githubapp-jenkins', usernameVariable: 'GITHUB_APP', passwordVariable: 'GITHUB_JWT_TOKEN')]) {
        sh '''
        curl -H "Content-Type: application/json" -H "Accept: application/vnd.github.antiope-preview+json" -H "authorization: Bearer ${GITHUB_JWT_TOKEN}" -d '{ "name": "check_run", "head_sha": "'${GIT_COMMIT}'", "status": "in_progress", "external_id": "42", "started_at": "2020-03-05T11:14:52Z", "output": { "title": "Check run from Jenkins!", "summary": "This is a check run which has been generated from Jenkins as GitHub App", "text": "...and that iss awesome"}}' https://github.xyz.com/api/v3/repos/ojacques/jenkins-with-githubapp/check-runs
        '''
        }
      }
    }
  }
}

timja · 2020-03-05T19:00:05Z

FYI @sladyn98 @uhafner, the functionality created here could be reused in the GitHub Checks API integration project. jenkins.io/projects/gsoc/2020/project-ideas/github-checks

Yes. Here is a PoC which posts a checks run using a valid authentication token issued from Jenkins as GitHub App, all with Jenkinsfile:

pipeline {
  agent any

  stages{
    stage('Check run PoC') { 
      steps {
        withCredentials([usernamePassword(credentialsId: 'githubapp-jenkins', usernameVariable: 'GITHUB_APP', passwordVariable: 'GITHUB_JWT_TOKEN')]) {
        sh '''
        curl -H "Content-Type: application/json" -H "Accept: application/vnd.github.antiope-preview+json" -H "authorization: Bearer ${GITHUB_JWT_TOKEN}" -d '{ "name": "check_run", "head_sha": "cd2cf268e59fdf2e11fb677a4ca1a53f58826e0c", "status": "in_progress", "external_id": "42", "started_at": "2020-03-05T11:14:52Z", "output": { "title": "Check run from Jenkins!", "summary": "This is a check run which has been generated from Jenkins as GitHub App", "text": "...and that iss awesome"}}' https://github.xyz.com/api/v3/repos/ojacques/jenkins-with-githubapp/check-runs
        '''
        }
      }
    }
  }
}

Nice!

It'll be great once it's fully integrated,

tests showing up there
warnings on PR
lint results
ability to re-run

etc

uhafner · 2020-03-05T19:32:57Z

Yes, that looks very promising!

sladyn98 · 2020-03-06T19:12:29Z

@ojacques Has this (reporting to Checks API) already been implemented as a Pipeline step
cc @timja

droidpl · 2020-03-10T09:49:36Z

When do you plan to create a release with this?

ojacques · 2020-03-10T12:33:06Z

@ojacques Has this (reporting to Checks API) already been implemented as a Pipeline step

We did not yet, but were looking to start.

sladyn98 · 2020-03-10T12:38:18Z

@ojacques okay cool, i would love to contribute, since it is a part of my proposal for the github Checks API

oleg-nenashev · 2020-03-17T17:25:15Z

@timja I will try to facilitate the release. Just in case, would you be interested to write a blogpost about this feature or to present it at one of the next Jenkins Online Meetups?

timja · 2020-03-17T19:47:37Z

@timja I will try to facilitate the release. Just in case, would you be interested to write a blogpost about this feature or to present it at one of the next Jenkins Online Meetups?

yes, either / both.

timja · 2020-03-20T10:57:40Z

@timja I will try to facilitate the release. Just in case, would you be interested to write a blogpost about this feature or to present it at one of the next Jenkins Online Meetups?

Draft blog post:
jenkins-infra/jenkins.io#2988

docs/github-app.adoc

src/main/java/org/jenkinsci/plugins/github_branch_source/GitHubAppCredentials.java

kad-kooija · 2020-04-09T10:56:12Z

When will this be released?

steve-norman-rft · 2020-04-21T09:04:29Z

https://jenkins.io/blog/2020/04/16/github-app-authentication/

bitwiseman · 2020-04-22T23:45:00Z

General release in the next week.

timja requested a review from bitwiseman January 25, 2020 16:07

timja mentioned this pull request Jan 25, 2020

Support all available endpoints for Github App with preview request hub4j/github-api#570

Closed

oleg-nenashev self-requested a review January 27, 2020 16:46

bitwiseman reviewed Jan 29, 2020

View reviewed changes

src/main/java/org/jenkinsci/plugins/github_branch_source/Connector.java Outdated Show resolved Hide resolved

src/main/java/org/jenkinsci/plugins/github_branch_source/Connector.java Outdated Show resolved Hide resolved

timja commented Jan 30, 2020

View reviewed changes

timja marked this pull request as ready for review January 30, 2020 18:05

timja force-pushed the github-app-support branch from 2875181 to 8f06734 Compare February 4, 2020 14:11

ojacques reviewed Feb 4, 2020

View reviewed changes

docs/github-app.adoc Show resolved Hide resolved

ojacques reviewed Feb 4, 2020

View reviewed changes

src/main/java/org/jenkinsci/plugins/github_branch_source/Connector.java Outdated Show resolved Hide resolved

timja and others added 6 commits February 5, 2020 09:06

Add support for GitHub app authentication

a277a71

Pass api url through to fix GHE

96bfe13

Add documentation

5225a5d

More logging on failure

42e6826

Update github-app.adoc

8010d4d

Update github-app.adoc

9415745

dwnusbaum approved these changes Mar 4, 2020

View reviewed changes

bitwiseman merged commit b42dd6b into jenkinsci:master Mar 4, 2020

timja deleted the github-app-support branch March 5, 2020 15:17

timja mentioned this pull request Mar 20, 2020

GitHub app blog jenkins-infra/jenkins.io#2988

Merged

jglick added the enhancement label Mar 20, 2020

jglick mentioned this pull request Mar 20, 2020

Fix bad merge #286

Merged

jglick reviewed Mar 20, 2020

View reviewed changes

docs/github-app.adoc Show resolved Hide resolved

jglick reviewed Mar 20, 2020

View reviewed changes

src/main/java/org/jenkinsci/plugins/github_branch_source/GitHubAppCredentials.java Show resolved Hide resolved

jglick mentioned this pull request Mar 24, 2020

Skip GitHubOrgWebHook.register when using a GH app receiving all relevant events #289

Draft

timja mentioned this pull request Mar 24, 2020

Allow GitHub app to be installed in multiple orgs #290

Merged

6 tasks

jglick mentioned this pull request Mar 26, 2020

Cache app installation token #291

Merged

8 tasks

This was referenced Jan 6, 2021

Upgrade to use jcasc test harness #266

Merged

Extract dep on jjwt-api plugin #366

Merged

jglick mentioned this pull request Mar 23, 2022

[JENKINS-62220] Automatically select owner for GitHubAppCredentials acc. to context #527

Merged

Add support for GitHub app authentication #269

Add support for GitHub app authentication #269

Conversation

timja commented Jan 25, 2020 • edited Loading

ojacques commented Jan 26, 2020

timja commented Jan 26, 2020

bitwiseman commented Jan 28, 2020

bitwiseman left a comment

Choose a reason for hiding this comment

timja commented Jan 29, 2020

timja Jan 30, 2020

Choose a reason for hiding this comment

bitwiseman Feb 18, 2020

Choose a reason for hiding this comment

dwnusbaum Feb 19, 2020

Choose a reason for hiding this comment

timja commented Feb 1, 2020

ojacques commented Feb 2, 2020

timja commented Feb 2, 2020 • edited Loading

ojacques commented Feb 2, 2020

timja commented Feb 2, 2020

ojacques commented Feb 2, 2020

timja commented Feb 2, 2020

ojacques commented Feb 3, 2020

timja commented Feb 4, 2020

ojacques commented Feb 4, 2020 • edited Loading

timja commented Feb 4, 2020

oleg-nenashev commented Mar 4, 2020

oleg-nenashev commented Mar 4, 2020

sladyn98 commented Mar 5, 2020

ojacques commented Mar 5, 2020 • edited Loading

timja commented Mar 5, 2020

uhafner commented Mar 5, 2020

sladyn98 commented Mar 6, 2020

droidpl commented Mar 10, 2020

ojacques commented Mar 10, 2020

sladyn98 commented Mar 10, 2020

oleg-nenashev commented Mar 17, 2020

timja commented Mar 17, 2020

timja commented Mar 20, 2020

kad-kooija commented Apr 9, 2020

steve-norman-rft commented Apr 21, 2020

bitwiseman commented Apr 22, 2020

timja commented Jan 25, 2020 •

edited

Loading

timja commented Feb 2, 2020 •

edited

Loading

ojacques commented Feb 4, 2020 •

edited

Loading

ojacques commented Mar 5, 2020 •

edited

Loading