CI History and tracability

[refack]

With nodejs/node#21059 we can see that Travis has better history retention and the new GitHub "checks API" does not disappear after the PR is closed (both of these should be taken into consideration for when/if we duplicate auto CI with Jenkins)

ATM Jenkins purges build logs quite often, and GitHub checklist at the bottom of the thread disappears when the PR is closed/merged.

[joyeecheung]

Is that something we can fix in Jenkins or with GitHub API?

I tend to use ncu-ci --copy to summarize the failures of CI and post them on the thread now, that also works, just takes a bit more manual effort.

[refack]

Is that something we can fix in Jenkins or with GitHub API?

1. The new checks API allows us to post more information to the PR
2. We need to either do that in Jenkins, or at least keep more logs.

[richardlau]

Checks API would seem to be a nice long term solution, but right now I think it is still limited to GitHub Apps.

[mhdawson]

The Jenkins history is purged relatively quickly due to the storage that was consumed by each of the builds. @jbergstroem might have more specific information.

[joaocgreis]

IIRC having a bigger Jenkins history is blocked on nodejs/build#453 . History is so limited because when opening a job Jenkins parses the output of all runs and this is slow with TAP.

[refack]

IIRC having a bigger Jenkins history is blocked on nodejs/build#453 . History is so limited because when opening a job Jenkins parses the output of all runs and this is slow with TAP.

That is good news, since I'm slowly migrating the jobs to JUnit (with either tap2junit and/or a JUnit reporter). I'll prioritize that so we can eliminate the use on the Jenkins TAP plugin.

[joyeecheung]

In addition we could use ncu-ci walk or part of the code to store the relevant bits periodically. It can generate a JSON that contains all the failures (including bulid failures) in the last 100 runs it parses in a JSON like this, which is not too big because it only store the context of the failures instead of all the logs.