Get CI running reliably

[isovector]

The burden of maintaining CI is getting to the point of being overwhelming for our rag-tag band of plucky volunteers. Discussing this today on IRC, @bgamari generously offered to let us use the GHC CI infrastructure, which would give us access to beefy, dedicated machines.

Gitlab CI supports being run from Github, as documented here: https://docs.gitlab.com/ee/ci/ci_cd_for_external_repos/github_integration.html

And Ben has an old branch with an attempt at this: https://gitlab.haskell.org/bgamari/haskell-language-server/-/tree/wip/gitlab-ci

[jneira]

Iirc not long ago the ghc gitlab was needing more resources cause all the ghc builds takes a huge amount of them. has that situation changed?

Otoh the major burden is fix ci itself and that can be done by maintainers of hls, wherever the builds are running on, no? How run the ci in gitlab would save out time for the last ci breaks?

[bgamari]

has that situation changed?

Yes, we have restructured our CI pipelines to reduce load and now have more resources thanks to support from Microsoft Azure.

[jneira]

Those are really great news, congrats. I am very gratefull for the offer but i dont think move ci would save developers time, the most valued resource we have. And it would open some questions: for example, will gitlab builds be integrated with github branch protection rules? Now we cant merge a pr if the ci is not green, not sure if gitlab ci could be integrated.

So i would not move main ci if there is no a convincing argument about saving maintainers time.

[isovector]

@jneira my argument in favor of it is that I've been blocked on broken CI for like three weeks now.

First there was the cachix outage due to our CI depending on some guy's nix store, and everything exploding if he ever decides to garbage collect. Then our CI machines started crashing while building, due to not having enough memory. The fix was to make everything dynamically linked. But now the CI cache is broken.

Every time I go to add a feature, I end up spending 5x longer on getting it through CI than I do on the feature.

Perhaps a better title for this bug would be Get CI running reliably, toward which using GHC's CI could be a possibility. But I don't really care how it gets done, so long as I can go back to working on software and not doing devops!

[pepeiborra]

I agree with all the points made by @jneira. Very few people enjoy doing DevOps in their free time, and I for one don't want anything to do with this migration.

@isovector if you or someone else wants to finish Ben's work and add a new CI backend to the current Github Actions and CircleCI, that's great. Maybe initially focus on replacing only the stack builds in CircleCI, since those are the ones giving the most trouble.

[jneira]

yeah, agree stack builds are a good candidate but please, keeping the pr required checks to merge

[isovector]

Very few people enjoy doing DevOps in their free time

Agreed. I like doing devops as much as the next person, but I sure seem to be doing a lot of it as of late! Could we just put the opencollective money towards hiring someone to do devops?

[jneira]

Hmm the tricky thing is the combination of dev and op. Some issues are caused directly by the underlying vm and its os, other ones are related with haskell tooling and most of them are a tricky combination of both.
For example the last breaks are not related directly with the infrastructure but haskell tooling problems.
Find out a specialized devop in haskell tooling would be great but i guess that will be difficult.

[ndmitchell]

Could we just put the opencollective money towards hiring someone to do devops?

With the amount of money we have, the amount of quality devops we could get is going to be nearly nothing. We also want a devops person who does the initial setup and then hangs around to fine tune it for a little bit of time every month or so. Getting a company like Tweak/FPComplete, who have expert devops on staff, to donate their expertise might be more plausible.

[wz1000]

will gitlab builds be integrated with github branch protection rules? Now we cant merge a pr if the ci is not green, not sure if gitlab ci could be integrated.

I think it would behave like any other CI provider, like Travis or Circle. This configuration is up to the repository, and I believe it is already enabled. We shouldn't have to do anything special for Gitlab CI.

[hasufell]

@ndmitchell there is talk about migrating cabal to https://gitlab.haskell.org as well. I believe HLS should migrate too. As such, a devops person wouldn't just work on HLS, but on the entire gitlab.haskell.org instance and help the GHC team. I believe there are more funds available for this sort of position.

The opencollective money could be used to set a bounty for an initial migration. It is my impression this would take an initial workload of 3 full days and then fixing bugs one after another in a course of weeks. I believe something between 1000-1500$ is a reasonable bounty for 3 days of work + occasional bugfixing.