enable golangci-lint for supported platforms

[crazy-max]

follow-up #4261 (comment)

[TBBle]

Just so I'm clear, it looks like a single lint takes 20 minutes anyway, so there's not much room for gains on GHA anyway? I note that this run took 24 minutes for lint-default, while the final run in #4261 took 20 minutes for the entire lint task.

Either that, or the GHA runner can't actually run all the lint RUN steps for different platforms in parallel, so they end up being in-effect re-serialised. If that's the case, could we distribute the platforms across a GHA matrix, rather than a single GHA task with BuildKit (fruitlessly) parallelising inside it?

Or it's something I'm not familiar-enough with to spot, like this extensive runtime being a cold cache problem that goes away in future runs, or we're just being unlucky in GHA task scheduling or something.

Wait, the jobs aren't split up properly

[jedevc]

@crazy-max I think we probably want to matrix the platforms as well? The problem with the linting is that it's cpu-intensive and uses multiple cores (I have no idea why, maybe something to dig into sometime 😢). To parallelize, we'd need to spin up a linting job for each platform.

[crazy-max]

or the GHA runner can't actually run all the lint RUN steps for different platforms in parallel

It does run the steps for different platforms in parallel but golangci-lint is quite resource intensive so contention is hardware related (linux public runners are limited to 2 vCPUs).

If that's the case, could we distribute the platforms across a GHA matrix, rather than a single GHA task with BuildKit (fruitlessly) parallelising inside it?

We could be that would increase the number of jobs in our pipeline quite heavily and therefore increase queue for other runs. From 5 lint-* jobs with this PR to 8 platforms x 5 lint-* jobs, that would create 40 jobs.

[thaJeztah]

The problem with the linting is that it's cpu-intensive and uses multiple cores

I know golangci-lint is always aggressively trying to do as many things in parallel as possible, which can be great to make use of all available resources, but I recall we had to tweak their concurrency option, as well as GOGC (and GOMAXPROCS ?) in the past for the best result (which could result in "less parallel", but faster runs).

Also don't forget build-tags and CGO_ENABLED if needed; I've had some repositories where that was overlooked, and large parts of the code were skipped. 😞

[crazy-max]

Follow-up: as discussed with @jedevc we could look at grouping default, labs and nydus and instead split by platform to reduce build time a bit more.