Build requests piling up on workers #250
Comments
|
I believe previously, maybe a few month ago, it was never more than 1-2 requests. |
|
Another example
It just started to build 3 days old patch. |
|
This has been discussed in discourse starting from this message - https://discourse.llvm.org/t/staging-buildmaster-no-longer-collapses-requests/80500/3. In short - a few days ago we had unusually large number of commits with changes to cmake files. For such commits we request clean builds to make sure if there is an issue it will not be hidden by incremental builders. Build requests with different sets of properties do not collapse. If this would become a regular problem, I'll rethink of the combining logic. I quickly checked and this is the case for the builder mentioned here as well. |
|
Is there an option to disable this for a particular builder? https://lab.llvm.org/buildbot/#/waterfall?tags=sanitizer Also they don't run incremental builds, but instead, as suggested, use ccache. So scheduling clean build unnecessary, |
|
None of them is marked as doing clean builds. Could you prepare a patch which either explicitly sets And I'll think of changing the collapse logic to use that information. |
This is step 1 to addressing the rapidly expanding build queues we're seeing (#250). The bot is currently on silent, and the plan is: * Mark all builds as clean (we use ccache so it shouldn't be a big impact). * Clear the existing requests. * See what happens to the queue. It's possible we will need a follow up to make collapsing work, but this is the first step either way.
We (Linaro) are working on changes to do this. |
|
We've been able to implement this new rule and it appears to be working. We're going to test it more over the weekend and have changes for review next week. |
This is to address llvm#250. Where build requests were piling up on slower builders because they were clean/don't clean/clean/ don't clean etc. and we could not merge those requests because of that difference. Galina suggested that if we knew that the builder was going to clean the build first anyway, we could merge them. This commit implements that. We do this by first setting a "clean" attribute on the builder's factory object, that we can find later via in the collapse callback. So far only ClangBuilder passes this on but any builder can opt in by copying what it does. In the collapse function when comparing properties we delete the "clean_obj" key from both sets of properties if we know that the builder's factory is always clean. With some logging (not included in this commit) we can see it now collapses a clean/non-clean pair of requests: ``` collapseRequests selfBuildSetProperties: {'scheduler': ('main:clang,clang-tools-extra,compiler-rt,flang,lld,llvm,mlir', 'Scheduler')} otherBuildSetProperties: {'scheduler': ('main:clang,clang-tools-extra,compiler-rt,flang,lld,llvm,mlir', 'Scheduler'), 'clean_obj': (True, 'Change')} Removing clean_obj from build set properties. Build set properties were the same. ``` This has beeen tested by my colleague Omair Javaid on a test build master. Of course we only tested it with one of Linaro's builders, so it's possible we see other side effects in production.
…R_RT_BUILD_SANITIZERS=OFF clean=True due to <#250> Not updating other riscv builders as they are about to be redeployed with a different config (PR forthcoming).
For #250. This bots can't catch-up with `clean_obj` requests, but they do clean build anyway. Maybe we should do the same for all sanitizer builders, but seems they are OK as is.
|
It's still happening. https://lab.llvm.org/buildbot/#/builders/94
|
|
I expect that the buildmaster(s) have not received the change yet. This timing of that is up to @gkistanova . |
Buildbot is not ready yet #250 (comment) This reverts commit 26a33cd.
Buildbot is not ready yet #250 (comment) This reverts commit 26a33cd.
|
Linaro has had https://lab.llvm.org/buildbot/#/builders/121 running on normal for a couple of days now and requests are not piling up. @vitalybuka can you try your bot again? If it still doesn't work, I will double check the implementation in case I managed to make it only work for Clang builders specifically. |
https://lab.llvm.org/staging/#/builders/sanitizer-x86_64-linux-bootstrap-msan
The text was updated successfully, but these errors were encountered: