If a user has multiple builds running at the same time, one or both will fail ("Unable to build the image using the OpenShift build service") #2729
Comments
|
ldimaggi
added
type/bug
area/pipelines
issue/intermittent
SEV2-high
and removed
issue/intermittent
labels
ldimaggi
changed the title
ldimaggi
changed the title
ldimaggi
changed the title
|
@chmouel Does currently OSIO support parallel builds for a tenant ? |
|
@krishnapaparaju AFAIK no, @kbsingh sometime ago pointed it to me, |
|
What makes this problem difficult for users is that the competing builds can be in different spaces, so that it is easy to overlook one or more of them. |
|
I don't think that this issue is occurring only when multiple builds are running at the same time. I reset my account, created a new project and build failed. |
|
I have seen that too - are you sure that all the build configs were
"cleaned out" of OSO by the reset? That's just a guess - but I think I saw
a build config not be cleaned by the reset.
…On Mon, Mar 26, 2018 at 9:45 AM, Pavol Pitonak ***@***.***> wrote:
I don't think that this issue is occurring only when multiple builds are
running at the same time. I reset my account, created a new project and
build failed.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
<#2729 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AAnOPY1DnR4KI1NhGTid2zVN1FOUxTPOks5tiPD3gaJpZM4S3VPG>
.
--
Len DiMaggio ([email protected])
JBoss by Red Hat
314 Littleton Road
Westford, MA 01886 USA
tel: 978.392.3179
cell: 781.472.9912
http://www.redhat.com
http://community.jboss.org/people/ldimaggio
|
Even though user run the two builds in the same space it will fall into the same issue. IssueThe primary issue is with Eventually, build throws following status event on OSO After some time build status become Edit: |
|
lets make sure we are fixing this at the right level. Parallel builds should be supported on OSIO; no question about that - but users might not have enough resources to do so before upgrading. In build.next we can hopefully manage this for the user; but until then would this be a matter of enabling something like https://wiki.jenkins.io/display/JENKINS/Throttle+Concurrent+Builds+Plugin or similar to by default have limit of concurrent builds to be 1 and everything else gets queued ? |
Queuing and enforcing build execution policies could be useful for build.next as well. Thanks for this plugin, will evaluate it for our use case. |
UpdateWe've evaluated the Jenkins throttling plugin. Its possible to throttle the parallel builds. Need to consider following facts.
// Throttle of a single operation
throttle(['category']) {
node {
....
}
}
At the end if two concurrent builds starts runing this is how pending job looks like. |
|
unfortunate if the throttling has to be defined by the user in their jobs - then that kinda defeats the purpose. |
Update:We have evaluated multiple approaches to limit concurrent build for the time being
Approach 2 is more feasible option where in we are changing |
Update:Setting Will submit the PR to fix this issue. |
|
FYI @sthaha @jaseemabid @krishnapaparaju entry point for the build locker plugin to dispatch or block the job for execution. @lordofthejars could you please help here to understand the plugin and Jenkins behaviour? |
|
Try this Regular expression , this works ' .*.* ' |
|
@hrishin I have no idea what else to add. What I see is that if a build is occurring and you need a new agent then this agent is started, but it takes some time to get started so if another build enters to the system since the agent is not already started then it starts another one. As you said it seems a race condition. So I was never involved in any master/agent communication task, so the only thing that comes to my mind would modify the plugin (probably k8s plugin if we have the knowledge) to do the next task: A request to create an agent comes, lock and create a file, and unlock. Then any other request just check this file, if it is present then it needs to wait until it disappears and after that create a new agent. Problem with that is that if you miss for any reason to delete this file after the first build finishes (maybe because a failure) then you are in a deadlock. So I don't like so much this solution because there is a risk of dead lock, but nothing else comes to my mind right now. Other option might be lock the process until the agent is up and running so any other request is locked until the agent is up and running, it is a different way of doing but again we really need to take care about dead locks. |
|
@krishnapaparaju Could you explain why |
Update:Using build blocker plugin Jenkins is able to throttle builds.
Final PR's for this issue:
|
|
accidentally closed by PR merge |
|
openshiftio/saas-openshiftio#904, raised PR to get this in Production |
|
Auto closed by PR merge |
|
Closing this is should be in prod #3752. |
|
Since this fix is in the production, only one pipeline can run at a time. It would make sure not throwing random |
|
great. thanks @hrishin |
Steps to recreate:
One or both of the pipelines will fail with this error:
The text was updated successfully, but these errors were encountered: