[L] Investigation: trouble with updates with large number of builds and slow to add packages

[juhp]

Current Fedora bodhi seems having serious problems accepting large updates with many packages.

• The web UI stays on the Create or Edit page after clicking Update or Save.
• The cli times out with a large package set (several hundred packages for example)
• Adding bunches of packages (my workaround since I can't create a mega-update in one go) seems to happen "async" - takes some minutes after pressing Save on an update edit.
• Seems like I have to Save twice when adding sets of packages...

This seems problematic and makes Bodhi look unreliable unfortunately.

[juhp]

Maybe the worst thing is that often web ui error messages are not shown: for example mistakenly I was trying to adding some builds again to an update and there was no feedback/error/warning shown - maybe it timeout because it takes too long??

[bowlofeggs]

There are a few different issues here:

• The web UI does not show unexpected errors from the backend - see Bodhi's update create form does not gracefully handle errors from the backend #3035
• The title field prevents updates from having very many builds - see Bodhi cannot create updates with ~300 builds in them #1542
• The server performs koji API calls for each build in an update, which takes a long time when there are hundreds of them.

Let's use this ticket to track that last problem. We have plans to add a task working system to bodhi in #2851. If we had that, we could split a lot of this work out into a background task to avoid server timeouts. This would make the create form much more responsive, and we could do the rest of the work on the backend.

[bowlofeggs]

#3061 may help with this as well.

[bowlofeggs]

One more that might help: #3062

[juhp]

I don't think #3061 is soo bad since creating an update with like 100 builds doesn't take that long, bicbw...

[AdamWill]

I've just run into this while doing the rebuilds for Meson 0.50.0. Constructing the list of builds to put in the update is quite painstaking so I've been editing in groups of five or six at a time so I don't lose a ton of work if something goes wrong with any one batch, but as the update gets bigger, each edit operation seems to get harder and harder to get to work. I'm just hitting the button over and over and refreshing the update page in another tab to see what's happening at this point. It's very awkward.

[cverna]

Scope

We need to get a better understanding of why updates with a large number are slow. Let's use this ticket to investigate and find possible solution.

[bowlofeggs]

This may also help with this: #3204

[bowlofeggs]

#3201 is planned for inclusion in the upcoming 4.0.0 release: #3221. That should help a little bit with this issue, but I do not yet consider the issue solved so we'll keep it open.

[bowlofeggs]

Bodhi 4.0.0 beta is built and deployed to staging:

https://copr.fedorainfracloud.org/coprs/bowlofeggs/bodhi-pre-release
https://bodhi.stg.fedoraproject.org/

[cverna]

#3448 should improve the situation. Doing a local test with creating an update of 103 builds it took 26 seconds before the change and 5seconds with the refactoring.
I ll keep looking for possible performance improvement.

[AdamWill]

Nice! I'll keep an eye on it next time I need to interact with a gigantic update.