Cirrus: Workaround F32 BFQ Kernel bug

[cevich]

Fixes #8068

[mheon]

LGTM once you're ready

…

On Thu, Oct 29, 2020 at 2:22 PM OpenShift CI Robot ***@***.***> wrote: [APPROVALNOTIFIER] This PR is *NOT APPROVED* This pull-request has been approved by: *cevich <#8188#>* To complete the pull request process <https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process>, please assign *tomsweeneyredhat* after the PR has been reviewed. You can assign the PR to them by writing /assign @TomSweeneyRedHat in a comment when ready. The full list of commands accepted by this bot can be found here <https://go.k8s.io/bot-commands?repo=containers%2Fpodman>. Needs approval from an approver in each of these files: - *OWNERS <https://github.com/containers/podman/blob/master/OWNERS>* Approvers can indicate their approval by writing /approve in a comment Approvers can cancel approval by writing /approve cancel in a comment — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#8188 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB3AOCG6QTJDBIWTAY5MA3TSNGXHHANCNFSM4TEDBK5Q> .

[cevich]

I've rebased my instrumented/testing PR on this: #8169 to help confirm the problem is only on F32 and that the previous workaround functions.

[cevich]

Update: Given the "lively" discussion in https://bugzilla.redhat.com/show_bug.cgi?id=1851783 I'm considering if we aught to simply and globally use the deadline scheduler. Also seeing (from other bugs) as this is the second time BFQ has broken for people.

[cevich]

Update: I'm investigating additional cases of agent-stopped-responding in F31. May need to include the fix there as well, until we can get up to F33 (#8074)

[rhatdan]

LGTM
@cevich ready to get this in. I think you can work on other changes in a different PR, we need to get this in now.

[cevich]

I think you can work on other changes in a different PR

No, not true. I'm still seeing remarkably similar failures on F31. I'm testing out application of the 'deadline' scheduler workaround for all VMs, and also to "even the playing field". Should be finished shortly...

[cevich]

Update: Rebased + deadline workaround for all platforms: Testing in parallel with #8169 (instrumented)

[cevich]

Important Observation: I'm noticing a significant increase in runtime for the all the "remote", especially Ubuntu. Unf. I'm also seeing (in other PRs) what appear like general google-cloud networking hiccups/slowdowns. There's no way I can separate these two (or more) affects on test runtime.

If the performance problem persists past the networking slowdown, we should consider following the google recommendations for increasing storage performance.

[cevich]

@mheon @rhatdan okay, I feel comfortable with merging this now.

[mheon]

/approve
LGTM

[openshift-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: cevich, mheon

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [mheon]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[mheon]

/lgtm