RHOAIENG-15772: tests(odh-nbc) significantly reduce the amount of sleeping in tests

[jiridanek]

https://issues.redhat.com/browse/RHOAIENG-15772

Relates to

• Upgrade ginkgo test with use of Eventually instead of time funcs  #380

Description

• remove time.sleep()s and rely on Eventually
• shorten the interval constant so that we check more frequently
• leaving one sleep that seems necessary for the tests (!)
• and two more sleeps that are a workaround for flakyness

This PR depends on

• RHOAIENG-15772: tests(nbcs): use the direct (uncached) client to drive tests #461

because with the caching client, a check for resource passes (resource is in cache) but server-side patching fails (resource is not yet on server). For example this one

kubeflow/components/odh-notebook-controller/controllers/notebook_controller_test.go

Lines 840 to 841 in 2555813

	patch := client.RawPatch(types.MergePatchType, []byte(`{"spec":{"to":{"name":"foo"}}}`))
	Expect(cli.Patch(ctx, route, patch)).Should(Succeed())

and also the create-get does not work well

kubeflow/components/odh-notebook-controller/controllers/notebook_controller_test.go

Lines 331 to 336 in 2555813

	By("By creating a new Notebook")
	Expect(cli.Create(ctx, notebook)).Should(Succeed())
	By("By updating the Notebook's image")
	key := types.NamespacedName{Name: Name, Namespace: Namespace}
	Expect(cli.Get(ctx, key, notebook)).Should(Succeed())

How Has This Been Tested?

make test

Merge criteria:

• The commits are squashed in a cohesive manner and have meaningful messages.
• Testing instructions have been added in the PR body (for PRs involving changes that are not immediately obvious).
• The developer has manually tested the changes and verified that the changes work

[openshift-ci]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please ask for approval from jiridanek. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[jstourac]

/lgtm

[jiridanek]

This makes the tests more flaky than they are now, so I plan to sit on it a while, until I get other (more important) things merged in opendatahub-io/notebooks.

[jiridanek]

I'm waiting with the for when some of the test fixing Jiras appear on our Sprint, so that I'm not the first to touch this long-time-untouched code.