tang-operator [scale-out]: timeout is expiring for POD to go to Running state

[sarroutbi]

tang-operator: timeout is expiring for PODs to go to Running state:

::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
::   Scale-out scalability test
::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::

:: [ 16:13:46 ] :: [   PASS   ] :: Creating scale out test [0] (Expected 0, got 0)
:: [ 16:13:47 ] :: [   PASS   ] :: Checking 1 POD is started [Timeout=120 secs.] (Expected 0, got 0)
:: [ 16:13:47 ] :: [   PASS   ] :: Checking 1 Service is started [Timeout=120 secs.] (Expected 0, got 0)
:: [ 16:13:48 ] :: [   PASS   ] :: Checking pod name not empty (Assert: "tangdeployment-tangserver-scaleout-6b6455fb5b-wn9c4" should not equal "")
:: [ 16:13:58 ] :: [   PASS   ] :: Checking POD in Running state [Timeout=120 secs.] (Expected 0, got 0)
:: [ 16:13:59 ] :: [   PASS   ] :: Creating scale out test [1] (Expected 0, got 0)
:: [ 16:14:00 ] :: [   PASS   ] :: Checking 1+1 PODs are started [Timeout=120 secs.] (Expected 0, got 0)
:: [ 16:14:00 ] :: [   PASS   ] :: Checking pod name not empty (Assert: "tangdeployment-tangserver-scaleout-6b6455fb5b-wzprb" should not equal "")
:: [ 16:16:56 ] :: [   FAIL   ] :: Checking added POD in Running state [Timeout=120 secs.] (Expected 0, got 1)
:: [ 16:18:07 ] :: [   PASS   ] :: Deleting scale out test (Expected 0, got 0)
:: [ 16:18:08 ] :: [   PASS   ] :: Checking no PODs continue running [Timeout=5 secs.] (Expected 0, got 0)
:: [ 16:18:08 ] :: [   PASS   ] :: Checking no Services continue running [Timeout=120 secs.] (Expected 0, got 0)
::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::

This needs to be investigated, to check if it is a ROSA specific issue, as this timeout has been enough on the rest of the platforms

[sarroutbi]

This issue occurs on scale out test, and is related to the fact that attaching more than one POD to a PVC is not possible in ROSA deployments. Fix is not related to timers, but rather to detect if PVC multi attach is supported

[sarroutbi]

This is not always reproducible. It seems related to issue:
kubernetes-csi/external-attacher#215

Not a ROSA specific issue.

It should be fixed when tang-operator is migrated to use Secrets instead of PVCs

[sarroutbi]

This might be of help also:
https://www.cyberithub.com/solved-multi-attach-error-for-volume-pvc-xx-volume-is-already/

[sarroutbi]

Closing. This is an implementation internal issue, not related to tests