If the database is unavailable and caching is enabled, allow operator to proceed

[johscheuer]

Description

Fixes: #1931

Type of change

Please select one of the options below.

• Bug fix (non-breaking change which fixes an issue)

Discussion

See the issue. The idea is to allow the operator to proceed and try to fetch the machine-readable status in the later steps again. This could resolve cases where multiple Pods are down and they must be recreated.

Testing

Have to add an e2e test for this.

Documentation

Follow-up

[foundationdb-ci]

Result of fdb-kubernetes-operator-pr on Linux CentOS 7

• Commit ID: fb70303
• Duration 1:44:24
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of fdb-kubernetes-operator-pr on Linux CentOS 7

• Commit ID: 818053e
• Duration 2:53:34
• Result: ❌ FAILED
• Error: Error while executing command: if $fail_test; then exit 1; fi. Reason: exit status 1
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of fdb-kubernetes-operator-pr on Linux CentOS 7

• Commit ID: ed5882e
• Duration 3:25:47
• Result: ❌ FAILED
• Error: Error while executing command: if $fail_test; then exit 1; fi. Reason: exit status 1
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of fdb-kubernetes-operator-pr on Linux CentOS 7

• Commit ID: ed5882e
• Duration 3:27:33
• Result: ❌ FAILED
• Error: Error while executing command: if $fail_test; then exit 1; fi. Reason: exit status 1
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[johscheuer]

The failure that I'm seeing is quite interesting and I have to spend some more time on this to fix it.

[foundationdb-ci]

Result of fdb-kubernetes-operator-pr on Linux CentOS 7

• Commit ID: e812e19
• Duration 3:30:50
• Result: ❌ FAILED
• Error: Error while executing command: if $fail_test; then exit 1; fi. Reason: exit status 1
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of fdb-kubernetes-operator-pr on Linux CentOS 7

• Commit ID: e812e19
• Duration 2:07:13
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[nicmorales9]

LGTM aside from the one comment!

[foundationdb-ci]

Result of fdb-kubernetes-operator-pr on Linux CentOS 7

• Commit ID: f885be1
• Duration 3:25:10
• Result: ❌ FAILED
• Error: Error while executing command: if $fail_test; then exit 1; fi. Reason: exit status 1
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of fdb-kubernetes-operator-pr on Linux CentOS 7

• Commit ID: f885be1
• Duration 1:58:00
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[johscheuer]

Result of fdb-kubernetes-operator-pr on Linux CentOS 7

• Commit ID: f885be1
• Duration 3:25:10
• Result: ❌ FAILED
• Error: Error while executing command: if $fail_test; then exit 1; fi. Reason: exit status 1
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

Seems like we are hitting this issue: apple/foundationdb#11104

I try to back port that change to 6.2 (which we use right now), to see it that solve the issue. In that case I will back port that change in FDB.

[foundationdb-ci]

Result of fdb-kubernetes-operator-pr on Linux CentOS 7

• Commit ID: 5880d59
• Duration 0:03:56
• Result: ❌ FAILED
• Error: Error while executing command: IMG=${REGISTRY}/${OPERATOR_IMAGE} make container-build container-push. Reason: exit status 2
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of fdb-kubernetes-operator-pr on Linux CentOS 7

• Commit ID: db383e2
• Duration 1:58:41
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of fdb-kubernetes-operator-pr on Linux CentOS 7

• Commit ID: 25ed035
• Duration 3:28:40
• Result: ❌ FAILED
• Error: Error while executing command: if $fail_test; then exit 1; fi. Reason: exit status 1
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of fdb-kubernetes-operator-pr on Linux CentOS 7

• Commit ID: 91157f5
• Duration 2:08:48
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)