pubsub: many tests failed

[flaky-bot]

Many tests failed at the same time in this package.

• I will close this issue when there are no more failures in this package and
  there is at least one pass.
• No new issues will be filed for this package until this issue is closed.
• If there are already issues for individual test cases, I will close them when
  the corresponding test passes. You can close them earlier, if you prefer, and
  I won't reopen them while this issue is still open.

Here are the tests that failed:

• TestFlowControllerCancel (pubsub: TestFlowControllerCancel failed #7072)
• TestFlowControllerLargeRequest (pubsub: TestFlowControllerLargeRequest failed #7073)
• TestFlowControllerNoStarve (pubsub: TestFlowControllerNoStarve failed #7074)
• TestFlowControllerSaturation (pubsub: TestFlowControllerSaturation failed #7075)
• TestFlowControllerUnboundedCount (pubsub: TestFlowControllerUnboundedCount failed #7076)
• TestFlowControllerUnboundedCount2 (pubsub: TestFlowControllerUnboundedCount2 failed #7077)
• TestFlowControllerUnboundedBytes (pubsub: TestFlowControllerUnboundedBytes failed #7078)
• TestIntegration_All (pubsub: TestIntegration_All failed #7079)
• TestIntegration_CancelReceive (pubsub: TestIntegration_CancelReceive failed #8195)
• TestIntegration_CreateSubscription_NeverExpire
• TestIntegration_UpdateSubscription
• TestIntegration_UpdateSubscription_ExpirationPolicy
• TestIntegration_UpdateTopicLabels
• TestIntegration_PublicTopic
• TestIntegration_Errors (pubsub: TestIntegration_Errors failed #8431)
• TestIntegration_MessageStoragePolicy_TopicLevel
• TestIntegration_MessageStoragePolicy_ProjectLevel
• TestIntegration_CreateTopic_KMS
• TestIntegration_CreateTopic_MessageStoragePolicy
• TestIntegration_OrderedKeys_JSON (pubsub: TestIntegration_OrderedKeys_JSON failed #6001)

---

commit: a05b6ed
buildURL: Build Status, Sponge
status: failed

[flaky-bot]

Looks like this issue is flaky. 😟

I'm going to leave this open and stop commenting.

A human should fix and close this.

---

When run at the same commit (a05b6ed), this test passed in one build (Build Status, Sponge) and failed in another build (Build Status, Sponge).

[hongalex]

This seems to be failing on one test panic that causes the remaining tests to fail:

=== RUN   TestIntegration_OrderedKeys_JSON
    integration_test.go:1311: CreateTopic error: key18: got -, want +
        	  []string{
          	"message6019",
        + 	"message6019",
          }
panic: sync: negative WaitGroup counter

[hongalex]

In the above, the error string is incorrectly named, it should actually be related to VerifyKeyOrdering.

In addition, the panic is caused by the negative WaitGroup, which itself was caused by a message redelivery. This makes sense given that Pub/Sub is at-least once delivery by default. However, it's strange that this is happening, given that messages should be deduped by this CL.

[hongalex]

Some additional findings

1. This has happened before in pubsub: many tests failed #8333, but that occurred in July which means it happens fairly infrequently
2. Running this continually locally does not reproduce the issue, so there might be additional concerns with tests running in noisier machines that doesn't happen locally
3. As mentioned above, message redelivery is possible for Pub/Sub, but we are deduping via message ID (which is unique per message and doesn't change during redeliveries). The fact that we are receiving multiple messages with the same data but different ID is potentially because Publish was called twice with the same data, perhaps because of retries.

For now, I will resolve this issue by cleaning up this test and deduping via message data rather than ID (which is guaranteed to be unique per the test data). We should also log any Publish fails inside a goroutine (can't use test failure inside goroutine, but logging should be sufficient).