Converge [New]GrpcMuxImpl

[htuch]

We have two distinct gRPC mux impls for xDS, which impacts velocity as we need to do the same thing twice when working with the xDS transport and will be a source of inconsistency long term. This is some hangover from earlier work on delta xDS.

CC @wgallagher @fredlas

[mattklein123]

What is left to do here? I thought we already merged most of the code?

[htuch]

That's not the case, there are distinct paths for delta and SotW, I'm making the same change in both places right now for #11362. @wgallagher has more complete state and a provisional plan here.

[wgallagher]

The remaining work is to merge GrpcMuxImpl and NewGrpcMuxImpl (the SoTW / Delta mux)

[fredlas]

At this point @wgallagher is probably more plugged in here than I am. However, I think there's one bit of important information to share here. When #8974, which would have resolved this, caused the weird obscure hang with that one user, the reporter and I pored over it together for like a week. There was nothing in the behavior of the new SotW code going wrong in any way. Rather, Envoy was simply not even asking for the config (IIRC; it was definitely something in the vein of "when the problem happens, it's because the unchanged bulk of Envoy never even calls the changed code"). I suspect a pre-existing bug in the init manager, especially because the user found that the hang could be broken out of by intentionally having the xDS server send a rejection.

[htuch]

Yeah, there have been quite a few changes around init manager since then as well. @fredlas @rgs1 would it be easy to validate whether or not a revived #8974 would be safe to merge again? Is there a concrete enough reproducer we can build into a test?

[htuch]

Another thing to look at while doing this work; today we have some pretty different behaviors for how discovery requests are handled during warming in SotW and delta. In SotW, we send on each new subscription, regardless of existing state. In delta, we take into account whether an existing request is pending and don't send when a new cluster is added (for example). I think this has (to some extent) to do with the differences in GrpcMuxImpl and NewGrpcMuxImpl.

[dmitri-d]

I'm picking this up as @wgallagher is unable to continue working on this. @fredlas & @rgs1 is there a way to reproduce the issue you were seeing in #8974?

[rgs1]

@dmitri-d it's been too long and even back then we couldn't get a repro. I am however happy to give a rebased branch a spin and see how it goes. The breakage was deterministic and quick.