control: Enforce timeouts on response stream

[olix0r]

When connecting to a control plane API, the API server can return an
HTTP response long before it returns the first stream response. To bound
this time, we now enforce timeouts so that failures may result in attempting
to use an alternate controller instances.

All controller response streams now use a generic gRPC middlware with
initial, idle, and lifetime timeouts. When an initial timeout is
encounterd, a DeadlineExceeded grpc status is synthesized. When the
other timeouts are encountered, the stream terminates gracefully.

These timeouts are configurable by the proxy injector. Timeouts are not
enabled without configuration:

• LINKERD2_PROXY_CONTROL_STREAM_INITIAL_TIMEOUT
• LINKERD2_PROXY_CONTROL_STREAM_IDLE_TIMEOUT
• LINKERD2_PROXY_CONTROL_STREAM_LIFETIME

Each of these parameters is optional.

[codecov]

Codecov Report

Merging #2587 (bcfd140) into main (e2bb210) will decrease coverage by 0.02%.
Report is 12 commits behind head on main.
The diff coverage is 75.48%.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2587      +/-   ##
==========================================
- Coverage   68.02%   68.00%   -0.02%     
==========================================
  Files         329      330       +1     
  Lines       14904    14989      +85     
==========================================
+ Hits        10139    10194      +55     
- Misses       4765     4795      +30     

Files	Coverage Δ
linkerd/app/inbound/src/policy/config.rs	30.43% <100.00%> (ø)
linkerd/app/outbound/src/lib.rs	64.55% <100.00%> (ø)
linkerd/app/src/dst.rs	96.29% <100.00%> (+0.46%)	⬆️
linkerd/app/src/lib.rs	88.67% <100.00%> (+0.10%)	⬆️
linkerd/app/src/policy.rs	100.00% <100.00%> (ø)
linkerd/service-profiles/src/client.rs	100.00% <100.00%> (ø)
linkerd/app/inbound/src/server.rs	90.00% <66.66%> (-2.86%)	⬇️
linkerd/app/inbound/src/policy/api.rs	81.81% <66.66%> (+0.56%)	⬆️
linkerd/app/outbound/src/policy/api.rs	78.37% <66.66%> (+0.60%)	⬆️
linkerd/app/src/env.rs	59.54% <77.77%> (+1.08%)	⬆️
... and 2 more

... and 2 files with indirect coverage changes

---

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e2bb210...bcfd140. Read the comment docs.