annotation support for http2

[shraykay]

What this PR does / why we need it:
http2 support at the ingress level via an annotation

Which issue this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close that issue when PR gets merged): fixes #

Special notes for your reviewer:
please lmk if I've missed any areas on the nginx.conf go template or if I should place the attachment from annotation to server in another function or section.

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: shraykay
To fully approve this pull request, please assign additional approvers.
We suggest the following additional approver: nicksardo

Assign the PR to them by writing /assign @nicksardo in a comment when ready.

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[aledbf]

@shraykay I am not sure if we should add this annotation. The reason is enabling http2 on port 80 breaks HTTP traffic and is not possible to do it only for some paths.

[aledbf]

I already tried this. Please check #2313

[mhamrah]

Can you elaborate on how it 'breaks http traffic'? I don't think it should be up to the nginx ingress to enforce which application protocol should be allowed on which port; let the upstream handler determine how to process the request. An upstream might only want to handle http2, gRPC, or do some stream-level introspection to determine whether it's an http/http2 request.

Yes, there might be some circumstances where an http2 rule interferes with an http rule, but that's a configuration problem of the user.

[aledbf]

Can you elaborate on how it 'breaks http traffic'?

If you enable http2 on port 80 and try to make an http request you receive a binary response (http2). There is no fallback to http so only http2 traffic works (his is fine if you have only one Ingress per hostname) This leads to not being able to distinguish if the error is the ingress controller and nginx.

An upstream might only want to handle http2, gRPC, or do some stream-level introspection to determine whether it's an http/http2 request.

Sure but nginx can't and that's the problem. Please check https://forum.nginx.org/read.php?29,278877,278929#msg-278929

[aledbf]

Closing. We can add this feature but not as an annotation but in the configuration configmap.
Please check #2189 (comment)