Enable session affinity for canaries

[wasker]

What this PR does / why we need it:

If canary ingress is configured, session affinity settings no longer apply, which results in requests within the same session being served by both canary and non-canary endpoints. This affects scenarios when backwards-incompatible changes were deployed to canary, and in inconsistent results being served in general.

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)

The change makes canaries respect session affinity settings by default, as this makes more sense than the current behavior. Nonetheless, there's a way to restore original behavior by adding a special annotation on a non-canary ingress definition, if someone prefers that behavior for some reason.

Which issue/s this PR fixes

Fixes #3717.

How Has This Been Tested?

Used repro steps from the bug to simulate the issue before and after changes.

Checklist:

• My change requires a change to the documentation.
• I have updated the documentation accordingly.
• I've read the CONTRIBUTION guide
• I have added tests to cover my changes.
• All new and existing tests passed.

[k8s-ci-robot]

Hi @wasker. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[wasker]

@rikatz This is a replacement PR for canary affinity fix.

[rikatz]

/ok-to-test
/assign

[rikatz]

Left some minor reviews from Go side (mostly about code cleanup/keeping the standard).

From lua side, have some questions that we will discuss in some zoom call :)

[rikatz]

/approve
/lgtm
Thanks

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: rikatz, wasker

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [rikatz]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment