remove resource limit on nginx ingress controller and add topology spread

[OliverMKing]

Description

Removes the resource limit on the nginx ingress controller. This allows for app routing to work on clusters with many cpus. NGINX automatically scales worker processes based on node cpus.

We also change the HPA to scale slightly more aggressively. This should lower the amount of CPU used by a single nginx ingress controller replica which means that the actual usage should be more in-line with the Resource Requests (which is good because it's guaranteed).

Pod topology spread ensures nginx ingress controller scheduled on different nodes when possible. Helps avoid single Node failure taking down the Ingress.

Type of change

Please delete options that are not relevant.

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• This change requires a documentation update

How Has This Been Tested?

Tested on e2e and locally.

Checklist:

• My code follows the style guidelines of this project
• I have performed a self-review of my code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes
• Any dependent changes have been merged and published in downstream modules

[coveralls]

Pull Request Test Coverage Report for Build 6339759755

• 52 of 52 (100.0%) changed or added relevant lines in 1 file are covered.
• No unchanged relevant lines lost coverage.
• Overall coverage increased (+0.07%) to 88.789%

---

Totals
Change from base Build 6331531917:	0.07%
Covered Lines:	1980
Relevant Lines:	2230

---

💛 - Coveralls

[OliverMKing]

/ok-to-test sha=386513c

[OliverMKing]

/ok-to-test sha=386513c

[OliverMKing]

/ok-to-test sha=386513c

[OliverMKing]

/ok-to-test sha=1da477d