Tests: VCH multi NIC configurations

[hickeng]

Story
As a VIC developer I want to know that the various multi-NIC deployments are functional
As a viadmin I expect clear documentation on what NIC configurations are permitted or not permitted

Detail
The 3 NIC limitation described in #2802 has been addressed so we need to expand the testing configurations that use multiple port groups. This should encompass subnet segregation between vCenter and ESX hosts to reflect best practice deployment guidelines.

@pdaigle @emeirell can you comment on use cases for more complex scenarios that we don't currently support and raise issues for support if necessary:

• link aggregation style (multiple NICs on same network, different roles)
• asymmetric setup (e.g. forwarded ports should be directly served to client network as well as public, but not visa versa)

Acceptance

• confirm that test env has subnet segregation between vCenter and ESX
• test each of the follow deployments and assert connectivity/lack of from the networks:
  • public/client/management shared
  • public/client shared (2)
  • no sharing (3)
• documentation updated as needed, with use cases for configurations

I'd suggest a mix of (2) and (3) as the normal test deployment basis as those are the common customer scenarios.

Assigned to @gigawhitlocks for decomposition into actual work required against current state of VC test env.

[mhagen-vmware]

This is best handled in nimbus, nightly tests. If I am understanding the scenarios correctly:

1. public/client/management all using the same DPG
2. public/client using same DPG and management using different DPG
3. all three interfaces all using different DPGs.

This should be easily accomplished via our existing patterns and govc commands to setup the networking as desired.

[mhagen-vmware]

Removing from the Epic, because if agreed on my previous comment, then this is unrelated to our VC CI efforts.

[hickeng]

@mhagen-vmware If CI is deployed following VMware best practice we will have most of this in place already (i.e. separate management network and VC/ESX subnet segregation).
I see no reason not to be able to exercise our recommended deployment config for VIC in CI as it doesn't seem to require significant additional configuration complexity (2 additional DPGs with VLAN assignment).

Using static IPs is something we should likely confine to nimbus for now as we'd otherwise need an IPAM mechanism for reserving/releasing addresses indirectly (e.g. out-of-band DHCP?).