fix: paginate DescribeNetworkInterfaces with deep filters

[sushrk]

Issue #, if available:
Fixes #188
Description of changes:
Paginate DescribeNetworkInterfaces API call by specifying a deep filter like vpc-id and subnet-id as unpaginated API call causes the operation to fail when there is a large number of network interfaces.
https://docs.aws.amazon.com/cli/latest/reference/ec2/describe-network-interfaces.html

Unpaginated calls were used in the periodic leaked ENI cleanup routine and during trunk initialization to fetch all branch ENIs on the node. These are now paginated to use vpc-id and subnet-id respectively, and also added jitter to avoid EC2 API throttling and increased call volume to address earlier issue with pagination.

Testing:
Security group for pods test succeeded:

Ran 19 of 23 Specs in 975.331 seconds
SUCCESS! -- 19 Passed | 0 Failed | 0 Pending | 4 Skipped
PASS

Ginkgo ran 1 suite in 16m18.325964387s
Test Suite Passed

Manual tests:

• Node scale up/down
• Pods using security groups scale up/down
• Created 500+ ENIs in same VPC to confirm call volume has not increased drastically

ec2_describe_network_interfaces_pages_api_call_count 4
ec2_describe_network_interfaces_pages_api_err_count 0

• Soak test with 100 nodes and scaling up pods using to security group from 0 to 1000 every 15mins. Did not observe a drastic increase in the API call volume.

Release notes:

Paginate DescribeNetworkInterfaces API call

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

[orsenthil]

Some overall comments

    Created 500+ ENIs in same VPC to confirm call volume has not increased drastically

ec2_describe_network_interfaces_pages_api_call_count 4
ec2_describe_network_interfaces_pages_api_err_count 0

1. Do we have before and after call count.

2. We could measure is the pod create latency of the SGPP operation before and after the change in cluster with 1000 ENI and ensure that they there is no change.

3. Does the paginated version return any empty pages at all, even with a filter? Can we assert against it?

4. What are the drawbacks on including a required vpc-id as a parameter in the new version of VPC RC?

5. So the trunk Init routine, WaitForNetworkInterfaceStatusChange will continue to use DescribeNetworkInterfaces (unpaginated), but only the ENI cleanup will use the paginated. Is that correct? Do we see any problems with trunk init using the unpaginated, with large number of ENI cases?

[sushrk]

Some overall comments

    Created 500+ ENIs in same VPC to confirm call volume has not increased drastically

ec2_describe_network_interfaces_pages_api_call_count 4
ec2_describe_network_interfaces_pages_api_err_count 0

1. Do we have _before_ and _after_ call count.

2. We could measure is the pod create latency of the SGPP operation _before_ and _after_ the change in cluster with 1000 ENI and ensure that they there is no change.

3. Does the paginated version return any empty pages at all, even with a filter? Can we assert against it?

4. What are the drawbacks on including a required vpc-id as a parameter in the new version of VPC RC?

5. So the trunk Init routine, `WaitForNetworkInterfaceStatusChange` will continue to use DescribeNetworkInterfaces (unpaginated), but only the ENI cleanup will use the paginated. Is that correct? Do we see any problems with trunk init using the unpaginated, with large number of ENI cases?

1. Yes, we do a benchmark test for every release, the first graph is from v1.4.3 vs second graph from the test. This is running the integration test cases in a loop for 5 times. Did not observe any drastic increase in call volumes with pagination.
   

2. SGPP pod creation latency is comparable to previous releases when scaled up to 1000 pods, approx 4 mins.

3. The paginated version could return empty pages, this is not deterministic. But using deep filter seems to have alleviated the problem of increased call volume. Graph for soak test with 100 nodes and scaling pods using security pods from 0 to 1000 every 15 mins, ran overnight
   

4. There should be no drawbacks, we will update the controller manifest accordingly with the release to populate the VPC-ID. This is required for the cleanup routine to pass the vpc-id in DescribeNetworkInterfaces

5. All other calls to DescribeNetworkInterfaces pass in the network interface ID directly, like in WaitForNetworkInterfaceStatusChange, and is not an issue with respect to throttling/being blocked by EC2 due to large number of ENIs in the account.

[haouc]

lgtm

[orsenthil]

All other calls to DescribeNetworkInterfaces pass in the network interface ID, like in WaitForNetworkInterfaceStatusChange, and is not an issue with respect to throttling.

Got it. Thank you!.

[orsenthil]

LGTM