Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make a deep copy for introspection #1179

Merged
merged 1 commit into from
Aug 31, 2020

Conversation

mogren
Copy link
Contributor

@mogren mogren commented Aug 30, 2020

Issue #, if available: #1101

Description of changes:
Follow up to #1178. The logs make it clear that map[string]*AddressInfo inside the ENI struct is causing the issue:

{"level":"info","ts":"2020-08-30T16:48:05.305Z","caller":"http/server.go:2416","msg":"Handling http request: GET, from: 127.0.0.1:56308, URI: /v1/enis"}
{"level":"debug","ts":"2020-08-30T16:48:05.305Z","caller":"http/server.go:2041","msg":"eniInfos: &{TotalIPs:42 AssignedIPs:3 ENIs:map[eni-04e635dd5bb78338c:{ID:eni-04e635dd5bb78338c createTime:{wall:13820161790756677995 ext:1420777325 loc:0x559806b2f3c0} IsPrimary:true IsTrunk:false DeviceNumber:0 IPv4Addresses:map[192.168.67.189:0xc000489620 192.168.69.119:0xc0004897a0 192.168.73.173:0xc000489320 192.168.74.37:0xc0004893e0 192.168.76.135:0xc000489440 192.168.79.14:0xc000489380 192.168.84.205:0xc0004892c0 192.168.86.63:0xc0004896e0 192.168.87.28:0xc000489560 192.168.88.155:0xc0004894a0 192.168.88.244:0xc000489740 192.168.90.222:0xc000489680 192.168.92.61:0xc0004895c0 192.168.94.188:0xc000489500]} eni-087c7067a58f9d3b0:{ID:eni-087c7067a58f9d3b0 createTime:{wall:13820161790756768172 ext:1420867502 loc:0x559806b2f3c0} IsPrimary:false IsTrunk:false DeviceNumber:2 IPv4Addresses:map[192.168.65.23:0xc0005de060 192.168.66.255:0xc000489f20 192.168.71.122:0xc000489da0 192.168.74.63:0xc000489ec0 192.168.75.147:0xc0005de000 192.168.77.190:0xc000489e60 192.168.80.132:0xc000489bc0 192.168.81.170:0xc000489b60 192.168.82.81:0xc000489f80 192.168.83.248:0xc000489d40 192.168.83.254:0xc000489e00 192.168.83.7:0xc000489c20 192.168.87.152:0xc000489c80 192.168.87.24:0xc000489ce0]} eni-099ea6395cdb509e6:{ID:eni-099ea6395cdb509e6 createTime:{wall:13820161790758681173 ext:1422780502 loc:0x559806b2f3c0} IsPrimary:false IsTrunk:true DeviceNumber:4 IPv4Addresses:map[192.168.69.116:0xc000489080 192.168.70.178:0xc000488fc0 192.168.70.45:0xc000488cc0 192.168.72.114:0xc000488f60 192.168.77.192:0xc000488d80 192.168.79.195:0xc000488e40 192.168.80.157:0xc000488f00 192.168.84.107:0xc000488c00 192.168.87.154:0xc000488ea0 192.168.87.206:0xc000488d20 192.168.87.234:0xc000488ba0 192.168.88.225:0xc000488de0 192.168.91.242:0xc000489020 192.168.92.75:0xc000488c60]}]}"}
  • Making a deep copy of the IPv4Addresses map.
  • Remove verbose logging
  • Rename currentENIIPPools to currentENIs since the struct was renamed in #972

e2e tests passing https://app.circleci.com/pipelines/github/mogren/amazon-vpc-cni-k8s/784/workflows/d2d2bd30-118e-408b-894d-490879722a7e/jobs/1587

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

@mogren mogren requested a review from SaranBalaji90 August 30, 2020 17:26
@@ -130,9 +130,7 @@ func (c *IPAMContext) setupIntrospectionServer() *http.Server {

func eniV1RequestHandler(ipam *IPAMContext) func(http.ResponseWriter, *http.Request) {
return func(w http.ResponseWriter, r *http.Request) {
eniInfos := ipam.dataStore.GetENIInfos()
log.Debugf("eniInfos: %+v", eniInfos)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess we could have caught this yesterday itself.. can you add the logger back and verify the change by looking at the output.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well, I did have the logger there when testing the change, but it's hard to say from those logs what the pointer is pointing to:

{"level":"info","ts":"2020-08-30T17:17:54.515Z","caller":"http/server.go:2416","msg":"Handling http request: GET, from: 127.0.0.1:41846, URI: /v1/enis"}
{"level":"debug","ts":"2020-08-30T17:17:54.515Z","caller":"http/server.go:2041","msg":"eniInfos: &{TotalIPs:28 AssignedIPs:3 ENIs:map[eni-04e635dd5bb78338c:{ID:eni-04e635dd5bb78338c createTime:{wall:13820163718338356131 ext:1018422474 loc:0x558e92c0b3c0} IsPrimary:true IsTrunk:false DeviceNumber:0 IPv4Addresses:map[192.168.67.189:0xc000373020 192.168.69.119:0xc0003734a0 192.168.73.173:0xc0003733e0 192.168.74.37:0xc0003731a0 192.168.76.135:0xc000373200 192.168.79.14:0xc000373260 192.168.84.205:0xc000373140 192.168.86.63:0xc000373380 192.168.87.28:0xc000373440 192.168.88.155:0xc000372fc0 192.168.88.244:0xc0003730e0 192.168.90.222:0xc000373080 192.168.92.61:0xc000373320 192.168.94.188:0xc0003732c0]} eni-099ea6395cdb509e6:{ID:eni-099ea6395cdb509e6 createTime:{wall:13820163718338447800 ext:1018514140 loc:0x558e92c0b3c0} IsPrimary:false IsTrunk:true DeviceNumber:4 IPv4Addresses:map[192.168.69.116:0xc000373620 192.168.70.178:0xc000373800 192.168.70.45:0xc0003736e0 192.168.72.114:0xc000373980 192.168.77.192:0xc000373920 192.168.79.195:0xc000373560 192.168.80.157:0xc0003737a0 192.168.84.107:0xc000373500 192.168.87.154:0xc0003735c0 192.168.87.206:0xc0003738c0 192.168.87.234:0xc000373680 192.168.88.225:0xc000373740 192.168.91.242:0xc0003739e0 192.168.92.75:0xc000373860]}]}"}
{"level":"info","ts":"2020-08-30T17:17:57.426Z","caller":"http/server.go:2416","msg":"Handling http request: GET, from: 127.0.0.1:41866, URI: /v1/enis"}

These refs now point to tmpENIInfo.IPv4Addresses[eni] = &ipAddrInfo instead of the values in the map owned by *eniInfo. Do we still want this debug log added?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we are good now :) Thanks.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

curl output:

$ curl localhost:61679/v1/enis
{"TotalIPs":42,"AssignedIPs":3,"ENIs":{"eni-04e635dd5bb78338c":{"ID":"eni-04e635dd5bb78338c","IsPrimary":true,"IsTrunk":false,"DeviceNumber":0,"IPv4Addresses":{"192.168.67.189":{"IPAMKey":{"networkName":"","containerID":"","ifName":""},"Address":"192.168.67.189","UnassignedTime":"0001-01-01T00:00:00Z"},"192.168.69.119":{"IPAMKey":{"networkName":"","containerID":"","ifName":""},"Address":"192.168.69.119","UnassignedTime":"0001-01-01T00:00:00Z"},"192.168.73.173":{"IPAMKey":{"networkName":"","containerID":"","ifName":""},"Address":"192.168.73.173","UnassignedTime":"0001-01-01T00:00:00Z"},"192.168.74.37":{"IPAMKey":{"networkName":"","containerID":"","ifName":""},"Address":"192.168.74.37","UnassignedTime":"0001-01-01T00:00:00Z"},...

@mogren mogren force-pushed the fix-introspection-panic branch from c24a70a to ce5c3a5 Compare August 31, 2020 15:56
@SaranBalaji90 SaranBalaji90 merged commit 93f934d into aws:master Aug 31, 2020
@mogren mogren deleted the fix-introspection-panic branch September 4, 2020 05:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants