test flake: capacity metrics check fails

[pohly]

Sometimes, go test ./pkg/capacity fails. I can reproduce this locally by running it continuously until it fails:

$ i=0; while go  test -count=1 ./pkg/capacity/; do echo $i; i=$((i + 1)); done
ok  	github.com/kubernetes-csi/external-provisioner/pkg/capacity	12.248s
0
ok  	github.com/kubernetes-csi/external-provisioner/pkg/capacity	12.243s
1
ok  	github.com/kubernetes-csi/external-provisioner/pkg/capacity	12.245s
2
ok  	github.com/kubernetes-csi/external-provisioner/pkg/capacity	12.246s
3
ok  	github.com/kubernetes-csi/external-provisioner/pkg/capacity	12.246s
4
--- FAIL: TestCapacityController (0.00s)
    --- FAIL: TestCapacityController/two_segments,_two_classes,_four_objects_updated (0.01s)
        capacity.go:336: INFO onTopologyChanges: topology changed added=[0x27ef470 = layer0: foo 0x27ef490 = layer0: bar] removed=[]
        capacity.go:371: INFO onSCAddOrUpdate: updated or added storageclass="direct-sc"
        capacity.go:479: INFO onSCAddOrUpdate: enqueuing storageclass="direct-sc" workitem={segment:0x27ef470 storageClassName:direct-sc}
        capacity.go:479: INFO onSCAddOrUpdate: enqueuing storageclass="direct-sc" workitem={segment:0x27ef490 storageClassName:direct-sc}
        capacity.go:371: INFO onSCAddOrUpdate: updated or added storageclass="triple-sc"
        capacity.go:479: INFO onSCAddOrUpdate: enqueuing storageclass="triple-sc" workitem={segment:0x27ef470 storageClassName:triple-sc}
        capacity.go:336: INFO onTopologyChanges: topology changed added=[0x27ef470 = layer0: foo 0x27ef490 = layer0: bar] removed=[]
        capacity.go:479: INFO onSCAddOrUpdate: enqueuing storageclass="triple-sc" workitem={segment:0x27ef490 storageClassName:triple-sc}
        capacity.go:479: INFO onTopologyChanges: enqueuing workitem={segment:0x27ef470 storageClassName:direct-sc}
        capacity.go:479: INFO onTopologyChanges: enqueuing workitem={segment:0x27ef490 storageClassName:direct-sc}
        capacity.go:479: INFO onTopologyChanges: enqueuing workitem={segment:0x27ef470 storageClassName:triple-sc}
        capacity.go:479: INFO onTopologyChanges: enqueuing workitem={segment:0x27ef490 storageClassName:triple-sc}
        capacity.go:274: INFO initial state topology segments=2 storage classes=2 potential CSIStorageCapacity objects=4
        capacity.go:286: INFO checking for existing CSIStorageCapacity objects
        capacity.go:729: INFO onCAddOrUpdate: found match capacity="testns/csisc-9" resourceVersion="CSISC-REV-0" workitem={segment:0x27ef470 storageClassName:direct-sc}
        capacity.go:719: INFO onCAddOrUpdate: CSIStorageCapacity already known to match capacity="testns/csisc-9" resourceVersion="CSISC-REV-0" workitem={segment:0x27ef470 storageClassName:direct-sc}
        capacity_test.go:1129: metrics after prepare: expected goal/current/obsolete object numbers 4/4/0: 
            metric output does not match expectation; want:
            
            # HELP csistoragecapacities_desired_current [ALPHA] Number of CSIStorageCapacity objects that exist and are supposed to be managed automatically.
            # TYPE csistoragecapacities_desired_current gauge
            csistoragecapacities_desired_current 4
            # HELP csistoragecapacities_desired_goal [ALPHA] Number of CSIStorageCapacity objects that are supposed to be managed automatically.
            # TYPE csistoragecapacities_desired_goal gauge
            csistoragecapacities_desired_goal 4
            # HELP csistoragecapacities_obsolete [ALPHA] Number of CSIStorageCapacity objects that exist and will be deleted automatically. Objects that exist and may need an update are not considered obsolete and therefore not included in this value.
            # TYPE csistoragecapacities_obsolete gauge
            csistoragecapacities_obsolete 0
            
            got:
            
            # HELP csistoragecapacities_desired_current [ALPHA] Number of CSIStorageCapacity objects that exist and are supposed to be managed automatically.
            # TYPE csistoragecapacities_desired_current gauge
            csistoragecapacities_desired_current 1
            # HELP csistoragecapacities_desired_goal [ALPHA] Number of CSIStorageCapacity objects that are supposed to be managed automatically.
            # TYPE csistoragecapacities_desired_goal gauge
            csistoragecapacities_desired_goal 4
            # HELP csistoragecapacities_obsolete [ALPHA] Number of CSIStorageCapacity objects that exist and will be deleted automatically. Objects that exist and may need an update are not considered obsolete and therefore not included in this value.
            # TYPE csistoragecapacities_obsolete gauge
            csistoragecapacities_obsolete 0
FAIL
FAIL	github.com/kubernetes-csi/external-provisioner/pkg/capacity	12.250s
FAIL

[pohly]

To get useful log messages, #639 is needed. Without it, go test ./pkg/capacity -args -v=5 has to be used, but that mixes output from all tests that run in parallel and is impossible to read.