changes to fix issue #2312

[maniSbindra]

What this PR does / why we need it: This PR fixes issue #2312

Which issue this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close that issue when PR gets merged): fixes #2312

Special notes for your reviewer:
Things checked are

• Policy file details from kube-apiserver logs (Loaded 8 audit policy rules from file /etc/kubernetes/addons/audit-policy.yaml)

$ kubectl --namespace=kube-system logs  kube-apiserver-k8s-master-38666652-0  | grep -i audit 
I0701 16:55:28.256862       1 reader.go:56] Loaded 8 audit policy rules from file /etc/kubernetes/addons/audit-policy.yaml
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters.(*auditResponseWriter).WriteHeader(0xc4225b6640, 0x1f4)
	/workspace/anago-v1.8.14-beta.0.16+9d72fafc46543e/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters/audit.go:215 +0x63
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithAudit.func1(0x7f3176f9ea40, 0xc424f4a9a8, 0xc4254abd00)
	/workspace/anago-v1.8.14-beta.0.16+9d72fafc46543e/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters/audit.go:105 +0x52a
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters.(*auditResponseWriter).WriteHeader(0xc42404d9a0, 0x1f4)
	/workspace/anago-v1.8.14-beta.0.16+9d72fafc46543e/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters/audit.go:215 +0x63
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithAudit.func1(0x7f3176f9ea40, 0xc42000f8f8, 0xc42282d100)
	/workspace/anago-v1.8.14-beta.0.16+9d72fafc46543e/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters/audit.go:105 +0x52a
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters.(*auditResponseWriter).WriteHeader(0xc427e51c70, 0x1f4)
	/workspace/anago-v1.8.14-beta.0.16+9d72fafc46543e/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters/audit.go:215 +0x63
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithAudit.func1(0x7f3176f9ea40, 0xc42be54668, 0xc42b0ee300)
	/workspace/anago-v1.8.14-beta.0.16+9d72fafc46543e/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters/audit.go:105 +0x52a
IND-CSE:kubeconfig mani$ kl run busybox --image=busybox

• Audit log after exec into API server pod (events related to busybox pod i created)

$ grep busybox /var/log/audit.log | tail -2
{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":"2018-07-01T17:25:51Z"},"level":"Request","timestamp":"2018-07-01T17:25:51Z","auditID":"9b120309-556a-4348-b63f-e712390d9c62","stage":"RequestReceived","requestURI":"/api/v1/namespaces/default/events/busybox-6944bc9f7b-v2888.153d4e658978585c","verb":"patch","user":{"username":"client","groups":["system:masters","system:authenticated"]},"sourceIPs":["10.240.0.4"],"objectRef":{"resource":"events","namespace":"default","name":"busybox-6944bc9f7b-v2888.153d4e658978585c","apiVersion":"v1"}}
{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":"2018-07-01T17:25:51Z"},"level":"Request","timestamp":"2018-07-01T17:25:51Z","auditID":"9b120309-556a-4348-b63f-e712390d9c62","stage":"ResponseComplete","requestURI":"/api/v1/namespaces/default/events/busybox-6944bc9f7b-v2888.153d4e658978585c","verb":"patch","user":{"username":"client","groups":["system:masters","system:authenticated"]},"sourceIPs":["10.240.0.4"],"objectRef":{"resource":"events","namespace":"default","name":"busybox-6944bc9f7b-v2888.153d4e658978585c","apiVersion":"v1"},"responseStatus":{"metadata":{},"code":200},"requestObject":{"count":25,"lastTimestamp":"2018-07-01T17:25:51Z","message":"Error syncing pod"}}

• Top few lines of Kubelet logs from master node

$ journalctl -u kubelet
-- Logs begin at Sun 2018-07-01 16:54:34 UTC, end at Sun 2018-07-01 17:22:31 UTC. --
Jul 01 16:54:33 k8s-master-38666652-0 systemd[1]: Stopped Kubelet.
Jul 01 16:54:33 k8s-master-38666652-0 systemd[1]: Starting Kubelet...
Jul 01 16:54:33 k8s-master-38666652-0 sysctl[9333]: net.ipv4.tcp_retries2 = 8
Jul 01 16:54:33 k8s-master-38666652-0 ebtables[9336]: Bridge table: nat
Jul 01 16:54:33 k8s-master-38666652-0 ebtables[9336]: Bridge chain: PREROUTING, entries: 0, policy: ACCEPT
Jul 01 16:54:33 k8s-master-38666652-0 ebtables[9336]: Bridge chain: OUTPUT, entries: 0, policy: ACCEPT
Jul 01 16:54:33 k8s-master-38666652-0 ebtables[9336]: Bridge chain: POSTROUTING, entries: 0, policy: ACCEPT
Jul 01 16:54:33 k8s-master-38666652-0 iptables[9339]: Chain PREROUTING (policy ACCEPT)
Jul 01 16:54:33 k8s-master-38666652-0 iptables[9339]: target     prot opt source               destination
Jul 01 16:54:33 k8s-master-38666652-0 iptables[9339]: DOCKER     all  --  anywhere             anywhere             ADDRTYPE match dst-type LOCAL
Jul 01 16:54:33 k8s-master-38666652-0 iptables[9339]: Chain INPUT (policy ACCEPT)
Jul 01 16:54:33 k8s-master-38666652-0 iptables[9339]: target     prot opt source               destination
Jul 01 16:54:33 k8s-master-38666652-0 iptables[9339]: Chain OUTPUT (policy ACCEPT)
Jul 01 16:54:33 k8s-master-38666652-0 iptables[9339]: target     prot opt source               destination
Jul 01 16:54:33 k8s-master-38666652-0 iptables[9339]: DOCKER     all  --  anywhere            !127.0.0.0/8          ADDRTYPE match dst-type LOCAL
Jul 01 16:54:33 k8s-master-38666652-0 iptables[9339]: Chain POSTROUTING (policy ACCEPT)
Jul 01 16:54:33 k8s-master-38666652-0 iptables[9339]: target     prot opt source               destination
Jul 01 16:54:33 k8s-master-38666652-0 iptables[9339]: MASQUERADE  all  --  172.17.0.0/16        anywhere
Jul 01 16:54:33 k8s-master-38666652-0 iptables[9339]: MASQUERADE  all  --  anywhere            !10.0.0.0/8           destination IP range ! 168.63.129.16-168.63.129.16 ADDRTYPE match dst-type !LOCAL
Jul 01 16:54:33 k8s-master-38666652-0 iptables[9339]: Chain DOCKER (2 references)
Jul 01 16:54:33 k8s-master-38666652-0 iptables[9339]: target     prot opt source               destination
Jul 01 16:54:33 k8s-master-38666652-0 iptables[9339]: RETURN     all  --  anywhere             anywhere
Jul 01 16:54:33 k8s-master-38666652-0 systemd[1]: Started Kubelet.
Jul 01 16:54:34 k8s-master-38666652-0 kubelet[9343]: Flag --keep-terminated-pod-volumes has been deprecated, will be removed in a future version
Jul 01 16:54:34 k8s-master-38666652-0 kubelet[9343]: Flag --non-masquerade-cidr has been deprecated, will be removed in a future version
Jul 01 16:54:34 k8s-master-38666652-0 kubelet[9343]: I0701 16:54:34.040371    9343 feature_gate.go:162] feature gates: map[]
Jul 01 16:54:35 k8s-master-38666652-0 kubelet[9343]: I0701 16:54:35.036757    9343 mount_linux.go:213] Detected OS with systemd
Jul 01 16:54:35 k8s-master-38666652-0 kubelet[9343]: I0701 16:54:35.036783    9343 client.go:75] Connecting to docker on unix:///var/run/docker.sock
Jul 01 16:54:35 k8s-master-38666652-0 kubelet[9343]: I0701 16:54:35.036805    9343 client.go:95] Start docker client with request timeout=2m0s
Jul 01 16:54:35 k8s-master-38666652-0 kubelet[9343]: I0701 16:54:35.061277    9343 azure.go:175] azure: using client_id+client_secret to retrieve access token
Jul 01 16:54:35 k8s-master-38666652-0 kubelet[9343]: I0701 16:54:35.457589    9343 server.go:301] Successfully initialized cloud provider: "azure" from the config file: "/etc/kubernetes/azure.json"
Jul 01 16:54:35 k8s-master-38666652-0 kubelet[9343]: I0701 16:54:35.457643    9343 server.go:537] cloud provider determined current node name to be k8s-master-38666652-0
Jul 01 16:54:35 k8s-master-38666652-0 kubelet[9343]: I0701 16:54:35.460443    9343 manager.go:149] cAdvisor running in container: "/sys/fs/cgroup/cpu,cpuacct/system.slice/kubelet.service"
Jul 01 16:54:35 k8s-master-38666652-0 kubelet[9343]: W0701 16:54:35.472537    9343 manager.go:157] unable to connect to Rkt api service: rkt: cannot tcp Dial rkt api service: dial tcp 127.0.0.1:15441: get
Jul 01 16:54:35 k8s-master-38666652-0 kubelet[9343]: W0701 16:54:35.472683    9343 manager.go:166] unable to connect to CRI-O api service: Get http://%2Fvar%2Frun%2Fcrio.sock/info: dial unix /var/run/crio
Jul 01 16:54:35 k8s-master-38666652-0 kubelet[9343]: I0701 16:54:35.481054    9343 fs.go:139] Filesystem UUIDs: map[626e2fcc-121d-43b3-9ade-7c313f813a52:/dev/sdc1 84e91f24-0077-40f3-aab8-b49b8e1f4506:/dev
Jul 01 16:54:35 k8s-master-38666652-0 kubelet[9343]: I0701 16:54:35.481083    9343 fs.go:140] Filesystem partitions: map[tmpfs:{mountpoint:/run major:0 minor:23 fsType:tmpfs blockSize:0} /dev/sda1:{mountp
Jul 01 16:54:35 k8s-master-38666652-0 kubelet[9343]: I0701 16:54:35.482749    9343 manager.go:216] Machine: {NumCores:2 CpuFrequency:2394453 MemoryCapacity:7284019200 HugePages:[{PageSize:1048576 NumPages
Jul 01 16:54:35 k8s-master-38666652-0 kubelet[9343]: I0701 16:54:35.483281    9343 manager.go:222] Version: {KernelVersion:4.15.0-1013-azure ContainerOsVersion:Ubuntu 16.04.4 LTS DockerVersion:1.13.1 Dock
Jul 01 16:54:35 k8s-master-38666652-0 kubelet[9343]: I0701 16:54:35.483792    9343 server.go:425] --cgroups-per-qos enabled, but --cgroup-root was not specified.  defaulting to /
Jul 01 16:54:35 k8s-master-38666652-0 kubelet[9343]: I0701 16:54:35.485424    9343 container_manager_linux.go:252] container manager verified user specified cgroup-root exists: /
Jul 01 16:54:35 k8s-master-38666652-0 kubelet[9343]: I0701 16:54:35.485450    9343 container_manager_linux.go:257] Creating Container Manager object based on Node Config: {RuntimeCgroupsName: SystemCgroup
Jul 01 16:54:35 k8s-master-38666652-0 kubelet[9343]: I0701 16:54:35.485533    9343 container_manager_linux.go:288] Creating device plugin handler: false
Jul 01 16:54:35 k8s-master-38666652-0 kubelet[9343]: I0701 16:54:35.485598    9343 server.go:537] cloud provider determined current node name to be k8s-master-38666652-0
Jul 01 16:54:35 k8s-master-38666652-0 kubelet[9343]: I0701 16:54:35.485609    9343 server.go:689] Using root directory: /var/lib/kubelet
Jul 01 16:54:35 k8s-master-38666652-0 kubelet[9343]: I0701 16:54:35.485651    9343 kubelet.go:350] cloud provider determined current node name to be k8s-master-38666652-0
Jul 01 16:54:35 k8s-master-38666652-0 kubelet[9343]: I0701 16:54:35.485673    9343 kubelet.go:275] Adding manifest file: /etc/kubernetes/manifests
Jul 01 16:54:35 k8s-master-38666652-0 kubelet[9343]: I0701 16:54:35.485691    9343 file.go:52] Watching path "/etc/kubernetes/manifests"
Jul 01 16:54:35 k8s-master-38666652-0 kubelet[9343]: I0701 16:54:35.485697    9343 kubelet.go:285] Watching apiserver
Jul 01 16:54:35 k8s-master-38666652-0 kubelet[9343]: E0701 16:54:35.489550    9343 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:415: Failed to list *v1.Service: Get https://10.255.255.5:443/
Jul 01 16:54:35 k8s-master-38666652-0 kubelet[9343]: E0701 16:54:35.489754    9343 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://10.255.255.5:
Jul 01 16:54:35 k8s-master-38666652-0 kubelet[9343]: E0701 16:54:35.489842    9343 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:424: Failed to list *v1.Node: Get https://10.255.255.5:443/api
Jul 01 16:54:35 k8s-master-38666652-0 kubelet[9343]: W0701 16:54:35.520356    9343 kubelet_network.go:69] Hairpin mode set to "promiscuous-bridge" but kubenet is not enabled, falling back to "hairpin-veth
Jul 01 16:54:35 k8s-master-38666652-0 kubelet[9343]: I0701 16:54:35.520394    9343 kubelet.go:520] Hairpin mode set to "hairpin-veth"
Jul 01 16:54:35 k8s-master-38666652-0 kubelet[9343]: I0701 16:54:35.520580    9343 plugins.go:187] Loaded network plugin "cni"
Jul 01 16:54:35 k8s-master-38666652-0 kubelet[9343]: I0701 16:54:35.545547    9343 plugins.go:187] Loaded network plugin "cni"
Jul 01 16:54:35 k8s-master-38666652-0 kubelet[9343]: I0701 16:54:35.545908    9343 docker_service.go:207] Docker cri networking managed by cni
Jul 01 16:54:35 k8s-master-38666652-0 kubelet[9343]: I0701 16:54:35.555501    9343 docker_service.go:212] Docker Info: &{ID:OP6F:DLWT:KXR6:2KFV:JIAG:RZJZ:CPO4:XA4T:IBGQ:EXQA:IRUU:H6RQ Containers:0 Contain
Jul 01 16:54:35 k8s-master-38666652-0 kubelet[9343]: I0701 16:54:35.555936    9343 docker_service.go:225] Setting cgroupDriver to cgroupfs
Jul 01 16:54:35 k8s-master-38666652-0 kubelet[9343]: I0701 16:54:35.557912    9343 docker_legacy.go:151] No legacy containers found, stop performing legacy cleanup.
Jul 01 16:54:35 k8s-master-38666652-0 kubelet[9343]: I0701 16:54:35.557952    9343 kubelet.go:609] Starting the GRPC server for the docker CRI shim.
Jul 01 16:54:35 k8s-master-38666652-0 kubelet[9343]: I0701 16:54:35.557966    9343 docker_server.go:51] Start dockershim grpc server
Jul 01 16:54:35 k8s-master-38666652-0 kubelet[9343]: I0701 16:54:35.566414    9343 remote_runtime.go:43] Connecting to runtime service unix:///var/run/dockershim.sock

If applicable:

• documentation
• unit tests
• tested backward compatibility (ie. deploy with previous version, upgrade with this branch)

Release note:

[codecov]

Codecov Report

Merging #3390 into master will decrease coverage by 1.86%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master    #3390      +/-   ##
==========================================
- Coverage   54.59%   52.72%   -1.87%     
==========================================
  Files         104      103       -1     
  Lines       15778    15461     -317     
==========================================
- Hits         8614     8152     -462     
- Misses       6412     6569     +157     
+ Partials      752      740      -12

[jackfrancis]

/lgtm

[acs-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jackfrancis, maniSbindra

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [jackfrancis]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment