How to enable auditing?

[bygui86]

The exact command to reproduce the issue:

minikube start \
--vm-driver hyperkit \
--extra-config=apiserver.authorization-mode=RBAC \
--extra-config=apiserver.Audit.LogOptions.Path=/var/log/apiserver/audit.log \
--extra-config=apiserver.Audit.PolicyFile=/etc/kubernetes/addons/audit-policy.yaml

I tried several extra-config variants:

• apiserver.Audit.LogOptions.Path, apiserver.Audit.PolicyFile
• apiserver.audit-logoptions-path, apiserver.audit-policyfile
• apiserver.audit-log-options-path, apiserver.audit-policy-file
• apiserver.audit-policy-path, apiserver.audit-log-dir, apiserver.audit-log-max-age

together with different values:

• log options path / log path / path
  • /var/log/kube-apiserver-audit.log
  • /etc/kubernetes/logs/apiserver-audit.log
• log dir
  • /var/log/kubernetes/
• policy path
  • /etc/kubernetes/addons/audit-policy.yaml

I tried also:

• --feature-gates=Auditing=true
• --feature-gates=AdvancedAuditing=true

It does not matter how many CPUs or memory I give, minikube does not start in any case :(

The full output of the command that failed:
Please see all attached files

The output of the minikube logs command:
No output because minikube does not start.

The operating system version:
MacBook Pro - macOS Mojave 10.14.5
Minikube v1.1.1

I already had a look at the issue #1609, but it doesn't helped :(

I attached all attempts I did.
If someone has ideas or can help it would be great!
Thanks guys!!

attempt-1.log
attempt-2.log
attempt-3.log
attempt-4.log
attempt-5.log
attempt-6.log
attempt-7.log
attempt-8.log
attempt-9.log
attempt-10.log

[tstromberg]

Our documentation isn't great here, but the output is fairly specific about what the problem is in your various attempts. I only checked 3, but:

❌  Problems detected in "kube-apiserver":
    error: unknown flag: --Audit.LogOptions.Path

    error: unknown flag: --audit-path

    error: unknown flag: --audit-max-age

https://kubernetes.io/docs/reference/command-line-tools-reference/kube-apiserver/ has the list of known good apiserver flags.

If you don't mind telling me which of the supported apiserver flags you are trying to set, I can help you translate it into the appropriate minikube command-line.

[tstromberg]

It's also worth noting that the documentation probably reflects Kubernetes v1.15, so you may want to upgrade to minikube v1.2 first.

[bygui86]

@tstromberg thank you very much for your answer.
I did this test before the release of Kubernetes 1.15 and Minikube 1.2. Now I upgraded already both.

This is the list of flags I want to enable:
--audit-dynamic-configuration
--audit-policy-file
--audit-log-path
--audit-log-mode
--audit-log-maxage
--audit-log-maxbackup
--audit-log-maxsize
--feature-gates
DynamicAuditing=true

The final goal is to enable auditing to run some security checks, for example using JQ filter for audit-events at the beginning and later on Falco for production.

[tstromberg]

Try this on:

minikube start \
  --extra-config=apiserver.audit-dynamic-configuration=true \
  --extra-config=apiserver.audit-policy-file=/some-path \
  --extra-config=apiserver.audit-log-path=- \
  --extra-config=apiserver.audit-log-mode=batch \
  --extra-config=apiserver.audit-log-maxage=5 \
  --extra-config=apiserver.audit-log-maxbackup=5 \
  --feature-gates=DynamicAuditing=true

Two notes:

• You'll need to provide a policy file. This example assumes that you've added one to ~/.minikube/files/some-path (which gets copied into the VM).
• If you have to wait >30 seconds for Kubernetes to start up, run this in another window to show what is almost certainly a flag issue:
  minikube logs --problems

[tstromberg]

If you find a command that works for you - please share it!

[bygui86]

Great thanks! I will try asap and keep you posted!

[bygui86]

After some tests, I have some findings...

These flags seems to be correct:

--extra-config=apiserver.audit-dynamic-configuration=true
--extra-config=apiserver.audit-log-path=-
--extra-config=apiserver.audit-log-mode=batch
--extra-config=apiserver.audit-log-maxage=5
--extra-config=apiserver.audit-log-maxbackup=5
--feature-gates=DynamicAuditing=true

but I can't be sure because I don't know how to verify and where to find logs... But at least Minikube starts properly.

This flag is not correct or the path is not correct:

--extra-config=apiserver.audit-policy-file=/some-path

Output of minikube logs --problems:

❌  Problems detected in "kube-addon-manager":
    error: unable to recognize "STDIN": Get https://localhost:8443/api?timeout=32s: dial tcp 127.0.0.1:8443: connect: connection refused
❌  Problems detected in "kube-apiserver":
    error: loading audit policy file: failed to read file path "/audit-policy/audit-policy.yaml": open /audit-policy/audit-policy.yaml: no such file or directory

I put the attached file (audit-policy.yaml) in the folder ~/.minikube/files/audit-policy/

@tstromberg what do you think?

[bygui86]

Sorry @tstromberg, any news about this flag?
Do you have the same issue?

[brichins]

For what it's worth, I think you'd have to put the audit-policy.yaml file in a minikube directory that gets mounted by kubeadm to apiserver. I'm guessing you'd need to access the kubeadm extraVolumes config (https://godoc.org/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1beta1).

[brichins]

For what it's worth, I was able to create a policy file as ~/.minikube/files/etc/ssl/certs/audit-policy.yaml since /etc/ssl/certs is already mounted by the apiserver container.

minikube start \
  --extra-config=apiserver.audit-policy-file=/etc/ssl/certs/audit-policy.yaml \
  --extra-config=apiserver.audit-log-path=-

[bygui86]

@brichins thanks for the tip! I will try once back from holidays :)
Meanwhile I found something like your example, but I was not able to test it yet.
Keep you posted on this!

[bygui86]

@brichins can you share the audit-policy.yaml you used? I want to try to reproduce a sample same as yours

[bygui86]

I tested and I can run same as @brichins

minikube start \
  --extra-config=apiserver.audit-policy-file=/etc/ssl/certs/audit-policy.yaml \
  --extra-config=apiserver.audit-log-path=-

But running minikube logs -f I don't see any specific auditing logs :(

[g-boros]

@bygui86 above steps worked fine with the example config from audit doc:

± % cat ~/.minikube/files/etc/ssl/certs/audit-policy.yaml
# Log all requests at the Metadata level.
apiVersion: audit.k8s.io/v1
kind: Policy
rules:
- level: Metadata

I can see audit logs in apiserver's stdout:

± % k logs kube-apiserver-minikube -n kube-system | tail -n5
{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"2147b766-6469-4c74-bf20-1fd7b2c52712","stage":"RequestReceived","requestURI":"/api/v1/namespaces/kube-system/endpoints/kube-controller-manager?timeout=10s","verb":"update","user":{"username":"system:kube-controller-manager","groups":["system:authenticated"]},"sourceIPs":["127.0.0.1"],"userAgent":"kube-controller-manager/v1.16.0 (linux/amd64) kubernetes/2bd9643/leader-election","objectRef":{"resource":"endpoints","namespace":"kube-system","name":"kube-controller-manager","apiVersion":"v1"},"requestReceivedTimestamp":"2019-09-20T12:29:15.164614Z","stageTimestamp":"2019-09-20T12:29:15.164614Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"1e20cce1-8aef-4cb9-94f9-25b8a4207db3","stage":"ResponseComplete","requestURI":"/api/v1/namespaces/kube-system/endpoints/kube-scheduler?timeout=10s","verb":"get","user":{"username":"system:kube-scheduler","groups":["system:authenticated"]},"sourceIPs":["127.0.0.1"],"userAgent":"kube-scheduler/v1.16.0 (linux/amd64) kubernetes/2bd9643/leader-election","objectRef":{"resource":"endpoints","namespace":"kube-system","name":"kube-scheduler","apiVersion":"v1"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2019-09-20T12:29:15.154858Z","stageTimestamp":"2019-09-20T12:29:15.175613Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by ClusterRoleBinding \"system:kube-scheduler\" of ClusterRole \"system:kube-scheduler\" to User \"system:kube-scheduler\""}}
{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"f8df99ff-4408-4bb3-a321-ef9ca64706ed","stage":"RequestReceived","requestURI":"/api/v1/namespaces/kube-system/endpoints/kube-scheduler?timeout=10s","verb":"update","user":{"username":"system:kube-scheduler","groups":["system:authenticated"]},"sourceIPs":["127.0.0.1"],"userAgent":"kube-scheduler/v1.16.0 (linux/amd64) kubernetes/2bd9643/leader-election","objectRef":{"resource":"endpoints","namespace":"kube-system","name":"kube-scheduler","apiVersion":"v1"},"requestReceivedTimestamp":"2019-09-20T12:29:15.180177Z","stageTimestamp":"2019-09-20T12:29:15.180177Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"2147b766-6469-4c74-bf20-1fd7b2c52712","stage":"ResponseComplete","requestURI":"/api/v1/namespaces/kube-system/endpoints/kube-controller-manager?timeout=10s","verb":"update","user":{"username":"system:kube-controller-manager","groups":["system:authenticated"]},"sourceIPs":["127.0.0.1"],"userAgent":"kube-controller-manager/v1.16.0 (linux/amd64) kubernetes/2bd9643/leader-election","objectRef":{"resource":"endpoints","namespace":"kube-system","name":"kube-controller-manager","uid":"7b6ac3ea-bbde-4c42-8438-d44a4c02558b","apiVersion":"v1","resourceVersion":"4055"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2019-09-20T12:29:15.164614Z","stageTimestamp":"2019-09-20T12:29:15.182485Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by ClusterRoleBinding \"system:kube-controller-manager\" of ClusterRole \"system:kube-controller-manager\" to User \"system:kube-controller-manager\""}}
{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"f8df99ff-4408-4bb3-a321-ef9ca64706ed","stage":"ResponseComplete","requestURI":"/api/v1/namespaces/kube-system/endpoints/kube-scheduler?timeout=10s","verb":"update","user":{"username":"system:kube-scheduler","groups":["system:authenticated"]},"sourceIPs":["127.0.0.1"],"userAgent":"kube-scheduler/v1.16.0 (linux/amd64) kubernetes/2bd9643/leader-election","objectRef":{"resource":"endpoints","namespace":"kube-system","name":"kube-scheduler","uid":"44fc8be7-af10-44e9-9350-54e63fa3871e","apiVersion":"v1","resourceVersion":"4056"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2019-09-20T12:29:15.180177Z","stageTimestamp":"2019-09-20T12:29:15.182903Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by ClusterRoleBinding \"system:kube-scheduler\" of ClusterRole \"system:kube-scheduler\" to User \"system:kube-scheduler\""}}

[bygui86]

Thanks @g-boros!! I will git it a try asap.

[tstromberg]

Does someone mind writing this up a tutorial? Depending on how it's written, I'd love to see it added to either

https://minikube.sigs.k8s.io/docs/tutorials/ or https://minikube.sigs.k8s.io/docs/reference/

[bygui86]

@g-boros I tested it as well and it works also on my side!

@tstromberg good idea!

[tstromberg]

Closing as a workaround was found.

@bygui86 - if you write a tutorial, I would be happy to review it for addition to our docs =)

[tstromberg]

Re-opening as the issue of this not being documented has not been resolved yet.

[medyagh]

This is a good first issue, to add a tutorial on minikube website, how to enable minikube auditing and maybe add a simple example of audting usage

I will be happy to review the PR for it

[bygui86]

Sorry @tstromberg :( too busy to write down the documentation...
Many thanks to @JanAhrens to have made it!

[bygui86]

The documentation of @JanAhrens is really good! Precise, concise and clear.
Great!

[JanAhrens]

Thanks @bygui86 and everyone who contributed. I found the steps in this issue very helpful.

[joelhoisko]

I've tried following the tutorial with the audit-policy.yaml workaraound file with this command :

minikube start \                                   
  --extra-config=apiserver.audit-policy-file=/etc/ssl/certs/audit-policy.yaml \
  --extra-config=apiserver.audit-log-path=-                               
😄  minikube v1.14.1 on Ubuntu 20.04
✨  Using the docker driver based on existing profile
👍  Starting control plane node minikube in cluster minikube
🔄  Restarting existing docker container for "minikube" ...
🐳  Preparing Kubernetes v1.19.2 on Docker 19.03.8 ...
    ▪ apiserver.audit-policy-file=/etc/ssl/certs/audit-policy.yaml
    ▪ apiserver.audit-log-path=/var/log/audit.log
🔎  Verifying Kubernetes components...
🌟  Enabled addons: storage-provisioner, default-storageclass
🏄  Done! kubectl is now configured to use "minikube" by default

but I just can't seem to get it to work on my machine. Running

kubectl logs kube-apiserver-minikube -n kube-system | grep audit.k8s.io/v1

Returns nothing, while the normal logs still keep looking that look like this with no extra loging:

...
I1211 10:25:26.393340       1 clientconn.go:948] ClientConn switching balancer to "pick_first"
I1211 10:26:03.066193       1 client.go:360] parsed scheme: "passthrough"
I1211 10:26:03.066233       1 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{https://127.0.0.1:2379  <nil> 0 <nil>}] <nil> <nil>}
I1211 10:26:03.066241       1 clientconn.go:948] ClientConn switching balancer to "pick_first"
...

I've also tried to add the --feature-gates=AdvancedAuditing=true and that gives me some extra Trace logs but still nothing regarding audit.k8s. I've also tried to set the --extra-config=apiserver.audit-log-path=/var/log/audit.log but that file never appeared. I can find the audit-policy.yaml file on the minikube but it doesn't seem to be doing anything.

Any leads? Do I need to enable some addons or something for this to work?

[kmohanram]

It appears that the audit parameters can be set on a fresh install (verified via minikube ssh and a check of the apiserver manifest file), but minikube start hangs as seen below (minikube ssh functions through another window fwiw). Would be extremely useful if auditing can be supported out of the box in the minikube environment. A vanilla minikube start without audit works well, so the breakdown occurs when setting up audit. Trying to setup audit on a stop and restart of a vanilla install fails as the --extra-config parameters never make it to the manifests (so the logs do not contain anything useful as reported elsewhere as well). Error reproduced using parallels and docker drivers.

$ minikube start --extra-config=apiserver.audit-policy-file=/etc/kubernetes/addons/audit-policy.yaml --extra-config=apiserver.audit-log-path=/var/log/audit.log --driver hyperkit --extra-config=apiserver.authorization-mode=RBAC

😄 minikube v1.15.1 on Darwin 10.15.7
✨ Using the hyperkit driver based on existing profile
👍 Starting control plane node minikube in cluster minikube
🔄 Restarting existing hyperkit VM for "minikube" ...
🐳 Preparing Kubernetes v1.19.4 on Docker 19.03.13 ...
▪ apiserver.audit-policy-file=/etc/kubernetes/addons/audit-policy.yaml
▪ apiserver.audit-log-path=/var/log/audit.log
▪ apiserver.authorization-mode=Node,RBAC
❌ Problems detected in kubelet:
Dec 20 02:43:04 minikube kubelet[2379]: E1220 02:43:04.623905 2379 pod_workers.go:191] Error syncing pod 63fe48f5aa7eb3075923540a7b9d93d3 ("kube-apiserver-minikube_kube-system(63fe48f5aa7eb3075923540a7b9d93d3)"), skipping: failed to "StartContainer" for "kube-apiserver" with CrashLoopBackOff: "back-off 20s restarting failed container=kube-apiserver pod=kube-apiserver-minikube_kube-system(63fe48f5aa7eb3075923540a7b9d93d3)"
Dec 20 02:43:10 minikube kubelet[2379]: E1220 02:43:10.000215 2379 pod_workers.go:191] Error syncing pod 6cb144f7d82285562d6fc7ed0aeee754 ("kube-controller-manager-minikube_kube-system(6cb144f7d82285562d6fc7ed0aeee754)"), skipping: failed to "StartContainer" for "kube-controller-manager" with CrashLoopBackOff: "back-off 20s restarting failed container=kube-controller-manager pod=kube-controller-manager-minikube_kube-system(6cb144f7d82285562d6fc7ed0aeee754)"
Dec 20 02:43:12 minikube kubelet[2379]: E1220 02:43:12.895104 2379 pod_workers.go:191] Error syncing pod 6cb144f7d82285562d6fc7ed0aeee754 ("kube-controller-manager-minikube_kube-system(6cb144f7d82285562d6fc7ed0aeee754)"), skipping: failed to "StartContainer" for "kube-controller-manager" with CrashLoopBackOff: "back-off 20s restarting failed container=kube-controller-manager pod=kube-controller-manager-minikube_kube-system(6cb144f7d82285562d6fc7ed0aeee754)"