Audit Logfile Not Created

[jcantosz]

Is this a BUG REPORT or FEATURE REQUEST? : Bug report

Minikube version (use minikube version): minikube version: v0.19.0

Environment:

• OS (e.g. from /etc/os-release):
  $ sw_vers
  ProductName: Mac OS X
  ProductVersion: 10.12.5
  BuildVersion: 16F73

• VM Driver (e.g. cat ~/.minikube/machines/minikube/config.json | grep DriverName):
  "DriverName": "virtualbox",

• ISO version (e.g. cat ~/.minikube/machines/minikube/config.json | grep -i ISO or minikube ssh cat /etc/VERSION):
  v0.18.0

• Install tools:

• Others:

What happened:
Started minikube with audit log options. Logs were not generated at the location specified.

$ minikube ssh cat /var/log/apiserver/audit.log
cat: /var/log/apiserver/audit.log: No such file or directory
E0619 15:35:04.404845   30420 ssh.go:44] Error attempting to ssh/run-ssh-command: exit status 1

What you expected to happen:
Audit log creation and output from server.

How to reproduce it (as minimally and precisely as possible):
minikube start --extra-config=apiserver.Authorization.Mode=RBAC --extra-config=apiserver.Audit.LogOptions.Path=/var/log/apiserver/audit.log --extra-config=apiserver.Audit.LogOptions.MaxAge=30 --extra-config=apiserver.Audit.LogOptions.MaxSize=100 --extra-config=apiserver.Audit.LogOptions.MaxBackups=5
kubectl get pods
minikube ssh cat /var/log/apiserver/audit.log

Anything else do we need to know:
I created pods that access the API service accounts to generate logs as well, with no success.

[r2d4]

Can you ensure that these config options are actually getting set? There should be a log entry in minikube logs that shows that the option was enabled or if it couldn't find the option.

It might be helpful to post the output of minikube logs here
Is RBAC itself working?

[jcantosz]

Thanks for the response!

It looks like it could not find the option

Jun 19 19:59:17 minikube localkube[3723]: I0619 19:59:17.571140    3723 localkube.go:119] Setting Audit.LogOptions.Path to /var/log/apiserver/audit.log on apiserver.
Jun 19 19:59:17 minikube localkube[3723]: W0619 19:59:17.571217    3723 localkube.go:121] Unable to set Audit.LogOptions.Path to /var/log/apiserver/audit.log. Error: Unable to find field by name: LogOptions

It looks like LogOptions should be a valid field. https://godoc.org/k8s.io/apiserver/pkg/server/options#AuditOptions

I do see a difference in the Audit's type in the struct: genericoptions vs kubeoptions of the Authorization. I don't know enough to guess if this is significant.
https://godoc.org/k8s.io/kubernetes/cmd/kube-apiserver/app/options#APIServer

Logs: minikubelogs.txt

[r2d4]

The tricky part here is that these structs have changed between 1.6 and 1.7. The docs are at HEAD, which is 1.7.0-beta, but minikube is still running kubernetes 1.6.4. You'll need to set the ones for the right version of kubernetes.

We might be able to log more helpful error messages here or bubble them up. Without inspecting the actual code, I'm not sure if theres a good way to discover the options on these structs for past versions of kubernetes.

[jcantosz]

I never would have figured that out, thanks! I will see if I can dig up the correct doc version.

A more descriptive error message would be useful. If I discover an easy way to link back to a version of docs, I'll update this thread.

[r2d4]

The real source of truth for minikube will always be the vendored in code. You can see it the struct that we reflect on for the apiserver here

https://github.com/kubernetes/minikube/blob/master/vendor/k8s.io/kubernetes/cmd/kube-apiserver/app/options/options.go#L43-L66

And specifically for the audit options here

https://github.com/kubernetes/minikube/blob/master/vendor/k8s.io/kubernetes/cmd/kube-apiserver/app/options/options.go#L43-L66

So it looks like apiserver.Audit.Path should work, without the LogOptions

[jcantosz]

Thanks @r2d4! That resolved this for me. I will keep that vendoring tip in mind, that is much easier than looking through the tagged release on the kubernetes/kubernetes project

For posterity (until the next version):

minikube start --extra-config=apiserver.Authorization.Mode=RBAC --extra-config=apiserver.Audit.Path=/var/log/apiserver/audit.log --extra-config=apiserver.Audit.MaxAge=30 --extra-config=apiserver.Audit.MaxSize=100 --extra-config=apiserver.Audit.MaxBackups=5

$ minikube logs | grep Audit
Jun 19 22:11:04 minikube localkube[3457]: I0619 22:11:04.432841    3457 localkube.go:119] Setting Audit.Path to /var/log/apiserver/audit.log on apiserver.
Jun 19 22:11:04 minikube localkube[3457]: I0619 22:11:04.432846    3457 localkube.go:119] Setting Audit.MaxAge to 30 on apiserver.
Jun 19 22:11:04 minikube localkube[3457]: I0619 22:11:04.432850    3457 localkube.go:119] Setting Audit.MaxSize to 100 on apiserver.
Jun 19 22:11:04 minikube localkube[3457]: I0619 22:11:04.432853    3457 localkube.go:119] Setting Audit.MaxBackups to 5 on apiserver.
Jun 19 22:11:04 minikube localkube[3457]: I0619 22:11:04.432860    3457 localkube.go:119] Setting Audit.Path to /var/log/apiserver/audit.log on apiserver.
Jun 19 22:11:04 minikube localkube[3457]: I0619 22:11:04.432863    3457 localkube.go:119] Setting Audit.MaxAge to 30 on apiserver.
Jun 19 22:11:04 minikube localkube[3457]: I0619 22:11:04.432867    3457 localkube.go:119] Setting Audit.MaxSize to 100 on apiserver.
Jun 19 22:11:04 minikube localkube[3457]: I0619 22:11:04.432871    3457 localkube.go:119] Setting Audit.MaxBackups to 5 on apiserver.

[wallrj]

With some hints from @r2d4 I've figured out that you can do this:

cat <<EOF > ~/.minikube/addons/audit-policy.yaml
# Log all requests at the Metadata level.
apiVersion: audit.k8s.io/v1beta1
kind: Policy
rules:
- level: Metadata
EOF

minikube start \
         --extra-config=apiserver.Authorization.Mode=RBAC \
         --extra-config=apiserver.Audit.LogOptions.Path=/hosthome/$USER/.minikube/logs/audit.log \
         --extra-config=apiserver.Audit.PolicyFile=/etc/kubernetes/addons/audit-policy.yaml \

Which puts the JSON logs direct on your laptop!

tail -f  ~/.minikube/logs/audit.log  | jq '.| select(.sourceIPs | contains(["127.0.0.1"]) | not)'

[jstrachan]

the following worked for me on a mac with xhyve:

cat <<EOF > ~/.minikube/addons/audit-policy.yaml
# Log all requests at the Metadata level.
apiVersion: audit.k8s.io/v1beta1
kind: Policy
rules:
- level: Metadata
EOF

minikube start --vm-driver hyperkit --cpus=5 --disk-size=50g --memory=7000 \
    --feature-gates=AdvancedAudit=true \
    --extra-config=apiserver.Audit.LogOptions.Path=/var/log/apiserver/audit.log \
    --extra-config=apiserver.Audit.PolicyFile=/etc/kubernetes/addons/audit-policy.yaml

[cscetbon]

@r2d4 @wallrj when I use the option --extra-config=apiserver.Audit.PolicyFile=/etc/kubernetes/addons/audit-policy.yaml I get

/var/log/containers/kube-apiserver-minikube_kube-system_kube-apiserver-.... {"log":"unknown flag: --Audit.PolicyFile\n","stream":"stderr","time":"2018-06-17T22:02:27.331807743Z"}

I'm using qinikube with kubernetes v1.9.3. I can see in the code that it's supposed to support this option but something goes wrong ...

Here is the command I used to get this behavior

minikube start --kubernetes-version v1.9.3 --feature-gates=AdvancedAuditing=true \ 
      --extra-config=apiserver.Audit.PolicyFile=/etc/kubernetes/addons/audit-policy.yaml