Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

advanced audit beta features #5300

Merged
merged 1 commit into from
Sep 6, 2017
Merged

advanced audit beta features #5300

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
189 changes: 124 additions & 65 deletions docs/tasks/debug-application-cluster/audit.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ answer the following questions:
- from where was it initiated?
- to where was it going?

## Audit logs
## Legacy Audit

Kubernetes audit is part of [Kube-apiserver][kube-apiserver] logging all requests
processed by the server. Each audit log entry contains two lines:
Expand All @@ -37,8 +37,13 @@ Example output for `admin` user listing pods in the `default` namespace:
2017-03-21T03:57:09.108403639-04:00 AUDIT: id="c939d2a7-1c37-4ef1-b2f7-4ba9b1e43b53" response="200"
```

Note that this format changes when enabling the `AdvancedAuditing` feature discussed
later in this document.
Note that Kubernetes 1.8 has switched to use the advanced structured audit log by default.
To fallback to this legacy audit, disable the advanced auditing feature
using the `AdvancedAuditing` feature gate on the [kube-apiserver][kube-apiserver]:

```
--feature-gates=AdvancedAuditing=false
```

### Configuration

Expand All @@ -62,18 +67,12 @@ and `audit-log-maxage` options.
## Advanced audit

Kubernetes 1.7 expands auditing with experimental functionality such as event
filtering and a webhook for integration with external systems. The rest of this
document covers features that are __alpha__ and may change in backward incompatible
ways.
filtering and a webhook for integration with external systems. Kubernetes 1.8
upgrades the advanced audit feature to beta, and some backward incompatible changes
have been committed.

Enable the alpha auditing features using the `AdvancedAuditing` feature gate on
the [kube-apiserver][kube-apiserver]:

```
--feature-gates=AdvancedAuditing=true
```

`AdvancedAuditing`is customizable in two ways. Policy, which determines what's recorded,
`AdvancedAuditing` is customizable in two ways. Policy, which determines what's recorded,
and backends, which persist records. Backend implementations include logs files and
webhooks.

Expand Down Expand Up @@ -110,10 +109,17 @@ The policy file holds rules that determine the level of an event. Known audit le
When an event is processed, it's compared against the list of rules in order.
The first matching rule sets the audit level of the event. The audit policy is
defined by the [`audit.k8s.io` API group][audit-api].
Some new fields are supported in beta version, like `resourceNames` and `omitStages`.

In Kubernetes 1.8 `kind` and `apiVersion` along with `rules` __must__ be provided in
the audit policy file. A policy file with 0 rules, or a policy file that doesn't provide
a valid `apiVersion` and `kind` value will be treated as illgal.

An example audit policy file:
Some example audit policy files:

```yaml
apiVersion: audit.k8s.io/v1beta1 #this is required in Kubernetes 1.8
kind: Policy
rules:
# Don't log watch requests by the "system:kube-proxy" on endpoints or services
- level: None
Expand Down Expand Up @@ -155,10 +161,46 @@ rules:
- level: Metadata
```

The next audit policy file shows new features introduced in Kubernetes 1.8:

```yaml
apiVersion: audit.k8s.io/v1beta1
kind: Policy
rules:
# Log pod changes at Request level
- level: Request
resources:
- group: ""
# Resource "pods" no longer matches requests to any subresource of pods,
# This behavior is consistent with the RBAC policy.
resources: ["pods"]
# Log "pods/log", "pods/status" at Metadata level
- level: Metadata
resources:
- group: ""
resources: ["pods/log", "pods/status"]

# Don't log requests to a configmap called "controller-leader"
- level: None
resources:
- group: ""
resources: ["configmaps"]
resourceNames: ["controller-leader"]

# A catch-all rule to log all other requests at the Metadata level.
# For this rule we use "omitStages" to omit events at "ReqeustReceived" stage.
# Events in this stage will not be sent to backend.
- level: Metadata
omitStages:
- "RequestReceived"
```

You can use a minimal audit policy file to log all requests at the `Metadata` level:

```yaml
# Log all requests at the Metadata level.
apiVersion: audit.k8s.io/v1beta1
kind: Policy
rules:
- level: Metadata
```
Expand All @@ -181,33 +223,36 @@ API can be found [here][audit-api] with more details about the exact fields capt
#### Log backend

The behavior of the `--audit-log-path` flag changes when enabling the `AdvancedAuditing`
feature flag. This includes the cleanups discussed above, such as changes to the `method`
values and the introduction of a "stage" for each event. As before, the `id` field of
the log indicates which events were generated from the same request. With default legacy
format, events are formatted as follows:
feature flag. All generated events defined by `--audit-policy-file` are recorded in structured
json format:

```
2017-06-15T21:50:50.259470834Z AUDIT: id="591e9fde-6a98-46f6-b7bc-ec8ef575696d" stage="RequestReceived" ip="10.2.1.3" method="update" user="system:serviceaccount:kube-system:default" groups="\"system:serviceaccounts\",\"system:serviceaccounts:kube-system\",\"system:authenticated\"" as="<self>" asgroups="<lookup>" namespace="kube-system" uri="/api/v1/namespaces/kube-system/endpoints/kube-controller-manager" response="<deferred>"
2017-06-15T21:50:50.259470834Z AUDIT: id="591e9fde-6a98-46f6-b7bc-ec8ef575696d" stage="ResponseComplete" ip="10.2.1.3" method="update" user="system:serviceaccount:kube-system:default" groups="\"system:serviceaccounts\",\"system:serviceaccounts:kube-system\",\"system:authenticated\"" as="<self>" asgroups="<lookup>" namespace="kube-system" uri="/api/v1/namespaces/kube-system/endpoints/kube-controller-manager" response="200"
{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":null},"level":"Metadata","timestamp":"2017-09-05T10:04:55Z","auditID":"77e58433-d345-40ac-b2d8-9866bd355cea","stage":"RequestReceived","requestURI":"/apis/rbac.authorization.k8s.io/v1/namespaces/default/roles","verb":"list","user":{"username":"kubecfg","groups":["system:masters","system:authenticated"]},"sourceIPs":["172.16.116.128"],"objectRef":{"resource":"roles","namespace":"default","apiGroup":"rbac.authorization.k8s.io","apiVersion":"v1"}}
{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":null},"level":"Metadata","timestamp":"2017-09-05T10:04:55Z","auditID":"77e58433-d345-40ac-b2d8-9866bd355cea","stage":"ResponseComplete","requestURI":"/apis/rbac.authorization.k8s.io/v1/namespaces/default/roles","verb":"list","user":{"username":"kubecfg","groups":["system:masters","system:authenticated"]},"sourceIPs":["172.16.116.128"],"objectRef":{"resource":"roles","namespace":"default","apiGroup":"rbac.authorization.k8s.io","apiVersion":"v1"},"responseStatus":{"metadata":{},"code":200}}
```

Logged events omit the request and response bodies. The `Request` and
`RequestResponse` levels are equivalent to `Metadata` for legacy format.
In alpha version, objectRef.apiVersion holds both the api group and version.
In beta version these were break out into objectRef.apiGroup and objectRef.apiVersion.

Since Kubernetes 1.8, structed json fromat is supported for log backend.
Use the following option to switch log to json format:
Starting from Kubernetes 1.8, structured json format is used for log backend by default.
Use the following option to switch log to legacy format:

```
--audit-log-format=json
--audit-log-format=legacy
```

With json format, events are formatted as follows:
With legacy format, events are formatted as follows:

```
{"kind":"Event","apiVersion":"audit.k8s.io/v1alpha1","metadata":{"creationTimestamp":null},"level":"Metadata","timestamp":"2017-07-12T11:02:43Z","auditID":"2e79f0c9-a941-45ae-a9ce-663a1b19ff14","stage":"RequestReceived","requestURI":"/api/v1/namespaces/default/pods","verb":"list","user":{"username":"kubecfg","groups":["system:masters","system:authenticated"]},"sourceIPs":["172.16.116.128"],"objectRef":{"resource":"pods","namespace":"default","apiVersion":"/v1"}}
{"kind":"Event","apiVersion":"audit.k8s.io/v1alpha1","metadata":{"creationTimestamp":null},"level":"Metadata","timestamp":"2017-07-12T11:02:43Z","auditID":"2e79f0c9-a941-45ae-a9ce-663a1b19ff14","stage":"ResponseComplete","requestURI":"/api/v1/namespaces/default/pods","verb":"list","user":{"username":"kubecfg","groups":["system:masters","system:authenticated"]},"sourceIPs":["172.16.116.128"],"objectRef":{"resource":"pods","namespace":"default","apiVersion":"/v1"},"responseStatus":{"metadata":{},"code":200}}
2017-09-05T06:08:19.885328047-04:00 AUDIT: id="c28a95ad-f9dd-47e1-a617-b6dc152db95f" stage="RequestReceived" ip="172.16.116.128" method="list" user="kubecfg" groups="\"system:masters\",\"system:authenticated\"" as="<self>" asgroups="<lookup>" namespace="default" uri="/apis/rbac.authorization.k8s.io/v1/namespaces/default/roles" response="<deferred>"
2017-09-05T06:08:19.885328047-04:00 AUDIT: id="c28a95ad-f9dd-47e1-a617-b6dc152db95f" stage="ResponseComplete" ip="172.16.116.128" method="list" user="kubecfg" groups="\"system:masters\",\"system:authenticated\"" as="<self>" asgroups="<lookup>" namespace="default" uri="/apis/rbac.authorization.k8s.io/v1/namespaces/default/roles" response="200"
```

Logged events omit the request and response bodies. The `Request` and
`RequestResponse` levels are equivalent to `Metadata` for legacy format. This legacy format
of advanced audit is different from the [Legacy Audit](# Legacy Audit) discussed above, such
as changes to the method values and the introduction of a "stage" for each event.

#### Webhook backend

The audit webhook backend can be used to have [kube-apiserver][kube-apiserver]
Expand Down Expand Up @@ -255,44 +300,54 @@ Events are POSTed as a JSON serialized `EventList`. An example payload:

```json
{
"kind": "EventList",
"apiVersion": "audit.k8s.io/v1alpha1",
"items": [
{
"metadata": {
"creationTimestamp": null
},
"level": "Metadata",
"timestamp": "2017-06-15T23:07:40Z",
"auditID": "4faf711a-9094-400f-a876-d9188ceda548",
"stage": "ResponseComplete",
"requestURI": "/apis/rbac.authorization.k8s.io/v1beta1/namespaces/kube-public/rolebindings/system:controller:bootstrap-signer",
"verb": "get",
"user": {
"username": "system:apiserver",
"uid": "97a62906-e4d7-4048-8eda-4f0fb6ff8f1e",
"groups": [
"system:masters"
]
},
"sourceIPs": [
"127.0.0.1"
],
"objectRef": {
"resource": "rolebindings",
"namespace": "kube-public",
"name": "system:controller:bootstrap-signer",
"apiVersion": "rbac.authorization.k8s.io/v1beta1"
},
"responseStatus": {
"metadata": {},
"code": 200
}
}
]
"apiVersion": "audit.k8s.io/v1beta1",
"items": [
{
"auditID": "24f30caf-d7d4-45d5-b7bd-e7af300d7886",
"level": "Metadata",
"metadata": {
"creationTimestamp": null
},
"objectRef": {
"apiGroup": "rbac.authorization.k8s.io",
"apiVersion": "v1",
"name": "jane",
"namespace": "default",
"resource": "roles"
},
"requestURI": "/apis/rbac.authorization.k8s.io/v1/namespaces/default/roles/jane",
"responseStatus": {
"code": 200,
"metadata": {}
},
"sourceIPs": [
"172.16.116.128"
],
"stage": "ResponseComplete",
"timestamp": "2017-09-05T10:20:24Z",
"user": {
"groups": [
"system:masters",
"system:authenticated"
],
"username": "kubecfg"
},
"verb": "get"
}
],
"kind": "EventList",
"metadata": {}
}
```

### Audit-Id

Audit-Id is a unique ID for each http request to kube-apiserver. The ID of events will be the
same if they were generated from the same request. Starting from Kubernetes 1.8, if an audit
event is generated for the request, kube-apiserver will respond with an Audit-Id in the HTTP header.
Note that for some special requests like `kubectl exec`, `kubectl attach`, kube-apiserver works
like a proxy, no Audit-Id will be returned even if audit events are recorded.

### Log Collector Examples

#### Use fluentd to collect and distribute audit events from log file
Expand Down Expand Up @@ -422,9 +477,13 @@ Note that in addition to file output plugin, logstash has a variety of outputs t
let users route data where they want. For example, users can emit audit events to elasticsearch
plugin which supports full-text search and analytics.

[audit-api]: https://github.com/kubernetes/kubernetes/blob/v1.7.0-rc.1/staging/src/k8s.io/apiserver/pkg/apis/audit/v1alpha1/types.go
<!---
herf of audit-api is not right, because https://github.com/kubernetes/kubernetes/pull/49280 is not included in this tag.
Maybe it will show up in another tag, then I will update it.
-->
[audit-api]: https://github.com/kubernetes/kubernetes/blob/v1.8.0-beta.0/staging/src/k8s.io/apiserver/pkg/apis/audit/v1beta1/types.go
[kube-apiserver]: /docs/admin/kube-apiserver
[gce-audit-profile]: https://github.com/kubernetes/kubernetes/blob/v1.7.0/cluster/gce/gci/configure-helper.sh#L490
[gce-audit-profile]: https://github.com/kubernetes/kubernetes/blob/v1.8.0-beta.0/cluster/gce/gci/configure-helper.sh#L532
[fluentd]: http://www.fluentd.org/
[fluentd_install_doc]: http://docs.fluentd.org/v0.12/articles/quickstart#step1-installing-fluentd
[logstash]: https://www.elastic.co/products/logstash
Expand Down