Filter sensitive data from the logs

[simianhacker]

We should make an effort to filter sensitive data from the logs. An good example would be to censor req.headers.authorization which would change this log entry:

{
    "@timestamp": "2015-09-25T16:19:02+00:00",
    "message": "GET /elasticsearch 200 5ms - 9.0B",
    "method": "get",
    "pid": 85866,
    "req": {
        "headers": {
            "accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8",
            "accept-encoding": "gzip, deflate, sdch",
            "accept-language": "en-US,en;q=0.8",
            "authorization": "Basic dGVzdDp0ZXN0",
            "connection": "keep-alive",
            "host": "localhost:5601",
            "upgrade-insecure-requests": "1",
            "user-agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.99 Safari/537.36"
        },
        "method": "get",
        "remoteAddress": "127.0.0.1",
        "url": "/elasticsearch",
        "userAgent": "127.0.0.1"
    },
    "res": {
        "contentLength": 9,
        "responseTime": 5,
        "statusCode": 200
    },
    "statusCode": 200,
    "tags": [],
    "type": "response"
}

to this

{
    "@timestamp": "2015-09-25T16:19:02+00:00",
    "message": "GET /elasticsearch 200 5ms - 9.0B",
    "method": "get",
    "pid": 85866,
    "req": {
        "headers": {
            "accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8",
            "accept-encoding": "gzip, deflate, sdch",
            "accept-language": "en-US,en;q=0.8",
            "authorization": "Basic XXXXXXXXXXXX",
            "connection": "keep-alive",
            "host": "localhost:5601",
            "upgrade-insecure-requests": "1",
            "user-agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.99 Safari/537.36"
        },
        "method": "get",
        "remoteAddress": "127.0.0.1",
        "url": "/elasticsearch",
        "userAgent": "127.0.0.1"
    },
    "res": {
        "contentLength": 9,
        "responseTime": 5,
        "statusCode": 200
    },
    "statusCode": 200,
    "tags": [],
    "type": "response"
}

[GlenRSmith]

Would it be possible for the filtered behavior to be the default, and require configuration/option application?

[spalger]

@GlenRSmith The defaults implemented in #5038 automatically remove any authorization values in log messages. Not sure if that answers your question.

[jcollie]

These patches never made it into 4.2.0 so the logging of sensitive data is still a problem.

[simianhacker]

It will go into 4.3 and possibly 4.2.1