Unable to parse traefik json log

[priyankub]

Hi,

I keep getting log time date format error with json access logs from traefik. here is what I am using as post arguments in docker:

--log-format='{"ClientAddr":"%h:%^","ClientHost":"%^","ClientPort":"%^","ClientUsername":"%^","DownstreamContentSize":"%b","DownstreamStatus":"%^","Duration":"%n","OriginContentSize":"%^","OriginDuration":"%^","OriginStatus":"%s","Overhead":"%^","RequestAddr":"%v","RequestContentSize":"%^","RequestCount":"%^","RequestHost":"%^","RequestMethod":"%m","RequestPath":"%U","RequestPort":"%^","RequestProtocol":"%H","RequestScheme":"%^","RetryAttempts":"%^","RouterName":"%^","ServiceAddr":"%^","ServiceName":"%^","ServiceURL":"%^","SpanId":"%^","StartLocal":"%^","StartUTC":"%dT%t.%^","TLSCipher":"%^","TLSVersion":"%^","TraceId":"%^","entryPointName":"%^","level":"%^","msg":"%^","time":"%^"}'
--date-format='%Y-%m-%d'
--time-format='%H:%M:%S'

this is a sample of traefik log:

{"ClientAddr":"192.168.11.2:40126","ClientHost":"192.168.11.2","ClientPort":"40126","ClientUsername":"-","DownstreamContentSize":2099,"DownstreamStatus":200,"Duration":1811271,"OriginContentSize":2099,"OriginDuration":1564210,"OriginStatus":200,"Overhead":247061,"RequestAddr":"nzbget.local.example.com","RequestContentSize":154,"RequestCount":103109,"RequestHost":"nzbget.local.example.com","RequestMethod":"POST","RequestPath":"/xmlrpc","RequestPort":"-","RequestProtocol":"HTTP/1.1","RequestScheme":"https","RetryAttempts":0,"RouterName":"https-nzbget-local@docker","ServiceAddr":"172.20.0.47:6789","ServiceName":"nzbget-service@docker","ServiceURL":"http://172.20.0.47:6789","SpanId":"0000000000000000","StartLocal":"2024-11-14T12:49:22.640540732-05:00","StartUTC":"2024-11-14T17:49:22.640540732Z","TLSCipher":"TLS_AES_128_GCM_SHA256","TLSVersion":"1.3","TraceId":"00000000000000000000000000000000","entryPointName":"https","level":"info","msg":"","time":"2024-11-14T12:49:22-05:00"}
{"ClientAddr":"192.168.1.206:47698","ClientHost":"192.168.1.206","ClientPort":"47698","ClientUsername":"-","DownstreamContentSize":491,"DownstreamStatus":200,"Duration":63791537,"OriginContentSize":491,"OriginDuration":63604875,"OriginStatus":200,"Overhead":186662,"RequestAddr":"cloud.example.com","RequestContentSize":0,"RequestCount":103110,"RequestHost":"cloud.example.com","RequestMethod":"GET","RequestPath":"/ocs/v2.php/cloud/user?format=json","RequestPort":"-","RequestProtocol":"HTTP/2.0","RequestScheme":"https","RetryAttempts":0,"RouterName":"https-nextcloud-secure@docker","ServiceAddr":"172.20.0.18:443","ServiceName":"nextcloud@docker","ServiceURL":"https://172.20.0.18:443","SpanId":"0000000000000000","StartLocal":"2024-11-14T12:49:24.642311238-05:00","StartUTC":"2024-11-14T17:49:24.642311238Z","TLSCipher":"TLS_AES_128_GCM_SHA256","TLSVersion":"1.3","TraceId":"00000000000000000000000000000000","entryPointName":"https","level":"info","msg":"","time":"2024-11-14T12:49:24-05:00"}

[allinurl]

This should get you started:

$ goaccess log --log-format='{ "ClientAddr": "%h:%^", "OriginStatus": "%s", "RequestAddr": "%v", "RequestMethod": "%m", "RequestPath": "%U", "RequestProtocol": "%H", "time": "%dT%t-%^" }' --date-format='%Y-%m-%d' --time-format=%T --date-spec=min

[mrpops2ko]

This should get you started:

$ goaccess log --log-format='{ "ClientAddr": "%h:%^", "OriginStatus": "%s", "RequestAddr": "%v", "RequestMethod": "%m", "RequestPath": "%U", "RequestProtocol": "%H", "time": "%dT%t-%^" }' --date-format='%Y-%m-%d' --time-format=%T --date-spec=min

this didn't work for me but i've found this does

cat access.log | docker run --rm -i -v ./report.html:/report.html -e LANG=$LANG allinurl/goaccess -a -o report.html --log-format='{ "ClientAddr": "%h:%^", "OriginStatus": "%s", "RequestAddr": "%v", "RequestMethod": "%m", "RequestPath": "%U", "RequestProtocol": "%H", "time": "%d" }' --date-format='%Y-%m-%dT%H:%M:%SZ' --time-format='%H:%M:%S' --date-spec=min -

i'll try and improve it if possible with more useful fields but at least this one is working for now

edit: since i use cloudflare, i've modified it to use the real client ip instead

cat access.log | docker run --rm -i -v ./report.html:/report.html -e LANG=$LANG allinurl/goaccess -a -o report.html \
--log-format='{ "ClientHost": "%h", "OriginStatus": "%s", "RequestHost": "%v", "RequestMethod": "%m", "RequestPath": "%U", "RequestProtocol": "%H", "time": "%d", "request_User-Agent": "%u" }' \
--date-format='%Y-%m-%dT%H:%M:%SZ' \
--time-format='%H:%M:%S' \
--date-spec=min \
-

edit 2:

cat access.log | docker run --rm -i -v ./report.html:/report.html -e LANG=$LANG allinurl/goaccess -a -o report.html \
--log-format='{ "ClientHost": "%h", "DownstreamStatus": "%s", "DownstreamContentSize": "%b", "RequestHost": "%v", "RequestMethod": "%m", "RequestPath": "%U", "RequestProtocol": "%H", "Duration": "%n", "TLSVersion": "%K", "TLSCipher": "%k", "ClientUsername": "%e", "time": "%d", "request_User-Agent": "%u" }' \
--date-format='%Y-%m-%dT%H:%M:%SZ' --time-format='%H:%M:%S' --date-spec=min -

in my traefik.yml file i'm doing

accessLog:
  filePath: /var/log/traefik/access.log
  format: json
  filters:
    statusCodes:
      - "200-299" # log successful http requests
      - "400-599" # log failed http requests
  # collect logs as in-memory buffer before writing into log file
  bufferingSize: 10000
  fields:
    headers:
      defaultMode: drop # drop all headers per default
      names:
          User-Agent: keep # log user agent strings

i'll keep trying some more to improve it but if anybody else wants to, the documentation is here for which ones could be potential matches and the traefik potential fields is here