-
Notifications
You must be signed in to change notification settings - Fork 15
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
how to create example.log file used in the performance measurements #132
Comments
Hi, unfortunately I cannot share this log file because it contains private and confidential information. But I can gather some statistics about it and share them. In the performance section I only shared the total file size and the number of log lines in it, the length of the lines varies from 109 to 763670 bytes. I think this is the most important information describing the source, and most real log files of similar size would show similar performance. But if you need additional statistics, I think I can easily collect them. For example, I've just collected the distribution of line lengths and the number of top-level keys in lines. Here is the distribution of the number of keys:
The data was collected using the following command: jq 'length' example.log | sort -rn | uniq -c >example.nkeys There are at least 5 keys in each line that are: "level", "ts", "logger", "msg", "caller". Here is the distribution of line lengths: example.len.zip. awk '{print length}' example.log | sort -rn | uniq -c >example.len Feel free to ask me to collect any additional statistics. |
I found another open dataset, transformed it slightly and made the measurements. Source fileWeb robot detection - Server logs Transformation commandpv -c -N input <public_v2.json | jq -c 'to_entries[] | {"request-id": .key} + .value + {"response": (.value.response | tonumber), "bytes": (.value.bytes | tonumber), level: "info"}' | (pv -c -N output >web-robot.log) Notes
MeasurementsRaw details❯ hl --version
hl 0.29.5
❯ time hl web-robot.log -c -o /dev/null
hl web-robot.log -c -o /dev/null 12.06s user 0.72s system 930% cpu 1.374 total
# ---
❯ humanlog --version
humanlog version 0.7.6+deb0543
❯ time humanlog <web-robot.log --color always >/dev/null
humanlog> reading stdin...
humanlog --color always < web-robot.log > /dev/null 92.02s user 4.38s system 108% cpu 1:28.76 total
# ---
❯ fblog --version
fblog 4.10.0
❯ time fblog web-robot.log >/dev/null
fblog web-robot.log > /dev/null 25.85s user 1.65s system 98% cpu 28.001 total
❯ time fblog -d web-robot.log >/dev/null
fblog -d web-robot.log > /dev/null 131.54s user 14.14s system 99% cpu 2:26.33 total
# ---
❯ wc -cl web-robot.log
4091155 3312769299 web-robot.log
❯ sysctl -n machdep.cpu.brand_string
Apple M1 Max |
The performance section mentions a log file used for testing, can you provide a link to that or what you used to generate it?
I'd like to try it out with lnav to see how it performs and if there is anything I can improve.
Thanks
The text was updated successfully, but these errors were encountered: