scanning caught an error: bufio.Scanner: token too long

[patroza]

Hi,

I have a long json payload in logfmt value, and I run into token too long.
Could you please increase the buffer, make it configurable, and perhaps at least catch the particular error and instead of crashing, mark the field as error and continue?

I found some background on the error: https://stackoverflow.com/a/37455465

[aybabtme]

@patroza do you mind me asking how long is the string you're dealing with? If you have a reproducer case that'd be awesome!

[patroza]

@aybabtme the JSON payload itself was 300059 bytes, but there are more log values.

I noticed that the latest versions have truncation enabled by default,
interestingly enough, when humanlog is installed with brew on mac, it registers as version devel,
and I have an older version than when you install.sh.

This was super confusing as a co-worker installed it on linux via install.sh and I had it via brew, and he was seeing truncation, and I wasn't :)

atm as workaround I use

if which humanlog >/dev/null; then
    # For some reason --skip-unchanged is enabled by default and can't figure out how to disable!
    # https://github.com/humanlogio/humanlog/issues/40
    tee dev.log | (humanlog --keep requestId --keep requestName --truncate-length 16384 || (echo 'humanlog crashed!' && cat))
else
    echo "!! It's recommended to install humanlog, to make these logs more readable. !!"
    # while read x ; do echo $x ; done
    tee dev.log
fi

[aybabtme]

@patroza which tap did you use for installing humanlog? If you still use aybabtme/homebrew-tap, it has moved to humanlogio/homebrew-tap :)

You can find that out by running:

$ brew info humanlog

The output should look like:

==> humanlogio/tap/humanlog: stable 0.7.3


/usr/local/Cellar/humanlog/0.7.3 (5 files, 9.2MB) *
  Built from source on 2023-01-22 at 11:55:27
From: https://github.com/humanlogio/homebrew-tap/blob/HEAD/humanlog.rb

If you're seeing aybabtme/tap/humanlog instead of humanlogio/tap/humanlog then you're using the old tap location. To fix that, I believe you have two choices.

First, I think simply updating brew, adding the new tap and then reinstalling humanlog should suffice.

$ brew update
$ brew tap humanlogio/homebrew-tap
$ brew reinstall humanlog

If this still installs from the wrong tap, or if it shows an error message, remove the old tap manually:

$ brew untap aybabtme/homebrew-tap
$ brew tap humanlogio/homebrew-tap
$ brew install humanlog # or `brew reinstall humanlog`

At this time the latest version of humanlog should print:

$ humanlog --version
humanlog version 0.7.3+a7354d3