hl

High-performance log viewer and processor that transforms logs in JSON and logfmt formats into a human-readable output. Built with efficiency in mind, it enables quick parsing and analysis of large log files with minimal overhead.

Features overview

Automatic Pager Integration: Automatically integrates with a pager for enhanced convenience, defaulting to less if available, but fully supporting any compatible pager.
Log Streaming Mode: Enable log streaming with the -P flag, which disables the pager.
Field-Based Filtering: Filter log records by key/value pairs using the -f option, with support for hierarchical keys.
Level Filtering: Easily filter logs by level with the -l option.
Timestamp Range Filtering: Filter logs by timestamp range using the --since and --until options with intuitive formats:
- RFC-3339 timestamp format.
- Current configured timestamp output format (via the -t option or environment variable).
- User-friendly shortcuts like today, yesterday, friday, or relative offsets such as -3h or -14d.
Field Visibility Control: Quickly hide or reveal specific fields using the -h option.
Empty Field Hiding: Automatically hide empty fields with the -e flag.
High-Speed Message Sorting: Achieve lightning-fast message sorting with automatic indexing via the -s flag.
- Performs the initial scan at approximately 2 GiB/s, enabling rapid filtering by timestamp and level without re-scanning.
- Efficiently handles hundreds of local files totaling hundreds of gigabytes.
- Reindexes large, growing files at speeds up to roughly 10 GiB/s by skipping unmodified blocks.
Live Follow Mode: Use the -F flag for live, timestamp-sorted message updates across multiple sources, with a preview of recent messages via the --tail option.
Complex Query Support: Construct custom queries with logical conditions (AND/OR) and additional advanced filtering options.
Non-JSON Prefix Handling: Process logs with non-JSON prefixes using the --allow-prefix flag.
Timezone Flexibility: Displays timestamps in UTC by default while allowing effortless timezone switching with the -Z option or local timezone adjustments using the -L flag.
Customizability and Themes: Fully customizable through configuration files and environment variables, with support for easy theme switching and custom themes.

Performance comparison chart

Performance comparison with humanlog, hlogf and fblog on a 2.3 GiB log file

See performance section for more details.

Installation options

macOS

Install using homebrew on macOS
```
brew install hl
```

Other options

Download and extract using curl and tar on macOS

curl -sSfL https://github.com/pamburus/hl/releases/latest/download/hl-macos.tar.gz | tar xz

Install using cargo

cargo install --locked --git https://github.com/pamburus/hl.git

Download latest release from download page

Linux

Download and extract using curl and tar on Linux (x86_64)

curl -sSfL https://github.com/pamburus/hl/releases/latest/download/hl-linux-x86_64-musl.tar.gz | tar xz

Install AUR package on Arch Linux
```
yay -S hl-log-viewer-bin
```

Other options

Download and extract using curl and tar on Linux (arm64/aarch64)

curl -sSfL https://github.com/pamburus/hl/releases/latest/download/hl-linux-arm64-musl.tar.gz | tar xz

Install using cargo

cargo install --locked --git https://github.com/pamburus/hl.git

Windows

Install from Scoop

scoop bucket add pamburus https://github.com/pamburus/scoop-bucket.git
scoop install hl

Tip

It is recommended to use Windows Terminal for better experience.

Tip

To make mouse scrolling work in the less pager, set the LESS environment variable to -R --mouse.

Important

Currently, hl does not provide a built-in pager and relies on external pagers such as less. However, the build for Windows referenced on the original download page and distributed in the WinGet package manager does not work as expected. The authors state that they have not tested or verified this build and suggest that you use it at your own risk. Unfortunately, this build breaks some ANSI escape sequences and does not work properly with hl and many other programs that use ANSI escape sequences for colors and styles. It is recommended to install less from the Scoop or Chocolatey package manager. If you are using Scoop and install hl by running scoop install hl, it already installs less automatically as a dependency. Just make sure you do not have any other conflicting installations by running where less in cmd or Get-Command less in powershell.

Download latest release from download page

Other options

Install using cargo

cargo install --locked --git https://github.com/pamburus/hl.git

NixOS

Run using nix

nix run github:pamburus/hl

or install with nix profile:

nix profile install github:pamburus/hl

Install the package using nix-flakes

Example how to update nix configuration

{
    inputs = {
        nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
        hl.url = "github:pamburus/hl";
    };
    outputs = {nixpkgs, hl, ...}:
    let
        system = "x86_64-linux";
    in
    {
        # this is just an example!
        nixosConfigurations.yourHost = nixpkgs.lib.nixosSystem {
            inherit system;
            modules = [
                ({...}: {
                  environment.systemPackages = [
                    hl.packages.${system}
                  ];
                })
            ];
        };
    };
}

Examples

Screenshot

See other screenshots

Features and usage

Concatenation of multiple log files

Concatenate all log files

Command
```
hl *.log
```
Concatenates and displays all *.log files found in the current directory.

Support for compressed (bzip2, gzip, xz, zstd) log files

Concatenate all log files including compressed log files

Command
```
hl $(ls -tr /var/log/example/*.{log,log.gz,log.zst,s})
```
Concatenates and displays all *.log, *.log.gz, *.log.zst and *.s (will detect compression) files found in /var/log/example/.

Automatic usage of pager

Use the default pager with the default parameters

Command
```
hl example.log
```
Automatically opens less pager with the default parameters.
Override options for default pager

Command
```
LESS=-SR hl example.log
```
Opens less pager with disabled line wrapping.
Use custom pager

Command
```
PAGER="most -w" hl example.log
```
Opens most pager with -w option.

Quick filtering by log level

Errors only

Command
```
hl -l e
```
Displays only error log level messages.
Errors and warnings

Command
```
hl -l w
```
Displays only warning and error log level messages.
Errors, warnings and informational

Command
```
hl -l i
```
Displays all log messages except debug level messages.

Using live log streaming

Command
```
tail -f example.log | hl -P
```
Tracks changes in the example.log file and displays them immediately. Flag -P disables automatic using of pager in this case.

Filtering by field values

Command
```
hl example.log --filter component=tsdb
```
Displays only messages where the component field has the value tsdb.
Command
```
hl example.log -f component!=tsdb -f component!=uninteresting
```
Displays only messages where the component field has a value other than tsdb or uninteresting.
Command
```
hl example.log -f provider~=string
```
Displays only messages where the provider field contains the string sub-string.
Command
```
hl example.log -f 'provider!~=string'
```
Displays only messages where the provider field does not contain the string sub-string.

Performing complex queries

Command
```
hl my-service.log --query 'level > info or status-code >= 400 or duration > 0.5'
```
Displays messages that either have a level higher than info (i.e. warning or error) or have a status code field with a numeric value >= 400 or a duration field with a numeric value >= 0.5.
Command
```
hl my-service.log -q '(request in (95c72499d9ec, 9697f7aa134f, bc3451d0ad60)) or (method != GET)'
```
Displays all messages that have the 'request' field with one of these values, or the 'method' field with a value other than 'GET'.
Complete set of supported operators
- Logical operators
  - Logical conjunction - and, &&
  - Logical disjunction - or, ||
  - Logical negation - not, !
- Comparison operators
  - Equal - eq, =
  - Not equal - ne, !=
  - Greater than - gt, >
  - Greater or equal - ge, >=
  - Less than - lt, <
  - Less or equal - le, <=
- String matching operators
  - Sub-string check - (contain, ~=), (not contain, !~=)
  - Wildcard match - (like), (not like)
    - Wildcard characters are: * for zero or more characters and ? for a single character
  - Regular expression match - (match, ~~=), (not match, !~~=)
- Operators with sets
  - Test if value is one of the values in a set - in (v1, v2), not in (v1, v2)
  - Test if value is one of the values in a set loaded from a file - in @filename, not in @filename, assuming that each element is a line in the file, which can be either a simple string or a JSON string
  - Test if value is one of the values in a set loaded stdin - in @-, not in @-
Notes
- Special field names that are reserved for filtering by predefined fields regardless of the actual source field names used to load the corresponding value: level, message, caller and logger.
- To address a source field with one of these names instead of predefined fields, add a period before its name, i.e., .level will perform a match against the "level" source field.
- To address a source field by its exact name, use a JSON-formatted string, i.e. -q '".level" = info'.
- To specify special characters in field values, also use a JSON-formatted string, i.e.
```
hl my-service.log -q 'message contain "Error:\nSomething unexpected happened"'
```

Filtering by time range

Command
```
hl example.log --since 'Jun 19 11:22:33' --until yesterday
```
Displays only messages that occurred after Jun 19 11:22:33 UTC of the current year (or the previous year if the current date is less than Jun 19 11:22:33) and before yesterday midnight.
Command
```
hl example.log --since -3d
```
Displays only messages from the past 72 hours.
Command
```
hl example.log --until '2021-06-01 18:00:00' --local
```
Displays only messages that occurred before 6 PM local time on June 1, 2021, and shows timestamps in local time.

Hiding or revealing selected fields

Command
```
hl example.log --hide provider
```
Hides field provider.

Command

hl example.log --hide '*' --hide '!provider'

Hides all fields except provider.

Command
```
hl example.log -h headers -h body -h '!headers.content-type'
```
Hides fields headers and body but shows a single sub-field content-type inside field headers.

Sorting messages chronologically

Command
```
hl -s *.log
```
Displays log messages from all log files in the current directory sorted in chronological order.

Sorting messages chronologically with following the changes

Command
```
hl --sync-interval-ms 500 -F <(kubectl logs -l app=my-app-1 -f) <(kubectl logs -l app=my-app-2 -f)
```
Runs without a pager in follow mode by merging messages from the outputs of these 2 commands and sorting them chronologically within a custom 500ms interval.
Command
```
hl -F --tail 100 app1.log app2.log app3.log
```
Runs without a pager in follow mode, following the changes in three log files in the current directory and sorting them chronologically at a default interval of 100ms. Preloads 100 lines from the end of each file before filtering.

Configuration files

Configuration files are automatically loaded if found in predefined platform-specific locations.

OS	System-Wide Location	User Profile Location
macOS	/etc/hl/config.{yaml,toml,json}	~/.config/hl/config.{yaml,toml,json}
Linux	/etc/hl/config.{yaml,toml,json}	~/.config/hl/config.{yaml,toml,json}
Windows	%PROGRAMDATA%\hl\config.{yaml,toml,json}	%USERPROFILE%\AppData\Roaming\hl\config.{yaml,toml,json}

The path to the configuration file can be overridden using the HL_CONFIG environment variable or the --config command-line option.

The order in which the configuration files are searched and loaded is as follows:
1. The system-wide location.
2. The user profile location.
3. The location specified by the HL_CONFIG environment variable (unless the --config option is used).
4. The locations specified by the --config option (can be specified multiple times).
If a configuration file is found in multiple locations, the file in each subsequent location overrides only the parameters it contains.

If HL_CONFIG or --config specifies - or an empty string, all default locations and any locations specified by previous --config options are discarded. The search for the configuration file locations starts over.

To disable loading of configuration files and use the built-in defaults, --config - can be used.
All parameters in the configuration file are optional and can be omitted. In this case, default values are used.

Default configuration file

config.yaml

Environment variables

Many parameters that are defined in command line arguments and configuration files can also be specified by environment variables.

Precedence of configuration sources (from lowest priority to highest priority)

Configuration file
Environment variables
Command-line arguments

Environment variables examples

HL_TIME_FORMAT='%y-%m-%d %T.%3N' overrides the time format specified in the configuration file.
HL_TIME_ZONE=Europe/Berlin overrides the time zone specified in the configuration file.
HL_CONCURRENCY=4 overrides the concurrency limit specified in the configuration file.
HL_PAGING=never specifies the default value for the paging option, but it can be overridden by command line arguments.

Themes

Stock themes

themes

Selecting current theme

Using theme value in the configuration file.
Using environment variable, i.e. HL_THEME=classic, overrides the value specified in configuration file.
Using command-line argument, i.e. --theme classic, overrides all other values.

Selecting themes with preview

To select themes with preview fzf tool can be used like this:

hl --list-themes | fzf --preview-window="top,80%" --preview="head -n 100 example.log | hl -c --theme {}"

Custom themes

Custom themes are automatically loaded when found in a predefined platform-specific location.

OS Location

macOS ~/.config/hl/themes/*.{yaml,toml,json}

Linux ~/.config/hl/themes/*.{yaml,toml,json}

Windows %USERPROFILE%\AppData\Roaming\hl\themes*.{yaml,toml,json}
Format description
- Section elements contains styles for predefined elements.
- Section levels contains optional overrides for styles defined in elements sections per logging level, which are [trace, debug, info, warning, error].
- Each element style contains optional background, foreground and modes parameters.
- Example
```
elements:
    <element>:
        foreground: <color>
        background: <color>
        modes: [<mode>, <mode>, ...]
levels:
    <level>:
        <element>:
            foreground: <color>
            background: <color>
            modes: [<mode>, <mode>, ...]
```
- Color format is one of
  - Keyword default specifies default color defined by the terminal.
  - ASCII basic color name, one of
    - black
    - red
    - green
    - yellow
    - blue
    - magenta
    - cyan
    - white
    - bright-black
    - bright-red
    - bright-green
    - bright-yellow
    - bright-blue
    - bright-magenta
    - bright-cyan
    - bright-white
  - 256-color palette code, from 0 to 255.
  - RGB color in hex web color format, i.e. #FFFF00 for bright yellow color.
- Modes is a list of additional styles, each of them is one of
  - bold
  - faint
  - italic
  - underline
  - slow-blink
  - rapid-blink
  - reverse
  - conceal
  - crossed-out

Used terminal color schemes

iTerm2

One Dark Neo
Built-in "Light Background" color scheme

Alacritty

One Dark Neo
- Note: It is recommended to use draw_bold_text_with_bright_colors: true setting
Light
- Note: It is recommended to use draw_bold_text_with_bright_colors: false setting

Complete set of options and flags

JSON and logfmt log converter to human readable representation

Usage: hl [OPTIONS] [FILE]...

Arguments:
  [FILE]...  Files to process

Options:
      --config <FILE>                    Configuration file path [env: HL_CONFIG=]
  -s, --sort                             Sort messages chronologically
  -F, --follow                           Follow input streams and sort messages chronologically during time frame set by --sync-interval-ms option
      --tail <N>                         Number of last messages to preload from each file in --follow mode [default: 10]
      --sync-interval-ms <MILLISECONDS>  Synchronization interval for live streaming mode enabled by --follow option [default: 100]
      --paging <WHEN>                    Control pager usage (HL_PAGER or PAGER) [env: HL_PAGING=] [default: auto] [possible values: auto, always, never]
  -P                                     Handful alias for --paging=never, overrides --paging option
      --help                             Print help
  -V, --version                          Print version

Filtering Options:
  -l, --level <LEVEL>    Filter messages by level [env: HL_LEVEL=]
      --since <TIME>     Filter messages by timestamp >= <TIME> (--time-zone and --local options are honored)
      --until <TIME>     Filter messages by timestamp <= <TIME> (--time-zone and --local options are honored)
  -f, --filter <FILTER>  Filter messages by field values [k=v, k~=v, k~~=v, 'k!=v', 'k!~=v', 'k!~~=v'] where ~ does substring match and ~~ does regular expression match
  -q, --query <QUERY>    Filter using query, accepts expressions from --filter and supports '(', ')', 'and', 'or', 'not', 'in', 'contain', 'like', '<', '>', '<=', '>=', etc

Output Options:
      --color [<WHEN>]        Color output control [env: HL_COLOR=] [default: auto] [possible values: auto, always, never]
  -c                          Handful alias for --color=always, overrides --color option
      --theme <THEME>         Color theme [env: HL_THEME=] [default: universal]
  -r, --raw                   Output raw source messages instead of formatted messages, which can be useful for applying filters and saving results in their original format
      --no-raw                Disable raw source messages output, overrides --raw option
      --raw-fields            Output field values as is, without unescaping or prettifying
  -h, --hide <KEY>            Hide or reveal fields with the specified keys, prefix with ! to reveal, specify '!*' to reveal all
      --flatten <WHEN>        Whether to flatten objects [env: HL_FLATTEN=] [default: always] [possible values: never, always]
  -t, --time-format <FORMAT>  Time format, see https://man7.org/linux/man-pages/man1/date.1.html [env: HL_TIME_FORMAT=] [default: "%b %d %T.%3N"]
  -Z, --time-zone <TZ>        Time zone name, see column "TZ identifier" at https://en.wikipedia.org/wiki/List_of_tz_database_time_zones [env: HL_TIME_ZONE=] [default: UTC]
  -L, --local                 Use local time zone, overrides --time-zone option
      --no-local              Disable local time zone, overrides --local option
  -e, --hide-empty-fields     Hide empty fields, applies for null, string, object and array fields only [env: HL_HIDE_EMPTY_FIELDS=]
  -E, --show-empty-fields     Show empty fields, overrides --hide-empty-fields option [env: HL_SHOW_EMPTY_FIELDS=]
      --input-info <VARIANT>  Show input number and/or input filename before each message [default: auto] [possible values: auto, none, full, compact, minimal]
  -o, --output <FILE>         Output file

Input Options:
      --input-format <FORMAT>       Input format [env: HL_INPUT_FORMAT=] [default: auto] [possible values: auto, json, logfmt]
      --unix-timestamp-unit <UNIT>  Unix timestamp unit [env: HL_UNIX_TIMESTAMP_UNIT=] [default: auto] [possible values: auto, s, ms, us, ns]
      --allow-prefix                Allow non-JSON prefixes before JSON messages [env: HL_ALLOW_PREFIX=]
      --delimiter <DELIMITER>       Log message delimiter, [NUL, CR, LF, CRLF] or any custom string

Advanced Options:
      --interrupt-ignore-count <N>  Number of interrupts to ignore, i.e. Ctrl-C (SIGINT) [env: HL_INTERRUPT_IGNORE_COUNT=] [default: 3]
      --buffer-size <SIZE>          Buffer size [env: HL_BUFFER_SIZE=] [default: "256 KiB"]
      --max-message-size <SIZE>     Maximum message size [env: HL_MAX_MESSAGE_SIZE=] [default: "64 MiB"]
  -C, --concurrency <N>             Number of processing threads [env: HL_CONCURRENCY=]
      --shell-completions <SHELL>   Print shell auto-completion script and exit [possible values: bash, elvish, fish, powershell, zsh]
      --man-page                    Print man page and exit
      --list-themes                 Print available themes and exit
      --dump-index                  Print debug index metadata (in --sort mode) and exit

Performance

MacBook Pro (16-inch, 2021)

CPU: Apple M1 Max CPU
OS: macOS Sequoia 15.2

Data: ~ 2.3 GiB log file, 6 000 000 lines

hl v0.30.2 ~ 1.1 seconds

$ time hl --config - example.log -c -o /dev/null
hl --config - example.log -c -o /dev/null  9.80s user 0.61s system 915% cpu 1.138 total

hlogf v1.4.1 ~ 8.7 seconds

$ time hlogf example.log --color always >/dev/null
hlogf example.log --color always > /dev/null  6.85s user 1.94s system 100% cpu 8.738 total

humanlog v0.7.8 ~ 79 seconds

$ time humanlog <example.log --color always >/dev/null
humanlog> reading stdin...
humanlog --color always < example.log > /dev/null  87.68s user 7.33s system 120% cpu 1:19.01 total

fblog v4.13.1 ~ 34 seconds

$ time fblog example.log >/dev/null
fblog example.log > /dev/null  31.32s user 2.22s system 99% cpu 33.553 total

fblog with -d flag v4.13.1 ~ 146 seconds

$ time fblog -d example.log >/dev/null
fblog -d example.log > /dev/null  131.88s user 14.55s system 99% cpu 2:26.45 total

See #132 for how to repeat measurements

Name		Name	Last commit message	Last commit date
Latest commit History 1,070 Commits
.build/capnp		.build/capnp
.cargo		.cargo
.github		.github
.vscode		.vscode
benches		benches
build		build
contrib		contrib
crate		crate
diagram		diagram
doc		doc
etc/defaults		etc/defaults
extra @ 947ce45		extra @ 947ce45
nix		nix
sample		sample
schema		schema
src		src
.envrc		.envrc
.gitignore		.gitignore
.gitmodules		.gitmodules
.markdownlint.json		.markdownlint.json
.rustfmt.toml		.rustfmt.toml
.taplo.toml		.taplo.toml
.tarpaulin.toml		.tarpaulin.toml
Cargo.lock		Cargo.lock
Cargo.toml		Cargo.toml
LICENSE		LICENSE
Makefile		Makefile
README.md		README.md
build.rs		build.rs
codecov.yml		codecov.yml
flake.lock		flake.lock
flake.nix		flake.nix

OS	Location
macOS	~/.config/hl/themes/*.{yaml,toml,json}
Linux	~/.config/hl/themes/*.{yaml,toml,json}
Windows	%USERPROFILE%\AppData\Roaming\hl\themes*.{yaml,toml,json}

License

pamburus/hl

Folders and files

Latest commit

History

Repository files navigation

hl

Features overview

Performance comparison chart

Performance comparison with humanlog, hlogf and fblog on a 2.3 GiB log file

Installation options

macOS

Linux

Windows

NixOS

Examples

Screenshot

Features and usage

Concatenation of multiple log files

Support for compressed (bzip2, gzip, xz, zstd) log files

Automatic usage of pager

Quick filtering by log level

Using live log streaming

Filtering by field values

Performing complex queries

Filtering by time range

Hiding or revealing selected fields

Sorting messages chronologically

Sorting messages chronologically with following the changes

Configuration files

Default configuration file

Environment variables

Precedence of configuration sources (from lowest priority to highest priority)

Environment variables examples

Themes

Stock themes

Selecting current theme

Selecting themes with preview

Custom themes

Used terminal color schemes

iTerm2

Alacritty

Complete set of options and flags

Performance

About

Topics

Resources

License

Stars

Watchers

Forks

Releases 106

Packages 0

Contributors 11

Languages

Packages