Multiple appenders with different namespace filters

[yayitswei]

I'd like to separate my logs into different files based on namespaces. Specifically, there's one noisy namespace I'd like to route to its own file, and have a catchall file for everything else. It's not clear from the docs how to do this (ns-blacklist and ns-whitelist seem to be for the global level rather than appender-level), is it possible in Timbre?

Thanks for making this useful library btw!

[ptaoussanis]

Hi Wei, hope you're well!

ns-blacklist and ns-whitelist seem to be for the global level rather than appender-level

That's correct, these are general-purpose high-level filters to include/exclude log output entirely for matching namespaces. If you need something more sophisticated, you can just ignore those and implement your own filtering however you'd like.

Timbre's inherently a very simple design, no magic:

1. Logging calls generate a data map. {:?ns-str _ :level _ ...}
2. The resulting data map goes through any middleware fns. (fn [data]) -> ?data
3. The resulting data map goes to all appender fns.
4. The appender fns can do what they want with the data. (fn [data]) -> ?effects

The built-in ns-blacklist andns-whitelist are just simple common-case middleware fns.

So if you want some kind of custom conditional logic, you can either add it within your appender fn/s, or you can add some middleware. How you choose to do this is entirely up to you, it's all just fns.

Specifically, there's one noisy namespace I'd like to route to its own file, and have a catchall file for everything else.

Two obvious options come to mind:

1. Mod your appender fn to use an output filename that depends on (:?ns-str <data>).
2. Use two appenders, one that outputs to "catchall.txt" and one that outputs to "noisy.txt". Mod each appender to whitelist/blacklist the appropriate namespaces.

In both cases you'll need to actually mod the appender fn though which might be inconvenient if you're using a built-in appender?

Give me 10mins, have an idea.

[ptaoussanis]

Okay have just cut [com.taoensso/timbre "4.7.0"] that makes things like this easier w/o needing to touch any appender fns.

Add two appenders, one that outputs to "catchall.txt" and one that outputs to "noisy.txt".

Then:

• If you only need to filter based on namespaces, use appender-level :ns-whitelists and :ns-blacklists (these are set in the appender map)
• If you need more general conditional logic, use appender-level :middleware-fns (set in the appender map)

So there's namespace filters at the global level, and per-appender,
And there's middleware at the global level, and per-appender.

Does that help / make sense?

[yayitswei]

Thanks Peter, you as well!

Just tried 4.7.0 and it works great. For future reference, here's my working config:

(defn timbre-config! []
  (log/merge-config!
   {:level :debug
    :appenders {:app (assoc (appenders/spit-appender {:fname "app.log"})
                            :ns-blacklist ["com.noisy.*"])
                :noisy (assoc (appenders/spit-appender {:fname "noisy.log"})
                              :ns-whitelist ["com.noisy.*"]
                              :min-level :info)}}))

Thanks for going out of your way to cut a new release!

[nathants]

👍