Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Version: 4.99.99 #105

Closed
wants to merge 2 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
310 changes: 28 additions & 282 deletions NEWS.md
Original file line number Diff line number Diff line change
@@ -1,308 +1,54 @@
4.7.1
=====

*This is the combination of the news entries of `4.7.0` and `4.7.1`.*
*`4.7.1` hotfixed two crashes related to configuration reload.*

Read Axoflow's [blog post](https://axoflow.com/axosyslog-release-4-7/) for more details.
You can read more about the new features in the AxoSyslog [documentation](https://axoflow.com/docs/axosyslog-core/).
4.99.99
=======

## Highlights

### Collecting Jellyfin logs

The new `jellyfin()` source, reads Jellyfin logs from its log file output.

Example minimal config:
```
source s_jellyfin {
jellyfin(
base-dir("/path/to/my/jellyfin/root/log/dir")
filename-pattern("log_*.log")
);
};
```

For more details about Jellyfin logging, see:
* https://jellyfin.org/docs/general/administration/configuration/#main-configuration
* https://jellyfin.org/docs/general/administration/configuration/#log-directory

As the `jellyfin()` source is based on a `wildcard-file()` source, all of the
`wildcard-file()` source options are applicable, too.
([#4802](https://github.com/syslog-ng/syslog-ng/pull/4802))

### Collecting *arr logs

Use the newly added `*arr()` sources to read various *arr logs:
* `lidarr()`
* `prowlarr()`
* `radarr()`
* `readarr()`
* `sonarr()`
* `whisparr()`

Example minimal config:
```
source s_radarr {
radarr(
dir("/path/to/my/radarr/log/dir")
);
};
```

The logging module is stored in the `<prefix><module>` name-value pair,
for example: `.radarr.module` => `ImportListSyncService`.
The prefix can be modified with the `prefix()` option.
([#4803](https://github.com/syslog-ng/syslog-ng/pull/4803))

## Features

* `opentelemetry()`, `syslog-ng-otlp()` source: Added `concurrent-requests()` option.

This option configures the maximal number of in-flight gRPC requests per worker.
Setting this value to the range of 10s or 100s is recommended when there are a
high number of clients sending simultaneously.

Ideally, `workers() * concurrent-requests()` should be greater or equal to
the number of clients, but this can increase the memory usage.
([#4827](https://github.com/syslog-ng/syslog-ng/pull/4827))

* `loki()`: Support multi-tenancy with the new `tenant-id()` option
([#4812](https://github.com/syslog-ng/syslog-ng/pull/4812))

* `s3()`: Added support for authentication from environment.

The `access-key()` and `secret-key()` options are now optional,
which makes it possible to use authentication methods originated
from the environment, e.g. `AWS_...` environment variables or
credentials files from the `~/.aws/` directory.

For more info, see:
https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html
([#4881](https://github.com/syslog-ng/syslog-ng/pull/4881))

* gRPC based drivers: Added `channel-args()` option.

Affected drivers are:
* `bigquery()` destination
* `loki()` destination
* `opentelemetry()` source and destination
* `syslog-ng-otlp()` source and destination

The `channel-args()` option accepts name-value pairs and sets channel arguments
defined in https://grpc.github.io/grpc/core/group__grpc__arg__keys.html

Example config:
```
opentelemetry(
channel-args(
"grpc.loadreporting" => 1
"grpc.minimal_stack" => 0
)
);
```
([#4827](https://github.com/syslog-ng/syslog-ng/pull/4827))

* `${TRANSPORT}` macro: Added support for locally created logs.

New values are:
* "local+unix-stream"
* "local+unix-dgram"
* "local+file"
* "local+pipe"
* "local+program"
* "local+devkmsg"
* "local+journal"
* "local+afstreams"
* "local+openbsd"
([#4777](https://github.com/syslog-ng/syslog-ng/pull/4777))

* `tags`: Added new built-in tags that help identifying parse errors.

New tags are:
* "message.utf8_sanitized"
* "message.parse_error"
* "syslog.missing_pri"
* "syslog.missing_timestamp"
* "syslog.invalid_hostname"
* "syslog.unexpected_framing"
* "syslog.rfc3164_missing_header"
* "syslog.rfc5424_unquoted_sdata_value"
([#4804](https://github.com/syslog-ng/syslog-ng/pull/4804))

* `mqtt()` source: Added `${MQTT_TOPIC}` name-value pair.

It is useful for the cases where `topic()` contains wildcards.

Example config:
```
log {
source { mqtt(topic("#")); };
destination { stdout(template("${MQTT_TOPIC} - ${MESSAGE}\n")); };
};
```
([#4824](https://github.com/syslog-ng/syslog-ng/pull/4824))

* `template()`: Added a new template function: `$(tags-head)`

This template function accepts multiple tag names, and returns the
first one that is set.

Example config:
```
# resolves to "bar" if "bar" tag is set, but "foo" is not
template("$(tags-head foo bar baz)")
```
([#4804](https://github.com/syslog-ng/syslog-ng/pull/4804))

* `s3()`: Use default AWS URL if `url()` is not set.
([#4813](https://github.com/syslog-ng/syslog-ng/pull/4813))

* `opentelemetry()`, `syslog-ng-otlp()` source: Added `log-fetch-limit()` option.

This option can be used to fine tune the performance. To minimize locking while
moving messages between source and destination side queues, syslog-ng can move
messages in batches. The `log-fetch-limit()` option sets the maximal size of
the batch moved by a worker. By default it is equal to `log-iw-size() / workers()`.
([#4827](https://github.com/syslog-ng/syslog-ng/pull/4827))

* `dqtool`: add option for truncating (compacting) abandoned disk-buffers
([#4875](https://github.com/syslog-ng/syslog-ng/pull/4875))
<Fill this block manually from the blocks below>

## Bugfixes

* `opentelemetry()`: fix crash when an invalid configuration needs to be reverted
([#4910](https://github.com/syslog-ng/syslog-ng/pull/4910))

* gRPC drivers: fixed a crash when gRPC drivers were used and syslog-ng was reloaded
([#4909](https://github.com/syslog-ng/syslog-ng/pull/4909))

* `opentelemetry()`, `syslog-ng-otlp()` source: Fixed a crash.

It occurred with multiple `workers()` during high load.
([#4827](https://github.com/syslog-ng/syslog-ng/pull/4827))

* `rename()`: Fixed a bug, which always converted the renamed NV pair to string type.
([#4847](https://github.com/syslog-ng/syslog-ng/pull/4847))

* With IPv6 disabled, there were linking errors
([#4880](https://github.com/syslog-ng/syslog-ng/pull/4880))

## Metrics

* `http()`: Added a new counter for HTTP requests.
* `csv-parser()`: fix escape-backslash-with-sequences dialect on ARM

It is activated on `stats(level(1));`.
`csv-parser()` produced invalid output on platforms where char is an unsigned type.
([#4947](https://github.com/axoflow/axosyslog/pull/4947))

Example metrics:
```
syslogng_output_http_requests_total{url="http://localhost:8888/bar",response_code="200",driver="http",id="#anon-destination0#0"} 16
syslogng_output_http_requests_total{url="http://localhost:8888/bar",response_code="401",driver="http",id="#anon-destination0#0"} 2
syslogng_output_http_requests_total{url="http://localhost:8888/bar",response_code="502",driver="http",id="#anon-destination0#0"} 1
syslogng_output_http_requests_total{url="http://localhost:8888/foo",response_code="200",driver="http",id="#anon-destination0#0"} 24
```
([#4805](https://github.com/syslog-ng/syslog-ng/pull/4805))

* gRPC based destination drivers: Added gRPC request related metrics.

Affected drivers:
* `opentelemetry()`
* `syslog-ng-otlp()`
* `bigquery()`
* `loki()`

Example metrics:
```
syslogng_output_grpc_requests_total{driver="syslog-ng-otlp",url="localhost:12345",response_code="ok"} 49
syslogng_output_grpc_requests_total{driver="syslog-ng-otlp",url="localhost:12345",response_code="unavailable"} 11
```
([#4811](https://github.com/syslog-ng/syslog-ng/pull/4811))

* New metric to monitor destination reachability

`syslogng_output_unreachable` is a bool-like metric, which shows whether a
destination is reachable or not.

`sum()` can be used to count all unreachable outputs, hence the negated name.

It is currently available for the `network()`, `syslog()`, `unix-*()`
destinations, and threaded destinations (`http()`, `opentelemetry()`, `redis()`,
`mongodb()`, `python()`, etc.).
([#4876](https://github.com/syslog-ng/syslog-ng/pull/4876))

* destinations: Added "syslogng_output_event_retries_total" counter.

This counter is available for the following destination drivers:
* `amqp()`
* `bigquery()`
* `http()` and all http based drivers
* `java()`
* `kafka()`
* `loki()`
* `mongodb()`
* `mqtt()`
* `opentelemetry()`
* `python()` and all python based drivers
* `redis()`
* `riemann()`
* `smtp()`
* `snmp()`
* `sql()`
* `stomp()`
* `syslog-ng-otlp()`

Example metrics:
```
syslogng_output_event_retries_total{driver="http",url="http://localhost:8888/${path}",id="#anon-destination0#0"} 5
```
([#4807](https://github.com/syslog-ng/syslog-ng/pull/4807))

* `syslogng_memory_queue_capacity`

Shows the capacity (maximum possible size) of each queue.
Note that this metric publishes `log-fifo-size()`, which only limits non-flow-controlled messages.
Messages coming from flow-controlled paths are not limited by `log-fifo-size()`, their corresponding
source `log-iw-size()` is the upper limit.
([#4831](https://github.com/syslog-ng/syslog-ng/pull/4831))

## Other changes

* `opentelemetry()`, `syslog-ng-otlp()` source: Changed the backpressure behavior.

syslog-ng no longer returns `UNAVAILABLE` to the gRPC request, when it cannot forward
the received message because of backpressure. Instead, syslog-ng will block until the
destination can accept more messages.
([#4827](https://github.com/syslog-ng/syslog-ng/pull/4827))

* `opentelemetry()`, `syslog-ng-otlp()` source: `log-iw-size()` is now split between workers.
([#4827](https://github.com/syslog-ng/syslog-ng/pull/4827))
* packages/dbld: add support for Ubuntu 24.04 (Noble Numbat)
([#4925](https://github.com/axoflow/axosyslog/pull/4925))

* APT packages: Dropped Debian Buster support.
* `syslog-ng-ctl`: do not show orphan metrics for `stats prometheus`

Old packages are still available, but new syslog-ng versions will not
be available on Debian Buster
([#4840](https://github.com/syslog-ng/syslog-ng/pull/4840))
As the `stats prometheus` command is intended to be used to forward metrics
to Prometheus or any other time-series database, displaying orphaned metrics
should be avoided in order not to insert new data points when a given metric
is no longer alive.

* `dbld`: AlmaLinux 8 support
([#4902](https://github.com/syslog-ng/syslog-ng/pull/4902))
In case you are interested in the last known value of orphaned counters, use
the `stats` or `query` subcommands.
([#4921](https://github.com/axoflow/axosyslog/pull/4921))

* `s3()`: new metric `syslogng_output_event_bytes_total`
([#4958](https://github.com/axoflow/axosyslog/pull/4958))

## syslog-ng Discord
* `bigquery()`, `loki()`, `opentelemetry()`, `cloud-auth()`: C++ modules can be compiled with clang

For a bit more interactive discussion, join our Discord server:
Compiling and using these C++ modules are now easier on FreeBSD and macOS.
([#4933](https://github.com/axoflow/axosyslog/pull/4933))

[![Axoflow Discord Server](https://discordapp.com/api/guilds/1082023686028148877/widget.png?style=banner2)](https://discord.gg/E65kP9aZGm)

## Credits

syslog-ng is developed as a community project, and as such it relies
on volunteers, to do the work necessarily to produce syslog-ng.
AxoSyslog is developed as a community project, and as such it relies
on volunteers, to do the work necessary to produce AxoSyslog.

Reporting bugs, testing changes, writing code or simply providing
feedback are all important contributions, so please if you are a user
of syslog-ng, contribute.
feedback is an important contribution, so please if you are a user
of AxoSyslog, contribute.

We would like to thank the following people for their contribution:

Arpad Kunszt, Attila Szakacs, Balazs Scheidler, Bálint Horváth, Hofi,
Kovács, Gergő Ferenc, László Várady, Peter Marko, shifter
Arpad Kunszt, Attila Szakacs, Balazs Scheidler, Ferenc HERNADI,
Gabor Kozma, Hofi, Kristof Gyuracz, László Várady, Mate Ory, Máté Őry,
Robert Fekete, Szilard Parrag, Wolfram Joost, shifter
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ docker pull ghcr.io/axoflow/axosyslog:nightly
> Note: These named packages are automatically updated when a new syslog-ng package is released. To install a specific version, run `docker pull ghcr.io/axoflow/axosyslog:<version-number>`, for example:
>
> ```shell
> docker pull ghcr.io/axoflow/axosyslog:4.7.1
> docker pull ghcr.io/axoflow/axosyslog:4.99.99
> ```

### Difference from upstream images
Expand Down
2 changes: 1 addition & 1 deletion VERSION.txt
Original file line number Diff line number Diff line change
@@ -1 +1 @@
4.7.1
4.99.99
2 changes: 1 addition & 1 deletion contrib/openbsd-packaging/syslog-ng.conf
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
# This should provide behavior similar to OpenBSD's syslog.conf(5).
# 2021-05-24 [email protected]

@version: 4.7
@version: 4.99
@requires openbsd

options {
Expand Down
4 changes: 2 additions & 2 deletions doc/man/dqtool.1.xml
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@
<refmeta>
<refentrytitle>dqtool</refentrytitle>
<manvolnum>1</manvolnum>
<refmiscinfo class="version">4.7</refmiscinfo>
<refmiscinfo class="version">4.99</refmiscinfo>
<refmiscinfo class="source"/>
</refmeta>
<refnamediv>
Expand Down Expand Up @@ -117,7 +117,7 @@ Mar 3 10:52:05 tristram localprg[1234]: seq: 0000011631, runid: 1267609923, sta
<link linkend="syslog-ng.8"><command>syslog-ng</command>(8)</link>
</para>
<note version="5.0">
<para>For the detailed documentation of see <link xmlns:ns1="http://www.w3.org/1999/xlink" ns1:href="https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.37/administration-guide"><command>The 4.7 Administrator Guide</command></link></para>
<para>For the detailed documentation of see <link xmlns:ns1="http://www.w3.org/1999/xlink" ns1:href="https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.37/administration-guide"><command>The 4.99 Administrator Guide</command></link></para>
<para>If you experience any problems or need help with syslog-ng, visit the <link xmlns:ns1="http://www.w3.org/1999/xlink" ns1:href="https://lists.balabit.hu/mailman/listinfo/syslog-ng"><command>syslog-ng mailing list</command></link>.</para>
<para>For news and notifications about of syslog-ng, visit the <link xmlns:ns1="http://www.w3.org/1999/xlink" ns1:href="https://syslog-ng.org/blogs/"><command>syslog-ng blogs</command></link>.</para>
</note>
Expand Down
4 changes: 2 additions & 2 deletions doc/man/loggen.1.xml
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@
<refmeta>
<refentrytitle>loggen</refentrytitle>
<manvolnum>1</manvolnum>
<refmiscinfo class="version">4.7</refmiscinfo>
<refmiscinfo class="version">4.99</refmiscinfo>
<refmiscinfo class="source"/>
</refmeta>
<refnamediv>
Expand Down Expand Up @@ -280,7 +280,7 @@
<link linkend="syslog-ng.conf.5"><command>syslog-ng.conf</command>(5)</link>
</para>
<note version="5.0">
<para>For the detailed documentation of see <link xmlns:ns1="http://www.w3.org/1999/xlink" ns1:href="https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.37/administration-guide"><command>The 4.7 Administrator Guide</command></link></para>
<para>For the detailed documentation of see <link xmlns:ns1="http://www.w3.org/1999/xlink" ns1:href="https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.37/administration-guide"><command>The 4.99 Administrator Guide</command></link></para>
<para>If you experience any problems or need help with syslog-ng, visit the <link xmlns:ns1="http://www.w3.org/1999/xlink" ns1:href="https://lists.balabit.hu/mailman/listinfo/syslog-ng"><command>syslog-ng mailing list</command></link>.</para>
<para>For news and notifications about of syslog-ng, visit the <link xmlns:ns1="http://www.w3.org/1999/xlink" ns1:href="https://syslog-ng.org/blogs/"><command>syslog-ng blogs</command></link>.</para>
</note>
Expand Down
Loading
Loading