[icinga2 agent] generates huge log file

[Nokius]

Describe the bug

In the last 24h several icinga2 agent went crazy and created huge (filled up the entire partition) log files. The systems run the linux agent xxx. They are different kind, and didn't had any changes lately. So fare we never faced that issue.

In all of the log we saw spamming entries for:

{{ [2020-09-30 20:46:38 +0200] critical/ApiListener: Cannot accept new connection.}}
{{ [2020-09-30 20:46:38 +0200] critical/Socket: accept() failed with error code 24, "Too many open files"}}
{{ [2020-09-30 20:46:38 +0200] critical/ApiListener: Cannot accept new connection.}}
{{ [2020-09-30 20:46:38 +0200] critical/Socket: accept() failed with error code 24, "Too many open files"}}

Did anyone has experienced the issue may too or any hind why that could happen.

To Reproduce

not known for now

Expected behavior

agent should not have partition flooding log file

Screenshots

Your Environment

Include as many relevant details about the environment you experienced the problem in

• Version used (icinga2 --version):
  [agent]

icinga2 - The Icinga 2 network monitoring daemon (version: r2.10.2-1)

Copyright (c) 2012-2018 Icinga Development Team (https://icinga.com/)
License GPLv2+: GNU GPL version 2 or later <http://gnu.org/licenses/gpl2.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

System information:
  Platform: Debian GNU/Linux
  Platform version: 8 (jessie)
  Kernel: Linux
  Kernel version: 3.16.0-7-amd64
  Architecture: x86_64

Build information:
  Compiler: GNU 4.9.2
  Build host: bc2295102b6c

Application information:

General paths:
  Config directory: /etc/icinga2
  Data directory: /var/lib/icinga2
  Log directory: /var/log/icinga2
  Cache directory: /var/cache/icinga2
  Spool directory: /var/spool/icinga2
  Run directory: /run/icinga2

Old paths (deprecated):
  Installation root: /usr
  Sysconf directory: /etc
  Run directory (base): /run
  Local state directory: /var

Internal paths:
  Package data directory: /usr/share/icinga2
  State path: /var/lib/icinga2/icinga2.state
  Modified attributes path: /var/lib/icinga2/modified-attributes.conf
  Objects path: /var/cache/icinga2/icinga2.debug
  Vars path: /var/cache/icinga2/icinga2.vars
  PID path: /run/icinga2/icinga2.pid

• Operating System and version:

Distributor ID: Debian
Description: Debian GNU/Linux 8.11 (jessie)
Release: 8.11
Codename: jessie

• Enabled features (icinga2 feature list):

Enabled features: api checker command ido-pgsql influxdb mainlog notification syslog

• Icinga Web 2 version and modules (System - About):

2.6.2

• Config validation (icinga2 daemon -C):

no errors

• If you run multiple Icinga 2 instances, the zones.conf file (or icinga2 object list --type Endpoint and icinga2 object list --type Zone) from all affected nodes.

Object 'XXXXXXXXX' of type 'Endpoint':
  % declared in '/var/lib/icinga2/api/packages/director/d5941998-44f8-462b-a8fd-67fdd7e66126/zones.d/xxxxxxx-satellite-zone/endpoints.conf', lines 180:1-180:53
  * __name = "xxxxxxx.xxxxxxx"
  * host = "xxx.xxx.xxx.xxx"
    % = modified in '/var/lib/icinga2/api/packages/director/d5941998-44f8-462b-a8fd-67fdd7e66126/zones.d/xxxxxxx-satellite-zone/endpoints.conf', lines 181:5-181:24
  * log_duration = 86400
    % = modified in '/var/lib/icinga2/api/packages/director/d5941998-44f8-462b-a8fd-67fdd7e66126/zones.d/xxxxxxx-satellite-zone/endpoints.conf', lines 183:5-183:21
  * name = "xxxxxxx.xxxxxxx"
  * package = "director"
  * port = "5665"
    % = modified in '/var/lib/icinga2/api/packages/director/d5941998-44f8-462b-a8fd-67fdd7e66126/zones.d/xxxxxxx-satellite-zone/endpoints.conf', lines 182:5-182:17
  * source_location
    * first_column = 1
    * first_line = 180
    * last_column = 53
    * last_line = 180
    * path = "/var/lib/icinga2/api/packages/director/d5941998-44f8-462b-a8fd-67fdd7e66126/zones.d/xxxxxxx-satellite-zone/endpoints.conf"
  * templates = [ "xxxxxxx.xxxxxxx" ]
    % = modified in '/var/lib/icinga2/api/packages/director/d5941998-44f8-462b-a8fd-67fdd7e66126/zones.d/xxxxxxx-satellite-zone/endpoints.conf', lines 180:1-180:53
  * type = "Endpoint"
  * zone = "xxxxxxx-satellite-zone"

Additional context

[sebek72]

I had similar issued, caused by some clients not being available. Relates to SSL issues and timeouts See #7203

[Al2Klimov]

Windows Event Log

Icinga 2.13 now supports logging to the Windows Event Log. Icinga will now also log messages from the early startup phase to the Windows Event Log. These were previously missing from the log file and you could only see them by manually starting Icinga in the foreground.

This feature is now enabled and replaces the existing mainlog feature logging to a file. When upgrading, the installer will enable the windowseventlog feature and disable the mainlog feature.

– https://icinga.com/docs/icinga-2/latest/doc/16-upgrading-icinga-2/

No mainlog – no mainlog rotation needed.