[inputs.socket_listener] Metrics size limitations

[MarekZydor]

Relevant telegraf.conf

[agent]
  interval = "10s"
  round_interval = true
  debug = true
  logfile = ""
  omit_hostname = true

[[inputs.socket_listener]]
  service_address = "unix:///tmp/telegraf.sock"
  data_format = "influx"
  read_buffer_size = "1000KiB"

[[outputs.file]]
  files = ["stdout"]

Logs from Telegraf

2024-10-27T22:20:00Z I! Starting Telegraf 1.32.1 brought to you by InfluxData the makers of InfluxDB
2024-10-27T22:20:00Z I! Available plugins: 235 inputs, 9 aggregators, 32 processors, 26 parsers, 62 outputs, 6 secret-stores
2024-10-27T22:20:00Z I! Loaded inputs: socket_listener
2024-10-27T22:20:00Z I! Loaded aggregators:
2024-10-27T22:20:00Z I! Loaded processors:
2024-10-27T22:20:00Z I! Loaded secretstores:
2024-10-27T22:20:00Z I! Loaded outputs: file
2024-10-27T22:20:00Z I! Tags enabled:
2024-10-27T22:20:00Z I! [agent] Config: Interval:10s, Quiet:false, Hostname:"", Flush Interval:10s
2024-10-27T22:20:00Z D! [agent] Initializing plugins
2024-10-27T22:20:00Z D! [agent] Connecting outputs
2024-10-27T22:20:00Z D! [agent] Attempting connection to [outputs.file]
2024-10-27T22:20:00Z D! [agent] Successfully connected to outputs.file
2024-10-27T22:20:00Z D! [agent] Starting service inputs
2024-10-27T22:20:00Z I! [inputs.socket_listener] Listening on unix:///tmp/telegraf.sock
2024-10-27T22:20:07Z E! [inputs.socket_listener] Error in plugin: bufio.Scanner: token too long

System info

telegraf 1.32.1, ubuntu 22.04

Docker

No response

Steps to reproduce

You can use the following C++ application to send a metric with very long text

#include <iostream>
#include <cstring>
#include <sys/socket.h>
#include <sys/un.h>
#include <unistd.h>

int main() {
    const char* socket_path = "/tmp/telegraf.sock";
    const char* message = R"(cpu,host=server1,region=us-west usage_idle="LONG TEXT 64KiB+" 1634729655000000000)";

    int sockfd = socket(AF_UNIX, SOCK_STREAM, 0);
    if (sockfd == -1) {
        perror("Socket creation failed");
        return EXIT_FAILURE;
    }

    struct sockaddr_un addr;
    std::memset(&addr, 0, sizeof(addr));
    addr.sun_family = AF_UNIX;
    std::strncpy(addr.sun_path, socket_path, sizeof(addr.sun_path) - 1);

    if (connect(sockfd, (struct sockaddr*)&addr, sizeof(addr)) == -1) {
        perror("Connect failed");
        close(sockfd);
        return EXIT_FAILURE;
    }

    ssize_t bytes_sent = send(sockfd, message, std::strlen(message), 0);
    if (bytes_sent == -1) {
        perror("Send failed");
        close(sockfd);
        return EXIT_FAILURE;
    }

    std::cout << "Sent message: " << message << std::endl;

    close(sockfd);

    return EXIT_SUCCESS;
}

You will get the same behavior when you try to transfer data over TCP, UDP, or Unix Domain Sockets (types SOCK_STREAM - unix and SOCK_DGRAM - unixgram). This problem is not related to IPv4 MTU or buffer sizes, but only to how the plugin was implemented. You could try putting smaller fields that exceed a total of 64 KB, and then you will get the same result.

Expected behavior

Data is correctly parsed and printed to stdout as defined in telegraf.conf

Actual behavior

When TCP (SOCK_STREAM) is used, an error related to Golang's bufio.Scanner is printed.
When UDP (SOCK_DGRAM) is used, an error about being unable to find the end character of a specific field is printed (as shown below):
2024-10-27T22:43:35Z E! [inputs.socket_listener] Error in plugin: metric parse error: expected field at 1:65537: "...skpxcj{part of long text}<-- here

Additional info

For now, I have found a workaround by using http_listener_v2, which is able to parse large metrics. But, I'd like to use UDP in the future.

[srebhan]

@MarekZydor please test the binary in PR #16111, available as soon as CI finished the tests, and let me know if this fixes the issue!

[MarekZydor]

Hey, that fixed the issue only when I use TCP (SOCK_STREAM). Interestingly, I don't have any problems resending measurements with a length of 200KB to InfluxDB, so that's great. But still, when receiving via UDP (SOCK_DGRAM), I get errors like this:

2024-10-30T23:50:35Z I! [inputs.socket_listener] Listening on unixgram:///tmp/telegraf.sock
2024-10-30T23:50:39Z E! [inputs.socket_listener] Error in plugin: metric parse error: expected field at 1:65537: "...jgpmyre<-- here"

[MarekZydor]

I guess another PR is needed to fix that.

[srebhan]

Could you please report this as a separate bug, best with a way to reproduce the issue!

[MarekZydor]

New issue created #16118