"Could not initialize input inputs.mysql" while testing #12591

[jfrye121]

Relevant telegraf.conf

# Global tags can be specified here in key="value" format.
[global_tags]
  # dc = "us-east-1" # will tag all metrics with dc=us-east-1
  # rack = "1a"
  ## Environment variables can be used as tags, and throughout the config file
  user = "sumomonitoring"

###############################################################################
#                            SECRETSTORE PLUGINS                              #
###############################################################################


# # File based Javascript Object Signing and Encryption based secret-store
[[secretstores.jose]]
#   ## Unique identifier for the secret-store.
#   ## This id can later be used in plugins to reference the secrets
#   ## in this secret-store via @{<id>:<secret_key>} (mandatory)
id = "my_secrets"
#
#   ## Directory for storing the secrets
path = "/etc/telegraf/secrets"
#
#   ## Password to access the secrets.
#   ## If no password is specified here, Telegraf will prompt for it at startup time.
#   # password = ""


[[inputs.mysql]]
  servers = ["${user}:@{my_secrets:sumomonitoring}@tcp(internalip:3306)/?tls=false"]
  metric_version = 2
  table_schema_databases = []
  perf_summary_events = []
  gather_table_schema = true
  gather_process_list = true
  gather_info_schema_auto_inc = true
  gather_user_statistics = true
  gather_slave_status = true
  gather_table_io_waits = true
  gather_table_lock_waits = true
  gather_index_io_waits = true
  gather_event_waits = true
  gather_file_events_stats = true
  gather_perf_events_statements = true
  interval_slow = "30m"
[inputs.mysql.tags]
  environment="prod"
  component="database"
  db_system="mariadb"
  db_cluster="default"
  db_cluster_address = "publicip"
  db_cluster_port = "3306"
[[outputs.sumologic]]
  url = "urlgoeshere"
  data_format = "prometheus"

Logs from Telegraf

2023-03-21T08:27:02-05:00 I! Starting Telegraf 1.26.0
2023-03-21T08:27:02-05:00 I! Available plugins: 235 inputs, 9 aggregators, 27 processors, 22 parsers, 57 outputs, 2 secret-stores
2023-03-21T08:27:02-05:00 I! Loaded inputs: cpu disk diskio kernel mem mysql processes swap system
2023-03-21T08:27:02-05:00 I! Loaded aggregators:
2023-03-21T08:27:02-05:00 I! Loaded processors:
2023-03-21T08:27:02-05:00 I! Loaded secretstores: my_secrets
2023-03-21T08:27:02-05:00 I! Loaded outputs: sumologic
2023-03-21T08:27:02-05:00 I! Tags enabled: host=ipaddress user=sumomonitoring
2023-03-21T08:27:02-05:00 I! [agent] Config: Interval:10s, Quiet:false, Hostname:"ipaddress", Flush Interval:10s
2023-03-21T08:27:02-05:00 E! [telegraf] Error running agent: could not initialize input inputs.mysql: getting server 0 failed

System info

Telegraf 1.26.0, Ubuntu Server 22.04 LTS, AWS ec2 instance

Docker

No response

Steps to reproduce

1. Install Telegraf
2. Enable secretsstore by editing config file, un-commenting per documentation on github page (https://github.com/influxdata/telegraf/tree/master/config)
3. Store a secret, and verify it is written to secrets subfolder.
4. Run telegraf --password "enteryourpasswordtoallowaccess"
5. No error thrown here, but service does not start and the log shows as provided in this bug report
   ...

Expected behavior

Telegraf starts, with provided password to unlock secret store

Actual behavior

Telegraf command does not error out, however, the service does not start and error thrown in log

E! [telegraf] Error running agent: could not initialize input inputs.mysql: getting server 0 failed

Additional info

I was testing #12591 when I ran into this issue. DM'd Sven Rebhan as I was working with him prior, and was advised to open an issue.

[srebhan]

@jfrye121 can you please test the binary built by CI in PR #12919 once the tests finish!? Please let me know if this fixes your issue.

[jfrye121]

Telegraf 1.27.0-dbb17b69 (git: pull/12919@dbb17b69)

Installed per instructions, but I am now receiving the following error:

2023-03-27T09:12:50-05:00 E! [inputs.mysql] Error in plugin: dial tcp ipaddress:3306: connect: connection refused

No changes to config file, but here is the server string
[[inputs.mysql]] servers = ["{$user}:@{my_secrets:sumomonitoring}@tcp(ipaddress:3306)/?tls=false"]

Additionally, this is unrelated and more lack of knowledge...but when I check service telegraf status it shows that service is not running...however it clearly is and it can be seen in htop

HTOP

Service Status

I even tried removing the global tag for user and set to the following with the same results:

servers = ["SumoMonitoring:@{my_secrets:sumomonitoring}@tcp(ipaddress:3306)/?tls=false"]

[jfrye121]

To be safe, I made sure I could login with the credentials stored in the secrets manager, and I can manually. So..

mysql -u SumoMonitoring -p

I enter the password "password" and i am able to connect to the db instance. So I know the password is correct/

[srebhan]

Are you sure the PR binary is stored as /usr/bin/telegraf?!? Because this is what the service is running... Can you please try

# <path/to/telegraf/binary/from/PR>/telegraf -config /etc/telegraf/telegraf.conf -config-directory /etc/telegraf/telegraf.d --test --debug

and post the (redacted) output?

[jfrye121]

@srebhan

This is the package I used from the PR. I did not download, and extract the files from the tar.gz for linux to my ec2 instance. I understood the instructions that I could run the .deb file if I so preferred. I can copy the folders and overwrite with the tar.gz contents for linux if that is what you prefer.

[jfrye121]

Attaching file

[jfrye121]

outputREDACTED.txt

[srebhan]

@jfrye121 I do not see any error message... Did you maybe only cut out a part of the log? If so, can you please post the full log?

[jfrye121]

Let me rebuild the instance start from scratch with the build.

[jfrye121]

@srebhan It looks like when telegraf is attempting to connect to mariadb, the connection is refused. After enabling logging for mariadb, I was able to see that the full error is as follows:

2023-03-30 18:08:07 31 [Warning] Access denied for user 'telegraf'@'localhost' (using password: NO)

So it seems that the password is not being passed. I verified that I can manually login to mariadb using the same credentials and I am able to.

[[inputs.mysql]] servers = ["telegraf:@{my_secrets:telegraf2}@tcp(ipaddress:3306)/?tls=false"]

Ran telegraf --test --debug and here are the results: attached

[jfrye121]

debugresults033023.txt

[jfrye121]

dsn: telegraf:password@tcp(ipaddress:3306)/?timeout=5s&tls=false

telegraf is the correct user and the password is correct.

manually logged in using the following

mysql -u telegraf --password=password

or if I enter this way, same thing, works like a charm

mysql -u telegraf -p
Enter Password:password

Connection is a success, mysql client loads.

telegraf.log:

2023-04-03T15:37:50Z E! [inputs.mysql] Error in plugin: dial tcp ipaddress:3306: connect: connection refused

error.log

2023-04-03 15:40:26 34 [Warning] Access denied for user 'telegraf'@'localhost' (using password: YES)

@srebhan

[jfrye121]

Telegraf 1.27.0-4d208f27 (git: pull/12919@4d208f27)'

version output on ec2 instance

[jfrye121]

After a quick discussion with @srebhan , I looked into the mariadb config file and noticed that the server was binding to local host. I commented this out and telegraf was able to successfully connect with the user credentials. Closing issue! Thank you all for the help.