fix(mqtt): rework connection and message tracking

[kitlaan]

My mqtt connection would randomly stall, or disconnect (without any logging on the telegraf side).
It turns out there's was a comedy of bugs all adding up to trigger the behavior.

1. Weird connection/disconnection behavior

On first glance, it seemed that telegraf would randomly disconnect and reconnect, just to disconnect again. That led to looking at paho's client management and finding that the plugin really should be making a new client after disconnect (specifically since the plugin is not using auto reconnect).

Reusing a Client is not completely safe. After calling Disconnect please create a new Client (NewClient()) rather than attempting to reuse the existing one (note that features such as SetAutoReconnect mean this is rarely necessary).

2. Weird data loss (deadlock?)

Through code inspection, I noticed #10687. Sadly, fixing this did not resolve the problem. But...

3. Metric tracking

As part of testing #10684 I noticed that the m.acc and m.sem channels were slowly-but-steadily growing larger over time. With the dedup bug fixed, both channels did not drain to zero as expected. It turns out that select has random behavior when both cases are ready. Since one path exits the for-loop, on average this causes m.acc to not process, and thus it fills up.

TL;DR

So all-together, this change both cleans up the client connection semantics, as well as cleaning up metric tracking.

Required for all PRs:

• Updated associated README.md.
• Wrote appropriate unit tests.
• Pull request title or commits are in conventional commit format

resolves #10687

[telegraf-tiger]

Download PR build artifacts for linux_amd64.tar.gz, darwin_amd64.tar.gz, and windows_amd64.zip.
Downloads for additional architectures and packages are available below.

👍 This pull request doesn't change the Telegraf binary size

📦 Click here to get additional PR build artifacts

Artifact URLs

DEB	RPM	TAR GZ	ZIP
amd64.deb	aarch64.rpm	darwin_amd64.tar.gz	windows_amd64.zip
arm64.deb	armel.rpm	darwin_arm64.tar.gz	windows_i386.zip
armel.deb	armv6hl.rpm	freebsd_amd64.tar.gz
armhf.deb	i386.rpm	freebsd_armv7.tar.gz
i386.deb	ppc64le.rpm	freebsd_i386.tar.gz
mips.deb	riscv64.rpm	linux_amd64.tar.gz
mipsel.deb	s390x.rpm	linux_arm64.tar.gz
ppc64el.deb	x86_64.rpm	linux_armel.tar.gz
riscv64.deb		linux_armhf.tar.gz
s390x.deb		linux_i386.tar.gz
		linux_mips.tar.gz
		linux_mipsel.tar.gz
		linux_ppc64le.tar.gz
		linux_riscv64.tar.gz
		linux_s390x.tar.gz
		static_linux_amd64.tar.gz