MQTT client does not connect to MQTT broker after a TCP reconnection

[DiKaY1969]

Hello,

I would like to report a BUG in the MQTT client of the SML reader which means that, for example after restarting the router, no more MQTT data is received from the MQTT broker. The SML reader can be reached via TCP, but the MQTT client no longer establishes the connection to the broker properly after the WLAN reconnects.

A restart of the SMLReader leads to success

Suggestion, simply close the MQTT client after the WLAN reconnects and reconnect if this is not already done?

Thanks in advance.

[DiKaY1969]

Hello,
I use the digitizer (project from https://github.com/jomjol/AI-on-the-edge-device) on an ESP32 - to record the water meter reading and transmit it to home automation.

But this module automatically reconnects to the MQTT broker after rebooting my router after about 2-5 minutes.

It is not necessary to intervene manually.

Maybe you can look at the code and install the same solution there?

Many Thanks.

[mruettgers]

Hey,

Thanks for reporting this issue.
I'll have a look at it next week.

[hamster65]

I ran into this today (again, it happend several times now). I did not look at the code, yet, but for my own projects, I usually do a simple connectivity check (wifi and mqtt) every 10 seconds or so. The libraries provide functions for that. I then either call a "reconnect function" or (if the project does not work without a connection) a esp reset.

[hamster65]

Happend again today after wifi was down because of a router reboot.

I thought I'd try a quick and dirty fix and add publisher.connect(); before yield() in the main loop.
But I cannot compile. Is there anything to set up?

src/main.cpp: In function 'void loop()':
src/main.cpp:117:74: error: 'class std::list<Sensor*>' has no member named 'eplatformio'
for (std::list<Sensor*>::iterator it = sensors->begin(); it != sensors->eplatformio-ide.useBuiltinPIOCorend(); ++it){
^
src/main.cpp:117:86: error: 'ide' was not declared in this scope
for (std::list<Sensor*>::iterator it = sensors->begin(); it != sensors->eplatformio-ide.useBuiltinPIOCorend(); ++it){

[mruettgers]

The problem is that when the connection is lost, the captive portal is activated and IotWebConf (the library I use for WiFi and configuration management) does not reconnect. But this seems to be fixed with the current version of IotWebConf. I will release a new version of SMLReader in the next few days, which will include an update of IotWebConf.

[mruettgers]

I have already released an upgraded version on https://github.com/mruettgers/SMLReader/tree/feature/update-iotwebconf, but haven't had the time to test it thoroughly yet.

You can find a binary compiled with this changes attached, so feel free to test ;-)
SMLReader_v3.0.0_beta.bin.zip

[mruettgers]

Hm. There still seems to be an issue with the PangolinMQTT library not working after a WiFi reconnect (even if I call the connect() method again).

[DiKaY1969]

Sorry, I've been on vacation for a few days.

Possibly you should also execute a disconnect() methode before connect() if it is still connected.

However, the debug output should provide information in our error case.

Thank you for your help.

[mruettgers]

I already tried to use disconnect() during my tests and it didn't change anything.

So I need to debug the third party libs a bit more.

[DiKaY1969]

PangolinMQTT comes with some built-in diagnostics.
These can be controlled by setting the debug level in config.h

/*
Debug levels:
0 - No debug messages
1 - connection / disconnection messages
2 - level 1 + MQTT packet types
3 - level 2 + MQTT packet data
4 - everything
*/

#define PANGO_DEBUG 4

[hamster65]

Going to the http config page and hitting "apply" (after filling out the ap and wifi passwords restores the mqtt connection.

[mruettgers]

Sorry, I did not have the time to look at this issue again yet. But what you say makes sense as the ESP is being rebooted after changing the config.

[stepk71]

I have seen the same problem in my setup using the SMLReader release version.

In my case the device successfully connected to wifi, but the mqtt client did not reconnect. I have noticed in the log of my router that the wifi reconnect took 70 seconds. 30 seconds is the default wifi timeout in IotWebCon v2, afterwards it jumps for 30 seconds into AP mode, before it starts to reconnect.

# define IOTWEBCONF_DEFAULT_WIFI_CONNECTION_TIMEOUT_MS 30000

[panhans]

I have seen the same problem in my setup using the SMLReader release version.

In my case the device successfully connected to wifi, but the mqtt client did not reconnect. I have noticed in the log of my router that the wifi reconnect took 70 seconds. 30 seconds is the default wifi timeout in IotWebCon v2, afterwards it jumps for 30 seconds into AP mode, before it starts to reconnect.

# define IOTWEBCONF_DEFAULT_WIFI_CONNECTION_TIMEOUT_MS 30000

I have the same disconnect issue. Could you solve it by simply increasing the timeout?

[stepk71]

I have seen the unit survived now multiple disconnects >30 seconds. But yesterday I had the problem again: The WIFI was reconnected after 10 minutes. I have no idea why this module gets disconnected for 10 minutes, it is located close to the router ...

By the way, I added `iotWebConf.setWifiConnectionTimeoutMs( 300000l );

Seems the issue is known in PangolinMQTT: philbowles/PangolinMQTT#40
I hope PangolinMQTT V3 fixes this issue.

[panhans]

Ok, good to know. Thanks for your investigation. ;)
Meanwhile I go with tasmota. I managed to read out the values of my smart meter and send it every second to my mqtt broker.

[borsti67]

Is the "beta" version still worth a try, if the disconnected phase is not too long? My FritzBox and -Repeaters also tend to reconnect in random intervals, so SMLReader gets offline.
Is there a possibility to send a "reset" command otherwise? I could trigger an action in FHEM if the offline status gets detected.

[mruettgers]

I have replaced PangolinMQTT with https://github.com/marvinroger/async-mqtt-client/.
Can you test if the reconnect is working now in the develop branch?

I also added a reset API endpoint (http://IP/reset) that can be used to restart the ESP.

[mruettgers]

I forgot to mention that the WiFi and MQTT configuration might need to be redone since I also updated IoTWebConf.

[borsti67]

Is there a way to read out the settings that were used to flash the actual version?
I would like to give it a try but my device came already functional (and I never compiled and flashed an ESP before). I don't want to break it...

[DiKaY1969]

Hello Martin and thank you for your effort.
I would like to try the code with the new MQTT clinet.
But, I'm using a fork of the SMLReader project running on an ESP8285.
https://github.com/esplesekopf
Unfortunately I am not able to adapt and compile the code for the 8285.

[mruettgers]

Is there a way to read out the settings that were used to flash the actual version? I would like to give it a try but my device came already functional (and I never compiled and flashed an ESP before). I don't want to break it...

The web interface will show you the basic settings (except the passwords).

But you can make a backup of your ESP with esptool:

For 1M flash:
Backup:
python esptool.py -b 115200 --port COM3 read_flash 0x000000 0x100000 flash_1M.bin
Restore:
python esptool.py -b 115200 --port COM3 write_flash 0x000000 flash_1M.bin

For 4M flash:
Backup:
python esptool.py -b 115200 --port COM3 read_flash 0x00000 0x400000 flash_4M.bin
Restore:
python esptool.py -b 115200 --port COM3 write_flash 0x000000 flash_4M.bin

[DiKaY1969]

Hello Martin and thank you for your effort. I would like to try the code with the new MQTT clinet. But, I'm using a fork of the SMLReader project running on an ESP8285. https://github.com/esplesekopf Unfortunately I am not able to adapt and compile the code for the 8285.

Hello Martin

I adapted the source code (PlatformIO.ini and config.h), compiled it and flashed it into the SMLReader (ESPLesekopf-Project).

The settings for the AP and MQTT broker are adopted as before.

1. The SMLReader now always connects to the MQTT Broker, even if the SMLReader or the AP is restarted several times!
   In the topic: LWT, the online/offline status is always updated correctly.
   The welcome string is displayed in the topic: Info.
   However, I am not getting any more data from my meter.
   Possibly something is still wrong here with the configuration of the sensor input!?

2. The new reset URL works but
   However, this is unfortunately not password-protected, which should be improved!
   In the fork: ESPLesekopf, Thomas has already integrated this with the transfer of the user and password.
   Example: http://admin:PASSWORD@IP/reset

[mruettgers]

Hey DiKaY1969

1. The SMLReader now always connects to the MQTT Broker, even if the SMLReader or the AP is restarted several times!
   In the topic: LWT, the online/offline status is always updated correctly.
   The welcome string is displayed in the topic: Info.
   However, I am not getting any more data from my meter.
   Possibly something is still wrong here with the configuration of the sensor input!?

It looks like you have to change the GPIO pin of the reader to 14 (default is D2) for use with the ESPLesekopf, see https://github.com/esplesekopf/SMLReader/blob/develop/src/config.h#L17.

2. The new reset URL works but
   However, this is unfortunately not password-protected, which should be improved!
   In the fork: ESPLesekopf, Thomas has already integrated this with the transfer of the user and password.
   Example: http://admin:PASSWORD@ip/reset

I just had a quick look at the code of IotWebConf and there does not seem to be an easy way to access the stored password easily. I'll come back to this issue sooner or later when I have more time.

Michael ;-)

[DiKaY1969]

Hey DiKaY1969

1. The SMLReader now always connects to the MQTT Broker, even if the SMLReader or the AP is restarted several times!
   In the topic: LWT, the online/offline status is always updated correctly.
   The welcome string is displayed in the topic: Info.
   However, I am not getting any more data from my meter.
   Possibly something is still wrong here with the configuration of the sensor input!?

It looks like you have to change the GPIO pin of the reader to 14 (default is D2) for use with the ESPLesekopf, see https://github.com/esplesekopf/SMLReader/blob/develop/src/config.h#L17.

2. The new reset URL works but
   However, this is unfortunately not password-protected, which should be improved!
   In the fork: ESPLesekopf, Thomas has already integrated this with the transfer of the user and password.
   Example: http://admin:PASSWORD@ip/reset

I just had a quick look at the code of IotWebConf and there does not seem to be an easy way to access the stored password easily. I'll come back to this issue sooner or later when I have more time.

Michael ;-)

Hi Michael,

yes I had configured the GPIO pin to 14.

But it still doesn't work :-(
Thomas from the "ESPLesekopf project" will take a look and then publish a compiled version.

Thomas' original project, based on SMLReader 2.2.1, works perfectly except for the MQTT problem.

Thanks anyway for your help.

greeting
Dierk

[ggrote]

I have replaced PangolinMQTT with https://github.com/marvinroger/async-mqtt-client/. Can you test if the reconnect is working now in the develop branch?

Could you please provide a precompiled binary? Thanks in advance.

[Stilzchen]

I had the same issue multiple times with version 2.2.0.
Yesterday I compiled version 3.0.0 via PlatformIO with D5 as GPIO pin of the phototransistor.
Works well until now. The first WIFI reconnect after powercycling my router was handled successfully. WIFI reconnect and TCP reconnect worked.
I will report how it works after some more reconnects.

What is verry helpful for me is the new reset link.
With the help of that I can monitor the last message input and can restart the D1mini remotely if last message is to old.
Thanks a lot for this, @mruettgers.

Stilzchen

[Stilzchen]

I have replaced PangolinMQTT with https://github.com/marvinroger/async-mqtt-client/. Can you test if the reconnect is working now in the develop branch?

Could you please provide a precompiled binary? Thanks in advance.

Feel free to test with this files.
3 versions included: Pin D2 with and without led and Pin D5 without led. I tested only the last one.
All 3 versions come with normal, debug and dev build.
smlreader_300.zip

EDIT: interval is set to 30 (more than enough for me)

[hamster65]

I flashed D2 with LED. So far, it is connected, but it updates the current power (W) only every 31 seconds. Is this intentional?

[mruettgers]

I'll release a new version including this fix and with the default values (configured for D2 and with data published in realtime) this evening.

[hamster65]

Cool, thanks! I have seen requests to change the publish rate. Maybe this could be added as an option. I prefer realtime updates because I display them on a screen and I also use that to switch and regulate heaters depending on PV power.

[ggrote]

Feel free to test with this files. 3 versions included: Pin D2 with and without led and Pin D5 without led. I tested only the last one. All 3 versions come with normal, debug and dev build. smlreader_300.zip

EDIT: interval is set to 30 (more than enough for me)

Thanks. I just compiled a version on my own yesterday with an interval of 15 min (had to change the interval datatype to uint16 for that to work. I'm only interested in the kWh not the watts. :)

[mruettgers]

https://github.com/mruettgers/SMLReader/releases/tag/2.3.0