Frequent mqtt publish eats heap

[kmihaylov]

Hello.

I have a problem that is probably related to String.

If the code is executed each second, the heap gets exhausted in few minutes. About a kB each few tens of seconds.

If the timer is increased to two seconds, then the heap stays at almost constant value (currently at 38856 for few minutes already).

I think that I saw somewhere through the issues a similar problem, but can't recall the topic.

Here is the code.
PS: I tried to pass the strings by reference, but this doesn't change anything.
PS2: I tried to track down if some memory is allocated within the mqtt class and couldn't resolve how this code releases the memory. However it should work properly since my other projects work fine.

MqttClient.cpp

#define COPY_STRING(TO, FROM)                                                                                          \
	{                                                                                                                  \
		TO.length = FROM.length();                                                                                     \
		TO.data = (uint8_t*)malloc(FROM.length());                                                                     \
		if(!TO.data) {                                                                                                 \
			debug_e("Not enough memory");                                                                              \
			return false;                                                                                              \
		}                                                                                                              \
		memcpy(TO.data, FROM.c_str(), FROM.length());                                                                  \
	}

struct electricData {
	float V;
	float I;
	float P;
	uint32_t status;
	uint32_t conf;
};

void publishElectricity() {
	electricData elData = measureElectricity();

	String topic;
	//topic.reserve(1);

	String data;
	//data.reserve(5);

	topic = String( "P" );
	data = String( elData.P , 5 );
	mqttClient.pubMsgRef( topic , data );

	topic = String( "V" );
	data = String( elData.V , 4 );
	mqttClient.pubMsgRef( topic , data );

	topic = String( "I" );
	data = String( elData.I , 4 );
	mqttClient.pubMsgRef( topic , data );

	topic = String( "kWh" );
	data = String( kWh , 4 );
	mqttClient.pubMsgRef( topic , data );

	topic = String( "status" );
	data = String( elData.status );
	mqttClient.pubMsgRef( topic , data );

	topic = String( "conf" );
	data = String( elData.conf );
	mqttClient.pubMsgRef( topic , data );

	topic = String( "heap" );
	data = String( system_get_free_heap_size() );
	mqttClient.pubMsgRef( topic , data );

/*
	mqttClient.pubMsg( String("P") , String( elData.P , 5 ) );
	mqttClient.pubMsg( String("V") , String( elData.V , 4 ) );
	mqttClient.pubMsg( String("I") , String( elData.I , 4 ) );
	mqttClient.pubMsg( String("kWh") , String( kWh , 4 ) );
	mqttClient.pubMsg( String("status") , String( elData.status ) );
	mqttClient.pubMsg( String("conf") , String( elData.conf ) );
	mqttClient.pubMsg( String("heap") , String( system_get_free_heap_size() ) );
	*/
}

void mqttHelper::pubMsgRef( String& topic , const String& data ) {
	if(mqtt.getConnectionState() == eTCS_Connected) {
		topic = MqttConfig.DeviceTopic + topic;
		mqtt.publish( topic , data);
	}
	//auto reconnect?
}

At the end - heap is still at 38856...

[Vinimuller]

I have a similar problem but only when there is no internet. The getConnectionState function takes around 10 minutes to identify that the internet is down, so depending on the frequency that you publish your messages it will consume a lot of heap mem.

The upside of this is that when it identifies that there is no internet the heap memory is freed.

[slaff]

Sming is an async framework. The publishing of the message is not done immediately. If you push stuff into the internal buffer it will start to grow. Only when the TCP stack is ready to send data then part of that data will be delivered. And the buffer will decrease. If you fill the buffer faster then it is decreased (or emptied) then you will end up eating all available memory.

It would be probably better to send a message with QoS 1 or 2. Wait for the confirmation and once it arrives send the next message.

[kmihaylov]

What you mention makes sense, especially when I read about mqtt QoS levels.
Given that we are validating the connection state (mqtt.getConnectionState() == eTCS_Connected) is there any way to monitor the internal buffer (is it a lwip connection buffer, I mean to the specified host?)?

Also some ideas for tuning the TCP stack parameters? (Just for the record).