Ignore duplicate MQTT message(s) when iotagent-ul in HA

[nmatsui]

this pull request fixes issue #151

When iotagent-ul in a High Availability architecture (active - active), a MQTT message is processed as many times as the number of iotagent-ul . Because the each iotagent-ul subscribes for the same topic of MQTT Broker, so a MQTT message published that topic is processed by each iotagent-ul individually.

This pull request fixes above problem.

1. When a MQTT message is received, each iotagent-ul calls a RESTAPI before processing the message.
   • The url of its RESTAPI is configured by config.js.
2. If its RESTAPI returns 200 OK, iotagent-ul continues processing the MQTT message as ordinary. But if its RESTAPI returns 409 Conflict, iotagent-ul stops processing.

This pull request assumes that the "RESTAPI" service to check message duplication is outside of iotagent-ul.

An example implementation is like below:
https://github.com/tech-sketch/fiware-mqtt-msgfilter

[dcalvoalonso]

@nmatsui , have you thought about the solution proposed by @AlvaroVega in #151 (comment)?

[mrutid]

First of all, thanks for your contribution to a complex problem. Currently we have another approach to this issue by changing the MQTT broker with a RabbitMQ. In this way Rabbit expose MQTT topics/channels to the devices, while offers an integrated AMQP queue to the Agents, so the messages get dispatched instead of distributed:

DEVICE <-> MQTT<->AMQP<->Agents

(in fact in order to make RabbitMQ(+MQTTplugin) identical to Mosquitto-MQTT we have implemented this Erlang plug https://github.com/telefonicaid/rabbit-acl-tool that should be installed).

https://github.com/telefonicaid/iotagent-ul/blob/3b53b8ac2135aa203fd48e0a6d1f554f23714636/docs/usermanual.md#amqp-binding

Your solution looks good, but it implies to deploy a new service... and in the real/producction world every new service has an operation cost (no matters how simple the service really is). Would you mind to check the RabbitMQ option? we know docs may be a bit confusing and probably the best contribution could be empower docs ant simplify if possible that part.

Arigato gozaimasu :)

[nmatsui]

Thanks for your comments, @dcalvoalonso and @mrutid . I'm bent on how to use MQTT Broker, but I did not think other solutions.
Thank you for your information, and I'll try to use RabbitMQ cluster in order to construct HA architecture.

[nmatsui]

@mrutid I tried the below environment. And I found that no message duplicate ! Thank you for your informations.

"DEVICE <-> (MQTT) <-> RabbitMQ cluster [3 node] <-> (AMQP) <-> iotagent-ul [3 node] <-> (NGSI/HTTP) <-> orion [3 node]"

[perrolobo]

Great news! Grats!

…

On Thu, Aug 30, 2018 at 1:08 PM Nobuyuki Matsui ***@***.***> wrote: @mrutid <https://github.com/mrutid> I tried the below environment. And I found that no message duplicate ! Thank you for your informations. "DEVICE <-> (MQTT) <-> RabbitMQ cluster [3 node] <-> (AMQP) <-> iotagent-ul [3 node] <-> (NGSI/HTTP) <-> orion [3 node]" — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#279 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA27Uhs2SYZxH3gWQe4wVL9hFj84_WFOks5uV8edgaJpZM4WBKPa> .

-- - Marcos -