Notification flow control on update

[fgalan]

Sometimes (especially with batch updates, i.e. POST /v2/op/update) a single update may trigger a large set of notifications which take some time to be processed (e.g. due to the receiver doesn't responses immediately). If the sender of the updates goes faster than notification processing (due to Orion responses immediately to the update sender, without waiting for notification finalization) this can cause saturation problems (typically, the notification queue gets full and new notifications are discarded). Thus, some flow control mechanism is needed.

Let's consider a flow control algorithm with depends on three parameters: notifications coefficient (C) from 0 to 1, evaluation step (S) in milliseconds, maximum time (M) in milliseconds.

The algorith is as follows:

1. When start to processing the update, get the current value of the notification queue. Let's name that value Q_0
2. During the processing of the update, count how many subscriptions are triggered (i.e. how many subscriptions are added to the notification queue). Let's name that value N.
3. Now that the processing as ended:
   1. Gets the new value of the queue: Q_i
   2. If Q_i <= Q_0 +(1 - C) * N then go to 4. Else wait for form S milliseconds and continue to 3.iii.
      • Note that if C = 1 corresponds to highest flow control. In this case: Q_i <= Q_0, that is: the size of the queue as large as we found it at the start of the update.
      • Note that If C = 0 corresponds to lowest flow control. In this case: Q_i <= Q_0 + N, that is: we won't want to wait for any of the notifications to be processed (regarding the initial size of the queue at the start of the update).
   3. If the cumulative wait time is more than M, then go to 4. Otherwise, go back to 3.i (a value of 0 for M could mean always go to 3.1).
4. Orion response to the client of the update

Additional notes:

• The client must explicitly require flow control. This could be done with an NGSIv2 option, e.g. ?options=flowControl.
• The flow control option could be applied to any NGSIv2 update operation. Note that although the one that would take more advantage of this new functionality is batch update even a single-entity update could trigger several notification in that entity has several matching subscriptions. From an implementation point of view, it doesn't involve an extra cost (as all updates are using the same core logic at mongoBackend at the end).
• A CLI will be used to enable flow control and define its parameters. For instance: -flowControl C:S:M. If this CLI is not active, ?options=flowControl doesn't have any effect.
• Flow control needs the notification queue so it will be allowed only when threadpool notification mode is used.
• Would it make sense allow C > 1? This will mean that the algorithms ends when the size of the queue is smaller than the size at the moment of starting processing the udpate. With a high enough value of C, we can rearch the condition Q_0 <=0, so the algorithm will end only when the queue is empty.

[mrutid]

OK

[fgalan]

Implemented by PR #3577