fast way to notice that a "load of events" has been processed (as indicator for sending)

[bernhardreiter]

A number of feeds come as one block, e.g. once a day. (This means that will have the same time.observation value in intelmq.)

Each recipient wants to get one aggregrated email with all notification for this block as fast and complete as possible.

technical implementation thoughts

@bernhard-herzog has implemented a way to notice that when a directive was inserted the last time for a specific set of aggregation values, so when used to aggregate on time.observation and max(d.inserted_at) is 2 hours ago, we trigger sending the email.

This methods has the drawback that if the first event and the last event of one load (or batch) is for this set of aggregation values, it will take a long time before another directive is entering the database, so the time intervall has to be quite long to have a good detection that processing has been through.

This issue is about using a better detection mechanism, that can detect the completion of processing faster, thus sending emails faster on the average.

implementation idea

Using an extra table that for each feed.name and time.observation keeps that last inserted directive time. This way the email aggregation script can use a simple additional query to see with a higher reliability that the batch has been processed fully.

Necessary implementation steps (roughly):

• extend trigger to enter information in the new table
• add new table to db schema
• extend mail generation to use the new information
• provide a way to deal with old pending notifications for the migration to the new system