[feature] Precalculated presence stats counters in store

[Kachit]

At the moment when we use presence_stats method under the hood executes method presence and calculate stats on the fly. But we can store presence stats counters by channels in store and change it after join/jeave events. After that presence_stats method speed will be increased.

[FZambia]

Hello, @Kachit

If I understand correctly you mean Redis Engine. This is a good idea, I think there was even TODO about it somewhere in the code – but can't quickly find. Will try to implement this at some point, or if you decide to send a PR - please warn when start working on this.

[Kachit]

For example yes - Redis Engine. Better way implement it to another storage engines.

[FZambia]

Memory engine is fast already, Tarantool calculates counters on database side - so should be very fast too. Redis Engine loads full presence to calculate on Go side, so definitely will benefit. I suppose number of clients may be straightforward (but who knows), but for for number of unique users - need to think what is the best Redis structure to precalculate it. If you have ideas on structures to use - please share.

[FZambia]

@Kachit hello, in centrifugal/centrifuge#343 I've made some changes which improve performance of presence stats. It's implemented in a way that Centrifugo will maintain additional per-user data structures in Redis. Similar to structures used for general client presence (ZSET + HASH). So we get a possibility to efficiently get the number of unique users.

For the channel with 100k unique subscribers number of ops bumped from 15 to 200000 per second in my benchmark.

This will increase memory usage in Redis, possibly up to 2x from what was spent on presence information before (less if you have extra info in client information attached, since we do not include info payload to user structures). To not increase memory usage unexpectedly for others need to introduce some option in Centrifugo, like enable_redis_presence_user_mapping or sth.

I also re-read your original proposal and realized that you've meant a bit different implementation – making incr/decr ops for two counters upon join/leave events. I don't see a way to implement it in such a way, and keeping eventual consistency property after edge/failure scenarios, and without additional data structures. Also, join/leave events may be completely disabled while presence enabled for channels.

What do you think - does it make sense for your use case?

[Kachit]

Hello @FZambia good job! I think this feature must be optional and regulated by some config flags because increased redis memory can create some problems in current projects which use centrifugo.
For us it is good because we can fetch presence stats from channel without redis lock.
But I must warn you. Don`t release this feature in friday before holydays ))

[FZambia]

Possibility to improve presence stats is part of v5.2.1 release. Closing - but would be nice to get some feedback here as you test it out @Kachit . Thx for raising!

[Kachit]

Good job. But I have a some question. You told about increase memory usage in Redis with this improve. What about CPU load stats ? Is it same or changing ?

[FZambia]

What about CPU load stats ? Is it same or changing

On Redis side may increase in general because presence updates involve couple of new calls to add data to new structures, I believe not dramatically though until presence add calls are a hot path in your case. But for the moments you called presence stats – it must improve. On Centrifugo side may decrease significantly since much less work required for calculations - but again, if this is not a hot path in your case then you may not even notice