Cohort analytics

[neilisfragile]

Create a new table to store daily user visit information which can be used in cohort analysis.

Rather than insert on each user action, instead slurp the user_ips table daily.

Not sure how best to test this - any pointers?

[neilisfragile]

This is probably okay? If not I can fetchone on each iteration of the subsequent loop

[neilisfragile]

unrelated to this PR just want to ignore Atom python ide

[erikjohnston]

I would be tempted to work out the active users for today as you go rather than yesterday. I.e. every hour running something like:

INSERT INTO user_daily_visits (user_id, device_id, timestamp)
SELECT user_id, device_id, $today
FROM user_ips AS u
LEFT JOIN user_daily_visits USING (user_id, device_id)
WHERE last_seen > $today AND timestamp IS NULL

(The join to ensure that we don't insert duplicate values)

[neilisfragile]

Note change in SQL to handle case of duplicates on insert (where there are multiple entries that did not match previous entries in user_daily_visits)

[erikjohnston]

Looks good! I'd just switch around when you set self._last_user_visit_update, but otherwise is good

[erikjohnston]

You can just use self.clock.time_msec()

[erikjohnston]

In fact, You probably want to get this before the query, so you do:

now = self.clock.time_msec()
txn.execute(sql, (
    today_start, today_start,
    self._last_user_visit_update,
    now,
))
self._last_user_visit_update = now

This ensures there isn't going to be any overlaps or missed updates.

[erikjohnston]

Do you mean:

it is important that the query is run often...

[erikjohnston]

(The identing style we use tends to not use this continuation indent. See other comment for how it normally looks)