You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Now, the cron job runs each day.
It tries to collect all the data several times and it could fail.
If it fails, I have the notification about it and I have to go and investigate it manually.
Our SRE team silently revoked my rights some time ago by mistake, and I haven't received any notifications for some time (maybe half a year).
Some of our tables fully depends on the consistency of the data in the other tables. If the data is missing, the new computed data is wrong.
So, I'm no longer sure if the data is correct.
Now, if something goes wrong and noone investigated that, next day we will just compute next day statistics, and therefore we fill the tables with potentially broken data.
I want to create the separate table where I put the last successful datetime of each analytics table we have.
Each day we should at first read these values and try to fill again the missing parts. We also should not even try to fill the dependant tables if we can't compute all the data they need.
After that, we should clean all the data we have and re-compute it again to be sure that everything is consistent.
The text was updated successfully, but these errors were encountered:
Now, the cron job runs each day.
It tries to collect all the data several times and it could fail.
If it fails, I have the notification about it and I have to go and investigate it manually.
Our SRE team silently revoked my rights some time ago by mistake, and I haven't received any notifications for some time (maybe half a year).
Some of our tables fully depends on the consistency of the data in the other tables. If the data is missing, the new computed data is wrong.
So, I'm no longer sure if the data is correct.
Now, if something goes wrong and noone investigated that, next day we will just compute next day statistics, and therefore we fill the tables with potentially broken data.
I want to create the separate table where I put the last successful datetime of each analytics table we have.
Each day we should at first read these values and try to fill again the missing parts. We also should not even try to fill the dependant tables if we can't compute all the data they need.
After that, we should clean all the data we have and re-compute it again to be sure that everything is consistent.
The text was updated successfully, but these errors were encountered: