Bug 1656934 - Scan pending pings directories after dealing with upload status #1205
Conversation
| // If upload is disabled, we delete all pending pings files | ||
| // and we need to do that **before** scanning the pending pings folder | ||
| // to ensure we don't enqueue pings before their files are deleted. | ||
| let _scanning_thread = glean.upload_manager.scan_pending_pings_directories(); |
There was a problem hiding this comment.
Eugh. So this was an actual bug and not a test error. We should add a changelog entry. Do we have any way to estimate how frequently we sent pings out?
I'd suspect that's not super frequent, but it's important to better understand. We could also draft a PSA email, if we find the impact being big.
There was a problem hiding this comment.
Do we have any way to estimate how frequently we sent pings out?
Hm, I need to think about this. When we have this bug it is usually followed by an error on deleting the ping file. Because we send the request to the caller -> it uploads succesfully (usually) -> and then it tries to delete an already deleted file, which logs an error. But that is all we have, from what I remember on the top of my head.
I'll think a bit further, if you have any ideas, let me know.
There was a problem hiding this comment.
I have an idea to validate this: check if any pings are sent from a client_id after this client_id sent a deletion-request ping. Will come back to this thread with results when I have them.
There was a problem hiding this comment.
BONUS: add CHANGEOG.md entry
| upload_manager.set_rate_limiter( | ||
| /* seconds per interval */ 60, /* max tasks per interval */ 15, | ||
| ); | ||
|
|
||
| // We only scan the pending ping sdirectories when calling this from a subprocess, |
There was a problem hiding this comment.
Having this makes me question the name of this function. If you have better ideas I'd be up for changing it.
| upload_manager.set_rate_limiter( | ||
| /* seconds per interval */ 60, /* max tasks per interval */ 15, | ||
| ); | ||
|
|
||
| // We only scan the pending ping sdirectories when calling this from a subprocess, |
There was a problem hiding this comment.
Oops: pending ping sdirectories
The bug that was causing that intermittent, was due to scanning the pings directories before running the
on_upload_disabledfunction that deleted non-deletion request pings.I fix that in this PR by moving the scanning to after function the
on_upload_disabledfunction.As I mentioned in the bug, I was able to reproduce the intermittent reliably in a specific branch of mine, with the fix I don't get the errors anymore on that branch either.