Stored Request Cache Bug?

[dbemiller]

The other day, I saw some weird behavior.

@anwzhang sent me an HTTP payload with Stored Request IDs to reproduce an issue she was seeing. I didn't see the same behavior locally. I retried the request several times, and always got back the same result. After a meeting, I tried it again and got a different error. Again, the result was consistent. Later that day, I tried it again and got the result she expected.

Judging by the symptoms, I think this is a bug in the Stored Request cache. Unfortunately I don't remember exactly what the response was on the first call for each of these cases... but with local dev & remote database, it's very common that the first request to the DB times out, and subsequent ones work fine.

I think what's happening is that the call to the database returns some corrupt data, which we then save to the cache. Depending on exactly when the request times out, we can see different behavior for the same request.

[dbemiller]

I found these issues, which may also be related: golang/go#23519 and lib/pq#706

[dbemiller]

As a workaround for now, I think we should only send back stored request data if all of it comes back error-free.

One botched request isn't a big deal... as long as we don't cache bad response data. And if we can only resolve "some" of the stored request data, then it still probably won't make a valid auction request anyway.