Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Workaround for If-Modified-Since HTTP request header causing cache corruption #665

Merged
merged 11 commits into from
Jan 18, 2020

Conversation

bitwiseman
Copy link
Member

@bitwiseman bitwiseman commented Jan 14, 2020

Overview

This PR shows a low level repro of an issue that has been plaguing Jenkins users for at least the last year.

EDIT: Similar to isaacs/github#692 but not the same

Key Takeaways

  • The root cause of this issue is NOT Jenkins. The issue is caused by a bug in OkHttp caching, GitHub server software, or the interaction between them.
    • In the short-term Jenkins will still need to make changes to mitigate this. What these steps will be is unknown.
  • This is an issue that can effect any GitHub query that returns a 404 why caching is turned on.
    • The existing work-around addresses the scenario that seems to hit Jenkins most often but it is possible there are any number of other places where this issue causes problem that we have not been able to reliably track down.

Basic Repro Steps

This basic process can be used for many queries that return 404. Using Ref creation as an example:

  1. Query CACHED connection: request git ref "refs/heads/does_not_exist" - File not found (expected)
  2. Create git ref "refs/heads/does_not_exist"
  3. Query CACHED connection: request git ref "refs/heads/does_not_exist" - File not found due erroneous GitHub 304 response (fail)

@bitwiseman bitwiseman force-pushed the task/cache-error-test branch from e29b1b2 to 0d9ec1c Compare January 14, 2020 08:28
@bitwiseman bitwiseman force-pushed the task/cache-error-test branch from 0d9ec1c to 72aedbb Compare January 14, 2020 08:29
//
// NOTE: This is even worse than you might think: 404 responses don't return an ETAG, but 304 responses do.
//
// Due erroneous 304 returned from "If-Modified-Since", the ETAG returned by the first 304
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@aglarendil
Copy link

@bitwiseman I would suggest looking into the logic of the github branch source plugin as well. The problem here as per https://issues.jenkins-ci.org/browse/JENKINS-60353 is that Jenkins checks the existence of the Jenkinsfile for the 'MERGE' head of a non-mergeable request, the one that does not exist a priori and then consequently marks the pr as dead/non-buildable.

@bitwiseman
Copy link
Member Author

@aglarendil
Yup, there is less than ideal behavior in the plugin (that's where my analysis to for this example came from). However, they would not be an issue if GitHub weren't silently behaving horribly.

This is a first cut at working round hub4j#669.  It is hacky as hell and I hate it.
This is much more reasonable way to address this issue.
When the Requester detects a 404 response with an ETag (only happpens when the server's 304
is bogus and would cause cache corruption), try the query again with new request header
that forces the server to not return 304 and return new data instead.

Ths solution is transparent to users of this library and autmatically fixes a situation that
was causing cache corruption. If GitHub ever fixes the issue and begins providing accurate
ETags to their 404 responses, this will result in two calls being made for each 404 response.
While that would be unfortunate, it would still be better than the current situation.
OkHttp2 doesn't invalidate caches sometimes when it probably should
@bitwiseman bitwiseman force-pushed the task/cache-error-test branch from 983c72b to 3479e4f Compare January 18, 2020 01:05
@bitwiseman bitwiseman changed the title JENKINS-54126 - Repro of github caching error Workaround for If-Modified-Since HTTP request header causing cache corruption Jan 18, 2020
@bitwiseman bitwiseman marked this pull request as ready for review January 18, 2020 01:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants