Record failed jobs when saving the result fails (dont retry) #28629
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
joelgriffith
assigned chrisdavies and tsullivan and unassigned chrisdavies and tsullivan
💚 Build Succeeded |
OMG 🙈 :D |
I agree with this as well |
|
Added a test for this corner-case, should be good to go now |
joelgriffith
added
review
(Deprecated) Feature:Reporting
💔 Build Failed |
💔 Build Failed |
💔 Build Failed |
|
retest |
💔 Build Failed |
|
I'm unclear as to why this job is hanging, but going to try and get it figured out + merged |
💔 Build Failed |
💔 Build Failed |
💚 Build Succeeded |
joelgriffith
commented
Jan 23, 2019
| const failStub = getFailStub(worker); | ||
|
|
||
| await worker._performJob(job); | ||
| worker.destroy(); |
There was a problem hiding this comment.
I think this was the missing piece.
💚 Build Succeeded |
|
Backport: #29211 |
joelgriffith
added a commit
that referenced
this pull request
Jan 23, 2019
|
Backported! #29211 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
This was a fun one to dig up, for only a one-line PR :/
The TL;DR is that ES-QUEUE was swallowing errors when the final output of the work was saved back into ES. Even more precarious is when content-bodies are bigger than ES is allowed (set via
http.max_content_length), and those errors couldn't be saved since the error message included the output body and that failed to save since it was too big (again).Anywho, this marks those issues as failed immediately and save the response without re-trying. We could fork the retry-action based upon the HTTP code, but that feels pre-carious and has my spidey-sense tingling. I think it's better to fail quickly here and let folks know vs reporting retrying again and again for the same outcome.
I'm currently trying to test this in a spec, but it's turning into a rabbit-hole given the lengthy promise chain, so I wanted to gather feedback before spending more time on it!
Checklist
Use
strikethroughsto remove checklist items you don't feel are applicable to this PR.[ ] This was checked for cross-browser compatibility, including a check against IE11[ ] Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support[ ] Documentation was added for features that require explanation or tutorials[ ] Unit or functional tests were updated or added to match the most common scenarios[ ] This was checked for keyboard-only and screenreader accessibility