Load non-HTML resources directly whenever possible #583
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- drop initial HEAD check (and obsolete 'agent' params to fetch, no longer used) - load cookies for each page for direct fetch - attempt direct fetch GET request on every page with cookies + correct user-agent - abort direct fetch if response is HTML, and then load in browser, otherwise proceed with direct fetch - ensure direct fetch timestamp is set correctly, populated in pageinfo
- handle direct fetch redirects via additional fetching - use manual redirect mode for AsyncFetcher - fallback to browser for error responses, just in case
… redirect records
…erialized even if need to redo via browser
tw4l
approved these changes
May 24, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Optimize the direct loading of non-HTML pages. Currently, the behavior is:
This changes the behavior to:
Also:
If fetch request results in HTML, the response is aborted and browser loading is used.
Note: initially attempted to handle redirects, but gets a bit complicated since many need to follow the redirect check to determine if request is non-HTML, and would need to buffer those responses, and not write them in case they need to be dropped in favor of browser loading. Erring on side of caution of requiring 4xx/5xx responses to still be loaded through browser, just in case.
Other optimization: it is possible to turn on http/2 loading for fetch, as mentioned in: nodejs/undici#2750 (comment) (the obsolete Agent setup from http/https modules has been removed)