Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

std.http.Client: support HTTP redirects #14202

Merged
merged 7 commits into from
Jan 6, 2023
Merged

std.http.Client: support HTTP redirects #14202

merged 7 commits into from
Jan 6, 2023

Conversation

andrewrk
Copy link
Member

@andrewrk andrewrk commented Jan 5, 2023

  • std.http.Status.Class: add a "nonstandard" enum tag. Instead of
    having class return an optional value, it can potentially return
    nonstandard.
  • extract out std.http.Client.Connection from std.http.Client.Request
    • this code abstracts over plain/TLS only
    • this is the type that will potentially be stored in a client's LRU
      connection map
  • introduce two-staged HTTP header parsing
    • API users can rely on a heap-allocated buffer with a maximum limit,
      which defaults to 16 KB, or they can provide a static buffer that
      is borrowed by the Request instance.
    • The entire HTTP header is buffered because there are strings in
      there and they must be accessed later, such as with the case of
      HTTP redirects.
    • When buffering the HTTP header, the parser only looks for the
      \r\n\r\n pattern. Further validation is done later.
    • After the full HTTP header is buffered, it is parsed into
      components such as Content-Length and Location.
  • HTTP redirects are handled, with a maximum redirect count option that
    defaults to 3.
    • Connection: close is always used for now; implementing keep-alive
      connections and an LRU connection pool in std.http.Client is a task
      for another day.

closes #2007. we can open separate issues for additional enhancements to the std lib http client.

There's no reason for this to ever run at runtime; it should always be
used to generate a constant.
 * std.http.Status.Class: add a "nonstandard" enum tag. Instead of
   having `class` return an optional value, it can potentially return
   nonstandard.
 * extract out std.http.Client.Connection from std.http.Client.Request
   - this code abstracts over plain/TLS only
   - this is the type that will potentially be stored in a client's LRU
     connection map
 * introduce two-staged HTTP header parsing
   - API users can rely on a heap-allocated buffer with a maximum limit,
     which defaults to 16 KB, or they can provide a static buffer that
     is borrowed by the Request instance.
   - The entire HTTP header is buffered because there are strings in
     there and they must be accessed later, such as with the case of
     HTTP redirects.
   - When buffering the HTTP header, the parser only looks for the
     \r\n\r\n pattern. Further validation is done later.
   - After the full HTTP header is buffered, it is parsed into
     components such as Content-Length and Location.
 * HTTP redirects are handled, with a maximum redirect count option that
   defaults to 3.
   - Connection: close is always used for now; implementing keep-alive
     connections and an LRU connection pool in std.http.Client is a task
     for another day.

see #2007
@andrewrk
Copy link
Member Author

andrewrk commented Jan 5, 2023

Is this fancy vector stuff better than the naive version? godbolt link

Unclear. I better measure some perf.

$ lscpu | grep Model
Model name:                      Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz
Model:                           158
$ stage3/bin/zig run bench.zig -OReleaseFast 
             noop:    1374170 MiB/s
            naive:       1094 MiB/s
            fancy:       5082 MiB/s

$ stage3/bin/zig run bench.zig -OReleaseFast -mcpu=baseline
             noop:    1288749 MiB/s
            naive:        859 MiB/s
            fancy:       5743 MiB/s

bench.zig

The throughput numbers are highly variable on my computer.

lib/std/http/Client.zig Show resolved Hide resolved
lib/std/http/Client.zig Show resolved Hide resolved
lib/std/http.zig Outdated Show resolved Hide resolved
lib/std/http/Client.zig Show resolved Hide resolved
lib/std/http/Client.zig Show resolved Hide resolved
RFC 9110 section 15:
Values outside the range 100..599 are invalid. Implementations often use
three-digit integer values outside of that range (i.e., 600..999) for
internal communication of non-HTTP status (e.g., library errors). A
client that receives a response with an invalid status code SHOULD
process the response as if it had a 5xx (Server Error) status code.
 * when HTTP header continuations are used
 * when content-type or location header occurs more than once
@andrewrk andrewrk enabled auto-merge January 5, 2023 20:44
lib/std/http/Client.zig Show resolved Hide resolved
lib/std/http/Client.zig Show resolved Hide resolved
@andrewrk andrewrk merged commit b3e495a into master Jan 6, 2023
@andrewrk andrewrk deleted the std.http branch January 6, 2023 02:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

http client in the standard library
4 participants