http client in the standard library

[andrewrk]

Issue extracted from #910.

Note that this is in some ways blocking on:

• ability to use async/await agnostic of blocking/event-based (ability to write code that is agnostic of blocking vs async I/O #1778 ), which is blocking on
• coroutine rewrite (coroutine semantics are unsound. proposal to fix them #1363), which is blocking on
• copy elision (result location mechanism (previously: well defined copy eliding semantics) #287 ), which I will start working on as soon as we release 0.4.0 next month.

However what is entirely possible right now is a blocking implementation of an http client. This would not go entirely to waste; it would be one component of the complete system if std lib let you write code agnostic of blocking vs event based.

It's a pretty clear case for including this in the standard library since the package manager will rely on it; however we can revisit the organization of where code lives closer to 1.0.0.

[shawnl]

I am assuming this would be a HTTP/1.1 client. The use case you mention has no need for the much more complicated HTTP/2 or HTTP/3.

[ghost]

I found a workaround here

zig/lib/std/net.zig

Lines 219 to 247 in 5026db1

	pub fn connectUnixSocket(path: []const u8) !std.fs.File {
	const opt_non_block = if (std.event.Loop.instance != null) os.SOCK_NONBLOCK else 0;
	const sockfd = try os.socket(
	os.AF_UNIX,
	os.SOCK_STREAM | os.SOCK_CLOEXEC | opt_non_block,
	0,
	);
	errdefer os.close(sockfd);
	var sock_addr = os.sockaddr{
	.un = os.sockaddr_un{
	.family = os.AF_UNIX,
	.path = undefined,
	},
	};
	if (path.len > @typeOf(sock_addr.un.path).len) return error.NameTooLong;
	mem.copy(u8, sock_addr.un.path[0..], path);
	const size = @intCast(u32, @sizeOf(os.sa_family_t) + path.len);
	if (std.event.Loop.instance) |loop| {
	try os.connect_async(sockfd, &sock_addr, size);
	try loop.linuxWaitFd(sockfd, os.EPOLLIN | os.EPOLLOUT | os.EPOLLET);
	try os.getsockoptError(sockfd);
	} else {
	try os.connect(sockfd, &sock_addr, size);
	}
	return std.fs.File.openHandle(sockfd);
	}

[baverman]

However what is entirely possible right now is a blocking implementation of an http client. This would not go entirely to waste; it would be one component of the complete system if std lib let you write code agnostic of blocking vs event based.

Most of HTTP client/server code can be done without any IO notion at all. For example python implementation https://h11.readthedocs.io/en/latest/. It would be great so see similar approach in Zig.

@andrewrk how do you plan to support SSL in stdlib? Self-hosted implementation of tls1.2 is a pretty big investment. Or package manager can be linked with external library on its own?

[adontz]

I think we do not need such huge and opinionated modules in standard library.

There are a lot of open questions to me.

Protocol:
Under Linux you don't have many options, just some TCP socket + more or less reliable protocol parsers. But under Windows there are not one, but two system client libraries (WinInet and WinHTTP) which should be used in different scenarios. There are some details, but most importantly WinInet is for interactive applications and heavily relies on settings of Internet Explorer for current user (which can be managed in enterprise environments via Active Directory Group Policies), while WinHTTP is for non-interactive services executing under specialized user accounts.
https://docs.microsoft.com/en-us/windows/win32/wininet/wininet-vs-winhttp
Pure socket solution will seem to misbehave from developer as well as user perspective.
WinInet is extensible platform by itself
https://docs.microsoft.com/en-us/previous-versions/windows/internet-explorer/ie-developer/platform-apis/aa767916(v=vs.85)
so one application can register some zig:// protocol and another can use it.

Not sure about Apple products, how much NSURLRequest is integrated with other OS components.

Trusted Certificates:
Should we depend on system list of trusted certificates, have own list (like Java does), provide some callbacks for an application to decide? Should be accepting untrusted certificates be opt-in, opt-out or prohibited?

Protocol versions and extensions:
Should we support websockets? HTTP/3? gzip/deflate compression? Web Proxy Auto-Discovery Protocol? What about FTP?

Various helpers:
Should we support Punycode? Should we support happy eyeballs if implement in raw sockets?

Uploading a file to an HTTP server is a process of creating MIME envelope for file data and supplying a few headers. Should HTTP client library be dependent on/play nicely with MIME library? MIME is a huge beast, I've wrote a few parsers, it's not an easy task. Should the same MIME library be reused for email?

Honestly, I'd rather wrap WinInet, WinHTTP and libcurl, maybe vendor these libraries like musl. Rewriting anything related to HTTP in Zig, taking into account limited resources, enormous size of task, and questionable benefits does not seem reasonable to me.

[marler8997]

Here is my simple HTTP client that I am now using as a replacement for wget and curl. It also supports SSL with OpenSSL. Feel free to reference/use it, the hope is that it aids in development of the standard library HTTP client, at which point it will be deprecated.

https://github.com/marler8997/ziget

[Mouvedia]

The blockers are now #2761 and #2765.

[marler8997]

@Mouvedia could you elaborate on how those issues are blocking this one? I just want to keep up-to-date with the goings on.

[Mouvedia]

@marler8997 #287 being the last blocker, what was left of it was divided into two separate issues.

[marler8997]

@Mouvedia I read through those issues but am unclear as to how issues with copy eliding affect inclusion of an HTTP client in the standard library?

[Mouvedia]

op

[nektro]

With how ubiquitous HTTP is in the world, I feel this should definitely not be one of those things where everyone rolls their own solution and that this should absolutely be implemented in the std lib. Not having base primitives for interacting with the Web in Zig would severely hurt its ecosystem.

[cristaloleg]

I kinda agree that HTTP is de facto standard package/module for any language and stdlib, but this might be out of scope for Zig 'cause it's targeting on another type of projects.

Probably an official package under Ziglang organisation might work well 👀

[marler8997]

Anything that is necessary for the package manager will need to go into the standard library, that includes an HTTP client.

[ducdetronquito]

Hi everyone !

I just finished a tiny library called http that provides request and response builders.
The API is pretty much a port from the Rust http crate.

Here is a snippet, and I would love to have your reviews on the current API !

const Request = @import("http").Request;
const std = @import("std");

var request = try Request.builder(std.testing.allocator)
    .get("https://ziglang.org/")
    .header("GOTTA GO", "FAST")
    .body("ᕕ( ᐛ )ᕗ");
defer request.deinit();

As of yet, the library does not contains a URI parser or a correct Header multimap implementations because I wanted to discuss with you where does the Zig HTTP story should be happening and what is missing to have a working HTTP stack.

Where everything should be happening ?
Right now we have some parts dispatched into several repositories and in the standard library.

• Should we move everything in the standard library ?
• Should we create a dedicated Zig HTTP organisation on Github ? Like ZigLibs or zigtools.
• Should it stay like it is ?

What is missing in the stack ?

As far as I know, we have:

• a URI parser: zuri from @Vexu
• a Header multimap in the standard library thanks to @daurnimator and other contributors
• A HTTP/1.1 request/response parser: h11
• A request/response builder: http
• TCP helpers: zig-network from @MasterQ32
• A client experiment: ziget from @marler8997

What are the missing parts to have a working HTTP/1.1 client, that for a start, can be used by the package manager?

For my part I wanted to start working again on a state machine implementation of the HTTP/1.1 protocol (like what's being done by the guys from python-hyper), that would allow to implement the protocol rules but without dealing with I/O yet. But before starting anything involving, I would like to have your ideas and opinions on what should be done and how.

My knowledge on the matter is thin, and for example I can't answer @adontz remarks, but I am willing to help :)

Anyway, hope my message make sens !

Have a nice day 🌴

[kuon]

Here are (I think) a list of requirements for an HTTP library:

• HTTP 1.1 but 2 and 3 might become a requirement at some point
• System integration
  • proxy configuration
  • battery management (can be put to sleep)
  • data restriction (like return an error on cellular)
  • custom certificate (in enterprise windows environment this is necessary)
• Websocket support
• gzip/deflate
• TLS
• IPv6 support
• HTTPS DNS record support
• Async
• Connection reuse and parallel connection, connection pool

I agree with #2007 (comment) that this is a huge task, and I am not sure it is sane to allocate resources to it.

But I think this is important to have this in the standard library. It is a very difficult task, and has wide security and performances implications.

[kuon]

We are talking of an HTTP lib for the package manager. Most of what I mentioned is required for that. There are some added feature, namely websocket, HTTP2&3 and mobile client (battery and data/cellular) that are not required, but the rest is.

I know that list is 5 years from now, but that's exactly my point, we should realize that for many use case this is a list of requirements and that it's huge, that's why I think we should be pragmatic and see if we can use existing libraries.

[sskras]

I second about the complexity. Also it was said the Zig stdlib won't depend on libcurl.

@andrewrk commented on Mar 13, 2019 in #2056:

However I am requesting for cURL binding - which would allow you to use essentially pure Zig code and idioms, while internally calling a C library

I'm saying "no" to this. The zig standard library will not depend on libcurl.

But the aforementioned people still can use libcurl in their projects easily.

As a newbie I even think some folks might be able to translate its code from C to Zig and to try incorporating logic of libcurl in the stdlib (as the licenses are seemingly compatible: MIT | somewhat modified MIT). But well, I am not sure about handling the async affairs during the translation.

[sskras]

PS. I find the cpp-netlib interface/syntax to be semantically a lot cleaner, though.

The minimal http-client using libcurl:

The screenshot of http-client hello-world using cpp-netlib:

Maybe this will shed some light during creation of the interface to $Subject : )