HTTP Server with chunked transfer encoding

[vladfaust]

I want to implement Twitter-like functionality of continuously streaming data without closing the connection. Websockets don't work for me for this particular situation.

Twitter server

Twitter Streaming API client shard currently has this implementation (simplified) of streaming tweets:

class Twitter::StreamAPI
  def stream_tweets(&block)
    http_client.get(path) do |response|
      puts "RESPONSE HEADERS: #{response.headers}"
      loop do
        yield response.body_io.gets.not_nil!
      end
    end
  end
end

And an application may do this:

def stream_tweets
  twitter_stream_api.stream_tweets do |tweet|
    pp tweet
  end
end

It has following headers: RESPONSE HEADERS: HTTP::Headers{"connection" => "close", "content-Encoding" => "gzip", "content-type" => "application/json", "date" => "...", "server" => "tsa", "transfer-encoding" => "chunked", "x-connection-hash" => "..."} and it actually continuously gets the response! Each time a new tweet is posted, this block is yielded, the connection stays alive forever!

Yes, that's true, I already have Telegram bot posting new tweets in a channel.

Implementing the same functionality

After some googling I've found that Twitter might use Chunked transfer encoding.

As I can see in the CHANGELOG:

Added support for sending chunked content in HTTP server (thanks @bcardiff)

But HTTP::Server::Response::Output#unbuffered_write is private. 🤔

Therefore, I clearly don't get how to serve chunked responses.

My trials

So far, this is what I've tried to do:

server.cr

require "http/server"

port = 5000

server = HTTP::Server.new(port) do |context|
  puts "Got request"
  context.response.headers["Transfer-Encoding"] = "chunked"
  context.response.upgrade do |socket|
    puts "Upgraded"
    loop do
      puts "Writing"
      chunked_write(socket, "Query: #{context.request.query}".to_slice)
      sleep 5
    end
  end
end

def chunked_write(io, slice)
  slice.size.to_s(16, io)
  io << "\r\n"
  io.write(slice)
  io << "\r\n"
end

puts "Listening on #{port}"
server.listen

client.cr

require "http/client"

client = HTTP::Client.new("localhost", port: 5000)

client.get "/" do |response|
  puts "Got response"
  loop do
    puts "Response: " + response.body_io.gets.not_nil!
  end
end

Output

This is server output:

Listening on 5000
Got request
Upgraded
Writing
Writing
...

And this is what I got on the client. Note that the output is empty until I press Ctrl-C on server:

Nil assertion failed (Exception)
0x481887: *CallStack::unwind:Array(Pointer(Void)) at ??
0x4a6310: not_nil! at /opt/crystal/src/class.cr 65:0
0x4f5b07: initialize at /opt/crystal/src/http/content.cr 55:26
0x4f5a69: new at /opt/crystal/src/http/content.cr 54:5
0x4676dd: __crystal_main at /opt/crystal/src/http/common.cr 36:11
0x47b469: main at /opt/crystal/src/main.cr 12:15
0x7f938bd4c830: __libc_start_main at ??
0x462659: _start at ??
0x0: ??? at ??

Please help me implementing a chunked server in Crystal!

[makenowjust]

You should append \n to sent text to gets in client and do io.flush. And you don't need to use upgrade and create chunk response manually. Simplified server.cr is here:

require "http/server"

port = 5000

server = HTTP::Server.new(port) do |context|
  puts "Got request"
  # `context.response.headers["Transfer-Encoding"] = "chunked"` is default.
  loop do
    puts "Writing"
    context.response.puts "Query: #{context.request.query}"
    context.response.flush
    sleep 5
  end
end

puts "Listening on #{port}"
server.listen

[RX14]

@vladfaust if @makenowjust didn't make that crystal clear: crystal already handles writing chunked responses. Just keep on writing to the output response, and call flush when you want to ensure the client receives the message. If there is no content length header, the response will automatically select chunked encoding for you.

[makenowjust]

@RX14 Thank you for detailed explanation!

[vladfaust]

@RX14 @makenowjust thank you guys. Should I create a separate issue stating that we need more docs for that? @RX14 Your quote could be used.

[asterite]

Regarding docs: no. I prefer to use StackOverflow, it's a living "missing docs and examples". It's simply impossible to put every use case in documentation. So this should be really asked in StackOverflow.

[vladfaust]

Also I'm thinking that I don't know much about IO. flush is kinda magic for me. I suppose there are lots of developers coming from Ruby / Python who never work with IO so close. They may struggle too.

[vladfaust]

@RX14 please: https://stackoverflow.com/questions/46298660/how-to-create-a-streaming-http-server-in-crystal