How to avoid Slowloris DoS Attack?

[josecelano]

This is a very critical issue. I opened an issue one month ago but have not yet found a complete answer.

When you use Axum there is no way to set up a timeout for the time the server waits until the client sends the first request. YOu can reproduce it with:

1. Setup a basic server with axum.
2. Open a connection to the server with: telnet localhost 3000.

The server will never close the connection even if the client does not send any request.

You can find more info in the discussion and an example project I have published.

I have converted the discussion into an issue because I think this is a critical issue for some people. I know people who have migrated from Axum to ActixWeb because of this security problem.

I'm even considering it since the main purpose of web framework is to abstract away the details of HTTP operations. And I'm having a lot of trouble trying to patch this problem. I will keep trying and I will post my solution here (if I find it) if I find a complete solution. I know that maintaining this type of library takes considerable effort so I'm not complaining. I just wanted to give more visibility to this problem because I think it's not only my problem but a problem that all users have without even knowing it.

If you want to know what I have tried. I'm trying to use a custom Accetor written by @programatik29 but it does not work when you enable TSL. Details here.

Discussed in #2716

^{Originally posted by josecelano April 18, 2024}

Summary

Relates to: #1383

I'm trying to set a timeout for the time the server keeps a connection open while waiting for the client to start sending a request.

IMPORTANT: it's NOT a timeout for:

• The time receiving the request headers (after the client starts sending the headers)
• The time processing the request (because it takes too long)
• The time building the response body.

I've created a repo to reproduce the problem with a detailed description:

https://github.com/josecelano/axum-server-timeout

It's very easy to perform a slowloris attack .

axum version

0.7.5

[mladedav]

axum could theoretically add a timeout for the connection that would be cancelled once the tower service for the connection (i.e. axum::Router) was first polled. This should cover both connections not sending anything (which is what you seem to be describing) as well as attackers sending single bytes with large intervals (the slowloris attack).

[jplatte]

So initially I thought since hyper doesn't do the connection accepting internally anymore, that this would need to be managed outside of hyper. Now I'm actually mostly convinced wbout the opposite: as soon as the conn is opened, we pass it to hyper, right? So it would be able to tell when nothing happens there. So if I'm right, this is almost certainly something hyper should handle (maybe it already does with some configuration).

Also if this is really so problematic security wise, it should not have been a public discussion / issue, but that ship has sailed I guess.. :/

[josecelano]

Thank you @jplatte @jplatte for your comments.

So initially I thought since hyper doesn't do the connection accepting internally anymore, that this would need to be managed outside of hyper. Now I'm actually mostly convinced wbout the opposite: as soon as the conn is opened, we pass it to hyper, right? So it would be able to tell when nothing happens there. So if I'm right, this is almost certainly something hyper should handle (maybe it already does with some configuration).

@jplatte I can't find the thread now, but some people reach the same conclusion that this is something should be handled in Hyper. However, it looks like the problem is also somehow postponed there:

• timeout while waiting for headers hyperium/hyper#3178
• Server idle/read/write timeout support hyperium/hyper#1628

I guess, Axum could apply a temporarily patch until it's fixed in the hyper repo. I've tried to collect all the info related to this problem in this repo:

https://github.com/josecelano/axum-server-timeout

It also reproduces the problem with different frameworks.

Also if this is really so problematic security wise, it should not have been a public discussion / issue, but that ship has sailed I guess.. :/

As you can see in all the links it's a really well-known problem since some years ago. I guess people use Nginx/Cloudflare etcetera when they have problem with this type of attack.

[josecelano]

@jplatte in fact, I've been trying to apply a patch written by @programatik29:

https://gist.github.com/programatik29/36d371c657392fd7f322e7342957b6d1

It's a TimeoutAcceptor. Maybe that would be the Axum official patch for this problem. I've tried to use and it works for HTTP but not for HTTPs. Maybe for someone that knows well Axum internal stuff it could be something easy to do. For me, it's still a complex task because I'm newbie in Rust and I don't know Axum very well. Here you can see a working example where I have applied the patch but I does not work for HTTPs.

[ryandotsmith]

+1 I am quite surprised to find that there is no easy way to prevent this attack. This is a pretty easy attack to create too.

[ryandotsmith]

I should also add that it is good security practice to always set timeouts on reading and writing along with the max number of bytes you would want to accept. Perhaps these options should be front and center so that users can easily do the right thing.

Perhaps an interface like Go's net/http

[Roardom]

This issue has been known for awhile and has caused users to move from axum to actix: Power2All/torrust-actix#25

They had also asked in the Discord long ago about preventing this and mentioned the attack, and ran into other issues as well that they reported.

I ended up just placing my axum app behind nginx with other custom ddos prevention measures (cloudflare isn't an option for us), but it would be nice not having to worry about it.

Seems to be a common issue to run into when developing/hosting BitTorrent trackers...

Edit: Example fix can be found here: https://discord.com/channels/500028886025895936/870760546109116496/threads/1104829785319944294, but indeed looks like they had issues with https as well.

[ttys3]

I don't think preventing attacks is the responsibility of an http framework. If it does, that's great. However, in real usage scenarios, we will basically add an api gateway such as traefik before the http service. These timeout configurations should be placed on the gateway.

[jplatte]

Hyper definitely cares about DOS attacks AFAIK. I don't know why this one seems low(er) priority. But anyways, the more I think about it the less I see why axum would try to do anything here, since it can be solved more easily in hyper.

Because of that, I'll close this. I think it would be best for the discussion to continue on the hyper issue.

[josecelano]

Hy @jplatte, hyper has released a new version 1.4.0 where the have changed the header_read_timeout. This new version starts the timer immediately, right when the connection is estabilished.

If I run a server with hyper I get the desired behavior. The connection is closed after 5 seconds if the client does not send any request. I have changed one of the examples in the hyper repo:

josecelano/hyper@def1444

I would like to get the same result with Axum. I'm trying to modify the serve-with-hyper to add the timeout, but it seems it's not trivial.

#[tokio::main]
async fn main() {
    tokio::join!(serve_plain(), serve_with_connect_info());
}

async fn serve_plain() {
    // Create a regular axum app.
    let app = Router::new().route("/", get(|| async { "Hello!" }));

    // Create a `TcpListener` using tokio.
    let listener = TcpListener::bind("0.0.0.0:3000").await.unwrap();

    // Continuously accept new connections.
    loop {
        // In this example we discard the remote address. See `fn serve_with_connect_info` for how
        // to expose that.
        let (socket, _remote_addr) = listener.accept().await.unwrap();

        // We don't need to call `poll_ready` because `Router` is always ready.
        let tower_service = app.clone();

        // Spawn a task to handle the connection. That way we can handle multiple connections
        // concurrently.
        tokio::spawn(async move {
            // Hyper has its own `AsyncRead` and `AsyncWrite` traits and doesn't use tokio.
            // `TokioIo` converts between them.
            let socket = TokioIo::new(socket);

            // Hyper also has its own `Service` trait and doesn't use tower. We can use
            // `hyper::service::service_fn` to create a hyper `Service` that calls our app through
            // `tower::Service::call`.
            let hyper_service = hyper::service::service_fn(move |request: Request<Incoming>| {
                // We have to clone `tower_service` because hyper's `Service` uses `&self` whereas
                // tower's `Service` requires `&mut self`.
                //
                // We don't need to call `poll_ready` since `Router` is always ready.
                tower_service.clone().call(request)
            });

            let mut server = server::conn::auto::Builder::new(TokioExecutor::new());

            server
                .http1()
                .timer(TokioTimer)
                .header_read_timeout(Duration::from_secs(5));

            // `server::conn::auto::Builder` supports both http1 and http2.
            //
            // `TokioExecutor` tells hyper to use `tokio::spawn` to spawn tasks.
            if let Err(err) = server
                // `serve_connection_with_upgrades` is required for websockets. If you don't need
                // that you can use `serve_connection` instead.
                .serve_connection_with_upgrades(socket, hyper_service)
                .await
            {
                eprintln!("failed to serve connection: {err:#}");
            }
        });
    }
}

Hyper sets a 30-second default value for the timeout, but it seems it's not affecting Axum. I'm using axum v0.7.5

[jplatte]

Sounds like a hyper bug or a problem with how the example is set up. I would recommend you try replacing axum with some sort of simpler service like the one in the test from the PR you linked, I strongly suspect that it will still be the same issue then.

[randomairborne]

I got this figured out. It seems as if for some reason TokioIo doesn't start the timer until it reads its first byte from the socket- I've yet to figure out why, but I will dig into that.

            let initial_timeout = tokio::time::sleep(Duration::from_secs(5));
            let mut peek_buf = [0; 1];
            tokio::select! {
                _ = socket.peek(&mut peek_buf) => {},
                _ = initial_timeout => return,
            }

For now, inserting the above snippet at the top of the spawned async block seems to effectively abort the connection if there isn't any activity for the duration of the sleep. This might not be GOOD, or performant, but it is easy.

[randomairborne]

You can also make your server http1 only and disable upgrades, but that is likely Not A Good Idea