Crowdsec reCAPTCHA remediation : http2 requests are not supported yet #44

shad-lp · 2023-09-03T13:49:06Z

Hello,

I've set up Crowdsec to use captcha remediation instead of ban for some collections, especially LePresidente/Emby.
On a banned IP, as soon as the captcha is resolved, I got an error 500 in the browser.
In the nginx (SWAG Linuxserver) error.log I get this error :

stack traceback:
coroutine 0:
        [C]: in function 'read_body'
        /usr/local/lua/crowdsec/crowdsec.lua:485: in function 'Allow'
        access_by_lua(http.d/crowdsec_nginx.conf:14):3: in main chunk while sending to client, client: 46.178.201.44, server: emby.*, request: "GET /web/index.html HTTP/2.0", host: "emby.xxx.ovh"

I tried to :

remove the http2 argument in the listen directive in my emby reverse proxy configuration file, but no change.
remove the http2 argument in the default.conf for main server, no change
both, no change

If I switch back to ban remediation, everything runs smoothly.

Here is my setup :

profiles.yaml (crowdsec) : https://pastebin.com/ftcdgtwT

crowdsec-nginx-bouncer.conf :

API_URL=http://crowdsec:8080
API_KEY=REDACTED
CACHE_EXPIRATION=1
# bounce for all type of remediation that the bouncer can receive from the local API
BOUNCING_ON_TYPE=all
FALLBACK_REMEDIATION=ban
REQUEST_TIMEOUT=3000
UPDATE_FREQUENCY=10
# live or stream
MODE=live
# exclude the bouncing on those location
EXCLUDE_LOCATION=
#those apply for "ban" action
# /!\ REDIRECT_LOCATION and RET_CODE can't be used together. REDIRECT_LOCATION take priority over RET_CODE
BAN_TEMPLATE_PATH=/var/lib/crowdsec/lua/templates/ban.html
REDIRECT_LOCATION=
RET_CODE=
#those apply for "captcha" action
CAPTCHA_PROVIDER=recaptcha
# ReCaptcha Secret Key
SECRET_KEY=REDACTED
# Recaptcha Site key
SITE_KEY=REDACTED
CAPTCHA_TEMPLATE_PATH=/var/lib/crowdsec/lua/templates/captcha.html
CAPTCHA_EXPIRATION=3600

I don't know if I'm doing something wrong, any help appreciated.

The text was updated successfully, but these errors were encountered:

LaurenceJJones · 2023-09-04T08:29:07Z

This is not us, this is the lua http module for NGINX does not support http 2. We are kind of helpless to this.

Seems the error you shown above is not directly http2. Within the proxy pass do you downgrade to http 1.1?

shad-lp · 2023-09-04T19:31:25Z

Hello,

Thanks for your answer, sorry I've been told on Discord to post my issue here.

Seems the error you shown above is not directly http2. Within the proxy pass do you downgrade to http 1.1?

AFAIK, I don't, but I'm not an expert on Nginx, barely an enthousiast homelabber.
Here are my nginx conf files if it can help.

default.conf :

# redirect all traffic to https
server {
    listen 80 default_server;
    listen [::]:80 default_server;

    location / {
        return 301 https://$host$request_uri;
    }
}

# main server block
server {
    listen 443 ssl http2 default_server;
    listen [::]:443 ssl http2 default_server;

    server_name _;

    include /config/nginx/ssl.conf;

    root /config/www;
    index index.html index.htm index.php;

    # enable subfolder method reverse proxy confs
    include /config/nginx/proxy-confs/*.subfolder.conf;

    # enable for ldap auth (requires ldap-location.conf in the location block)
    #include /config/nginx/ldap-server.conf;

    # enable for Authelia (requires authelia-location.conf in the location block)
    #include /config/nginx/authelia-server.conf;

    # enable for Authentik (requires authentik-location.conf in the location block)
    #include /config/nginx/authentik-server.conf;

    location / {
        # enable for basic auth
        #auth_basic "Restricted";
        #auth_basic_user_file /config/nginx/.htpasswd;

        # enable for ldap auth (requires ldap-server.conf in the server block)
        #include /config/nginx/ldap-location.conf;

        # enable for Authelia (requires authelia-server.conf in the server block)
        #include /config/nginx/authelia-location.conf;

        # enable for Authentik (requires authentik-server.conf in the server block)
        #include /config/nginx/authentik-location.conf;

        try_files $uri $uri/ /index.html /index.php$is_args$args =404;
    }

    location ~ ^(.+\.php)(.*)$ {
        # enable the next two lines for http auth
        #auth_basic "Restricted";
        #auth_basic_user_file /config/nginx/.htpasswd;

        # enable for ldap auth (requires ldap-server.conf in the server block)
        #include /config/nginx/ldap-location.conf;

        # enable for Authelia (requires authelia-server.conf in the server block)
        #include /config/nginx/authelia-location.conf;

        # enable for Authentik (requires authentik-server.conf in the server block)
        #include /config/nginx/authentik-location.conf;

        fastcgi_split_path_info ^(.+\.php)(.*)$;
        fastcgi_pass 127.0.0.1:9000;
        fastcgi_index index.php;
        include /etc/nginx/fastcgi_params;
    }

    # deny access to .htaccess/.htpasswd files
    location ~ /\.ht {
        deny all;
    }
}

# enable subdomain method reverse proxy confs
include /config/nginx/proxy-confs/*.subdomain.conf;
# enable proxy cache for auth
proxy_cache_path cache/ keys_zone=auth_cache:10m;

proxy headers embedded :

## Version 2023/02/09 - Changelog: https://github.com/linuxserver/docker-swag/commits/master/root/defaults/nginx/proxy.conf.sample

# Timeout if the real server is dead
proxy_next_upstream error timeout invalid_header http_500 http_502 http_503;

# Proxy Connection Settings
proxy_buffers 32 4k;
proxy_connect_timeout 240;
proxy_headers_hash_bucket_size 128;
proxy_headers_hash_max_size 1024;
proxy_http_version 1.1;
proxy_read_timeout 240;
proxy_redirect http:// $scheme://;
proxy_send_timeout 240;

# Proxy Cache and Cookie Settings
proxy_cache_bypass $cookie_session;
#proxy_cookie_path / "/; Secure"; # enable at your own risk, may break certain apps
proxy_no_cache $cookie_session;

# Proxy Header Settings
proxy_set_header Connection $connection_upgrade;
proxy_set_header Early-Data $ssl_early_data;
proxy_set_header Host $host;
proxy_set_header Proxy "";
proxy_set_header Upgrade $http_upgrade;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Method $request_method;
proxy_set_header X-Forwarded-Port $server_port;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Forwarded-Server $host;
proxy_set_header X-Forwarded-Ssl on;
proxy_set_header X-Forwarded-Uri $request_uri;
proxy_set_header X-Original-Method $request_method;
proxy_set_header X-Original-URL $scheme://$http_host$request_uri;
proxy_set_header X-Real-IP $remote_addr;

There is indeed a line proxy_http_version 1.1; but seems like it can either be 1.0 and 1.1 as of now.

Finally, my crowdsec reverse proxy file :

## Version 2023/05/31
# make sure that your crowdsec container is named crowdsec
# make sure that your dns has a cname set for crowdsec

server {
    listen 443 ssl http2;
    listen [::]:443 ssl http2;

    server_name crowdsec.*;

    include /config/nginx/ssl.conf;

    client_max_body_size 0;

    # enable for ldap auth (requires ldap-location.conf in the location block)
    #include /config/nginx/ldap-server.conf;

    # enable for Authelia (requires authelia-location.conf in the location block)
    #include /config/nginx/authelia-server.conf;

    # enable for Authentik (requires authentik-location.conf in the location block)
    #include /config/nginx/authentik-server.conf;

    location / {
        # enable the next two lines for http auth
        #auth_basic "Restricted";
        #auth_basic_user_file /config/nginx/.htpasswd;

        # enable for ldap auth (requires ldap-server.conf in the server block)
        #include /config/nginx/ldap-location.conf;

        # enable for Authelia (requires authelia-server.conf in the server block)
        #include /config/nginx/authelia-location.conf;

        # enable for Authentik (requires authentik-server.conf in the server block)
        #include /config/nginx/authentik-location.conf;

        include /config/nginx/proxy.conf;
        include /config/nginx/resolver.conf;
        set $upstream_app crowdsec;
        set $upstream_port 8080;
        set $upstream_proto http;
        proxy_pass $upstream_proto://$upstream_app:$upstream_port;

    }
}

I need captcha because of some false positives on crowdsec bans.
That way my users can unban themselves if they shouldn't have been.

alleyu2 · 2023-10-11T21:23:15Z

I am getting the same error as well and using SWAG for proxy

alleyu2 · 2023-10-12T23:02:06Z

I have opened a ticket at linuxserver since I am assuming this issue has something to do with SWAG. In case someone needs to follow the issue progress: linuxserver/docker-mods#772

dreadkopp · 2023-11-06T18:02:42Z

running into the same issue.

Is it possible to roll back to http 1.1 when captcha remidation is detected ?

neither my lua nor nginx internals knowledge is good enough to answer that by myself ;)

LaurenceJJones · 2023-11-06T18:50:31Z

running into the same issue.

Is it possible to roll back to http 1.1 when captcha remidation is detected ?

neither my lua nor nginx internals knowledge is good enough to answer that by myself ;)

No, as it changes how the http server works. The only other way would be to set up a different subdomain on a different nginx install that is http 1 only, and all other servers redirect to that sever to which the webserver on the captcha domain redirects back to a query parameter after a valid solve.

Or once our NJS bouncer is out that could be an alternative 🤷🏻 However, this limitation is nginx lua integration itself not ours so we have no control over this.

LaurenceJJones · 2024-01-12T14:59:38Z

Hey all finally found sometime to look why my nginx is okay and SWAG seems to be affected?

It seems this PR fixed it https://github.com/openresty/lua-nginx-module/pull/2237/files that is the request has a content length it will be accept as http2

I am using a compiled version from ubuntu which was before the PR that added the unsupported flag. If the lua code is bumped up to latest it should start working again.

shad-lp · 2024-01-12T22:42:40Z

Right now I'm having still having the issue.

nginx -V

nginx version: nginx/1.24.0
built with OpenSSL 3.1.3 19 Sep 2023 (running with OpenSSL 3.1.4 24 Oct 2023)
TLS SNI support enabled
[...]

nginx -V 2>&1 | tr ' ' '\n'

[...]
arguments:
--prefix=/var/lib/nginx
--sbin-path=/usr/sbin/nginx
--modules-path=/usr/lib/nginx/modules
--conf-path=/etc/nginx/nginx.conf
--pid-path=/run/nginx/nginx.pid
--lock-path=/run/nginx/nginx.lock
--http-client-body-temp-path=/var/lib/nginx/tmp/client_body
--http-proxy-temp-path=/var/lib/nginx/tmp/proxy
--http-fastcgi-temp-path=/var/lib/nginx/tmp/fastcgi
--http-uwsgi-temp-path=/var/lib/nginx/tmp/uwsgi
--http-scgi-temp-path=/var/lib/nginx/tmp/scgi
--with-perl_modules_path=/usr/lib/perl5/vendor_perl
--user=nginx
--group=nginx
--with-threads
--with-file-aio
--without-pcre2
--with-http_ssl_module
--with-http_v2_module
--with-http_realip_module
--with-http_addition_module
--with-http_xslt_module=dynamic
--with-http_image_filter_module=dynamic
--with-http_geoip_module=dynamic
--with-http_sub_module
--with-http_dav_module
--with-http_flv_module
--with-http_mp4_module
--with-http_gunzip_module
--with-http_gzip_static_module
--with-http_auth_request_module
--with-http_random_index_module
--with-http_secure_link_module
--with-http_degradation_module
--with-http_slice_module
--with-http_stub_status_module
--with-http_perl_module=dynamic
--with-mail=dynamic
--with-mail_ssl_module
--with-stream=dynamic
--with-stream_ssl_module
--with-stream_realip_module
--with-stream_geoip_module=dynamic
--with-stream_ssl_preread_module
--add-dynamic-module=/home/buildozer/aports/main/nginx/src/njs-0.7.11/nginx
--add-dynamic-module=/home/buildozer/aports/main/nginx/src/ngx_devel_kit-0.3.2/
--add-dynamic-module=/home/buildozer/aports/main/nginx/src/traffic-accounting-nginx-module-2.0/
--add-dynamic-module=/home/buildozer/aports/main/nginx/src/array-var-nginx-module-0.06/
--add-dynamic-module=/home/buildozer/aports/main/nginx/src/nginx-auth-jwt-0.2.1/
--add-dynamic-module=/home/buildozer/aports/main/nginx/src/ngx_brotli-1.0.0rc/
--add-dynamic-module=/home/buildozer/aports/main/nginx/src/ngx_cache_purge-2.5.3/
--add-dynamic-module=/home/buildozer/aports/main/nginx/src/nginx_cookie_flag_module-1.1.0/
--add-dynamic-module=/home/buildozer/aports/main/nginx/src/nginx-dav-ext-module-3.0.0/
--add-dynamic-module=/home/buildozer/aports/main/nginx/src/echo-nginx-module-0.63/
--add-dynamic-module=/home/buildozer/aports/main/nginx/src/encrypted-session-nginx-module-0.09/
--add-dynamic-module=/home/buildozer/aports/main/nginx/src/ngx-fancyindex-0.5.2/
--add-dynamic-module=/home/buildozer/aports/main/nginx/src/ngx_http_geoip2_module-3.4/
--add-dynamic-module=/home/buildozer/aports/main/nginx/src/headers-more-nginx-module-0.34/
--add-dynamic-module=/home/buildozer/aports/main/nginx/src/nginx-keyval-0.1.0/
--add-dynamic-module=/home/buildozer/aports/main/nginx/src/nginx-log-zmq-1.0.0/
--add-dynamic-module=/home/buildozer/aports/main/nginx/src/lua-nginx-module-0.10.24/
--add-dynamic-module=/home/buildozer/aports/main/nginx/src/lua-upstream-nginx-module-0.07/
--add-dynamic-module=/home/buildozer/aports/main/nginx/src/naxsi-1.3/naxsi_src
--add-dynamic-module=/home/buildozer/aports/main/nginx/src/nchan-1.3.6/
--add-dynamic-module=/home/buildozer/aports/main/nginx/src/redis2-nginx-module-0.15/
--add-dynamic-module=/home/buildozer/aports/main/nginx/src/set-misc-nginx-module-0.33/
--add-dynamic-module=/home/buildozer/aports/main/nginx/src/nginx-http-shibboleth-2.0.1/
--add-dynamic-module=/home/buildozer/aports/main/nginx/src/ngx_http_untar_module-1.1/
--add-dynamic-module=/home/buildozer/aports/main/nginx/src/nginx-upload-module-2.3.0/
--add-dynamic-module=/home/buildozer/aports/main/nginx/src/nginx-upload-progress-module-0.9.2/
--add-dynamic-module=/home/buildozer/aports/main/nginx/src/nginx-upstream-fair-0.1.3/
--add-dynamic-module=/home/buildozer/aports/main/nginx/src/ngx_upstream_jdomain-1.4.0/
--add-dynamic-module=/home/buildozer/aports/main/nginx/src/nginx-vod-module-1.31/
--add-dynamic-module=/home/buildozer/aports/main/nginx/src/nginx-module-vts-0.2.1/
--add-dynamic-module=/home/buildozer/aports/main/nginx/src/mod_zip-1.3.0/
--add-dynamic-module=/home/buildozer/aports/main/nginx/src/nginx-rtmp-module-1.2.2/

nginx -V 2>&1 | tr ' ' '\n' | grep lua

--add-dynamic-module=/home/buildozer/aports/main/nginx/src/lua-nginx-module-0.10.24/
--add-dynamic-module=/home/buildozer/aports/main/nginx/src/lua-upstream-nginx-module-0.07/

I'm newbie, but from what I read, the PR has been merged, the lua module is in 0.10.26 version right now, while SWAG is still running the 0.10.24 version. So maybe that's the reason the issue persists ?

LaurenceJJones · 2024-01-12T22:46:56Z

I'm newbie, but from what I read, the PR has been merged, the lua module is in 0.10.26 version right now, while SWAG is still running the 0.10.24 version. So maybe that's the reason the issue persists ?

Yes alpine repositories have not been updated to 0.10.26, so the issue will still be there until alpine repositories are updated AND they have push a backport to 3.18 since that is alpine image linuxserver is currently using.

I never had the issue because im on ubuntu and they dropped support so I never got the updated code to fix http2 issue they saw, however, they now merged in a new fix to allow http2 requests with a content-length header

Zoey2936 · 2024-02-11T13:03:01Z

Hello, I've yesterday tried a bit around with Captchas/AppSec, and it works ... if the request uses HTTP/1 (1.0 or 1.1), with HTTP/2 and HTTP/3 request my log is spammed (two examples from DoH requests, the error also appears when opening a site in Firefox/chromium):

2024/02/11 13:45:59 [error] 1318#1318: *544 lua entry thread aborted: runtime error: /usr/local/nginx/lib/lua/crowdsec.lua:410: http2 requests are not supported without content-length header
stack traceback:
coroutine 0:
        [C]: in function 'read_body'
        /usr/local/nginx/lib/lua/crowdsec.lua:410: in function 'get_body'
        /usr/local/nginx/lib/lua/crowdsec.lua:523: in function 'AppSecCheck'
        /usr/local/nginx/lib/lua/crowdsec.lua:613: in function 'Allow'
        access_by_lua(conf.d/crowdsec.conf:18):6: in main chunk, client: <ip>, server: *.dns.example.com, request: "GET /dns-query/<device-name>?dns=<dns-request-string> HTTP/2.0", host: "dns.zvcdn.de"
2024/02/11 13:45:59 [error] 1318#1318: *627 lua entry thread aborted: runtime error: /usr/local/nginx/lib/lua/crowdsec.lua:410: http3 requests are not supported without content-length header
stack traceback:
coroutine 0:
        [C]: in function 'read_body'
        /usr/local/nginx/lib/lua/crowdsec.lua:410: in function 'get_body'
        /usr/local/nginx/lib/lua/crowdsec.lua:523: in function 'AppSecCheck'
        /usr/local/nginx/lib/lua/crowdsec.lua:613: in function 'Allow'
        access_by_lua(conf.d/crowdsec.conf:18):6: in main chunk, client: <ip>, server: *.dns.example.com, request: "GET /dns-query?dns=<dns-request-string> HTTP/3.0"

I get an 500 error with all clients: DoH (IOS native, Adguardhome, cloudflared, etc.), curl, Firefox and chromium, if I disable HTTP/2 and HTTP/3 it works without problems.
This makes appsec and captchas unusable. So it seems that in newer versions of lua-nginx-module the issue persists but has a different error name...

LaurenceJJones · 2024-02-11T15:34:15Z

This makes appsec and captchas unusable. So it seems that in newer versions of lua-nginx-module the issue persists but has a different error name...

The problem we have there is no way in current nginx lua to ask if there is a body and the way they intend for you to do it call the function which they added the log spam on and check if nil was returned. However, within the new implementation its throw an error of just returning nil.

We could wrap it around a try, catch because in your example I guess they didnt have any body cause they are GET requests

Zoey2936 · 2024-02-11T15:39:41Z

In my short test this error only occurred with HEAD and GET requests, but I don't know if there was even a POST request made which could throw this error

LaurenceJJones · 2024-02-11T15:41:10Z

In my short test this error only occurred with HEAD and GET requests, but I don't know if there was even a POST request made which could throw this error

Most likely those two request types would throw this error as they dont have a content-length header as they dont have bodies.

Zoey2936 · 2024-02-11T15:43:37Z

yes

LaurenceJJones · 2024-02-22T09:34:23Z

Just adding a note for anyone who comes here: You MUST be using nginx-http-mod-lua version 0.10.26 anything less you will still get issues with http2/3

If you are on this version and still encounter issues please open a new issue with information need to replicate

audioscavenger · 2024-03-29T17:39:15Z

upgraded swag yesterday to latest, image: ghcr.io/linuxserver/swag
nginx -V shows lua 0.10.25 not 0.10.26:
--add-dynamic-module=/home/buildozer/aports/main/nginx/src/lua-nginx-module-0.10.25/

LaurenceJJones · 2024-03-29T18:27:17Z

upgraded swag yesterday to latest, image: ghcr.io/linuxserver/swag nginx -V shows lua 0.10.25 not 0.10.26: --add-dynamic-module=/home/buildozer/aports/main/nginx/src/lua-nginx-module-0.10.25/

Yes as stated the maintainer of lua nginx on alpine said they would back port it, however, it still doesnt seem like they have 😟

Zoey2936 added a commit to ZoeyVid/NPMplus that referenced this issue Feb 11, 2024

disable appsec by default because of crowdsecurity/lua-cs-bouncer#44

59fa36f

Zoey2936 added a commit to ZoeyVid/NPMplus that referenced this issue Feb 11, 2024

disable appsec by default because of crowdsecurity/lua-cs-bouncer#44

d0948b9

LaurenceJJones mentioned this issue Feb 11, 2024

HTTP2/3 Check request before invoking read_body #63

Closed

blotus mentioned this issue Feb 13, 2024

do not try to read request body in HTTP2/3 if there's no content-length header #64

Merged

blotus closed this as completed in #64 Feb 22, 2024

shad-lp mentioned this issue Mar 9, 2024

[BUG] cs-nginx-bouncer v1.0.8 : http2 requests are not supported yet linuxserver/docker-mods#862

Closed

1 task

inthreedee mentioned this issue Dec 27, 2024

Appsec installation help crowdsecurity/example-docker-compose#30

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Crowdsec reCAPTCHA remediation : http2 requests are not supported yet #44

Crowdsec reCAPTCHA remediation : http2 requests are not supported yet #44

shad-lp commented Sep 3, 2023 •

edited

Loading

LaurenceJJones commented Sep 4, 2023

shad-lp commented Sep 4, 2023 •

edited

Loading

alleyu2 commented Oct 11, 2023

alleyu2 commented Oct 12, 2023 •

edited

Loading

dreadkopp commented Nov 6, 2023

LaurenceJJones commented Nov 6, 2023 •

edited

Loading

LaurenceJJones commented Jan 12, 2024 •

edited

Loading

shad-lp commented Jan 12, 2024

LaurenceJJones commented Jan 12, 2024 •

edited

Loading

Zoey2936 commented Feb 11, 2024

LaurenceJJones commented Feb 11, 2024

Zoey2936 commented Feb 11, 2024 •

edited

Loading

LaurenceJJones commented Feb 11, 2024 •

edited

Loading

Zoey2936 commented Feb 11, 2024

LaurenceJJones commented Feb 22, 2024

audioscavenger commented Mar 29, 2024

LaurenceJJones commented Mar 29, 2024

Crowdsec reCAPTCHA remediation : http2 requests are not supported yet #44

Crowdsec reCAPTCHA remediation : http2 requests are not supported yet #44

Comments

shad-lp commented Sep 3, 2023 • edited Loading

LaurenceJJones commented Sep 4, 2023

shad-lp commented Sep 4, 2023 • edited Loading

alleyu2 commented Oct 11, 2023

alleyu2 commented Oct 12, 2023 • edited Loading

dreadkopp commented Nov 6, 2023

LaurenceJJones commented Nov 6, 2023 • edited Loading

LaurenceJJones commented Jan 12, 2024 • edited Loading

shad-lp commented Jan 12, 2024

LaurenceJJones commented Jan 12, 2024 • edited Loading

Zoey2936 commented Feb 11, 2024

LaurenceJJones commented Feb 11, 2024

Zoey2936 commented Feb 11, 2024 • edited Loading

LaurenceJJones commented Feb 11, 2024 • edited Loading

Zoey2936 commented Feb 11, 2024

LaurenceJJones commented Feb 22, 2024

audioscavenger commented Mar 29, 2024

LaurenceJJones commented Mar 29, 2024

shad-lp commented Sep 3, 2023 •

edited

Loading

shad-lp commented Sep 4, 2023 •

edited

Loading

alleyu2 commented Oct 12, 2023 •

edited

Loading

LaurenceJJones commented Nov 6, 2023 •

edited

Loading

LaurenceJJones commented Jan 12, 2024 •

edited

Loading

LaurenceJJones commented Jan 12, 2024 •

edited

Loading

Zoey2936 commented Feb 11, 2024 •

edited

Loading

LaurenceJJones commented Feb 11, 2024 •

edited

Loading