[SWAG - Crowdsec Docker mod] 500 Internal Server Error with recaptcha

[alleyu2]

I am using the crowdsec docker mod and it works perfectly with the "ban" remediation. However, when using "captcha" remediation, I get an error "500 Internal Server Error". When I use incognito mode, the captcha works but when the captcha is successfully solved, I get the error "500 Internal Server Error". Here are my configs:

crowdsec-nginx-bouncer.conf:

ENABLED=true
API_URL=http://crowdsec:8080
API_KEY=XXX
CACHE_EXPIRATION=1
# bounce for all type of remediation that the bouncer can receive from the local API
BOUNCING_ON_TYPE=all
FALLBACK_REMEDIATION=ban
REQUEST_TIMEOUT=3000
UPDATE_FREQUENCY=10
# live or stream
MODE=live
# exclude the bouncing on those location
EXCLUDE_LOCATION=
#those apply for "ban" action
# /!\ REDIRECT_LOCATION and RET_CODE can't be used together. REDIRECT_LOCATION take priority over RET_CODE
BAN_TEMPLATE_PATH=/var/lib/crowdsec/lua/templates/ban.html
REDIRECT_LOCATION=
RET_CODE=
#those apply for "captcha" action
#valid providers are recaptcha, hcaptcha, turnstile
CAPTCHA_PROVIDER=recaptcha
# Captcha Secret Key
SECRET_KEY=XXX
# Captcha Site key
SITE_KEY=XXX
CAPTCHA_TEMPLATE_PATH=/var/lib/crowdsec/lua/templates/captcha.html
CAPTCHA_EXPIRATION=3600

profiles.yaml:

name: captcha_remediation
filters: 
- Alert.Remediation == true && Alert.GetScope() == "Ip" && Alert.GetScenario() startsWith "crowdsecurity/http-"
decisions: 
- type: captcha
  duration: 4h
duration_expr: Sprintf('%dh', (GetDecisionsCount(Alert.GetValue()) + 1) * 4)
notifications:
 - discord
on_success: break

Nginx Error Log:
2023/10/11 13:49:37 [alert] 553#553: *10002 [lua] crowdsec.lua:542: Allow(): [Crowdsec] denied 'XXX' with 'captcha', client: XXX, server: request.*, request: "POST / HTTP/2.0", host: "XXX.duckdns.org", referrer: "https://XXX.duckdns.org/" 2023/10/11 13:49:40 [error] 553#553: *10002 lua entry thread aborted: runtime error: /usr/local/lua/crowdsec/crowdsec.lua:485: http2 requests are not supported yet stack traceback: coroutine 0: [C]: in function 'read_body' /usr/local/lua/crowdsec/crowdsec.lua:485: in function 'Allow' access_by_lua(http.d/crowdsec_nginx.conf:14):3: in main chunk, client: XXX, server: request.*, request: "GET /favicon.ico HTTP/2.0", host: "XXX.org", referrer: "https://XXX.duckdns.org/" 2023/10/11 13:49:43 [error] 553#553: *10002 lua entry thread aborted: runtime error: /usr/local/lua/crowdsec/crowdsec.lua:485: http2 requests are not supported yet stack traceback: coroutine 0: [C]: in function 'read_body' /usr/local/lua/crowdsec/crowdsec.lua:485: in function 'Allow' access_by_lua(http.d/crowdsec_nginx.conf:14):3: in main chunk, client: XXX, server: request.*, request: "POST / HTTP/2.0", host: "XXX.duckdns.org", referrer: "https://XXX.duckdns.org/" 2023/10/11 13:49:43 [error] 553#553: *10002 lua entry thread aborted: runtime error: /usr/local/lua/crowdsec/crowdsec.lua:485: http2 requests are not supported yet stack traceback: coroutine 0: [C]: in function 'read_body' /usr/local/lua/crowdsec/crowdsec.lua:485: in function 'Allow' access_by_lua(http.d/crowdsec_nginx.conf:14):3: in main chunk, client: XXX, server: request.*, request: "GET /favicon.ico HTTP/2.0", host: "XXX.duckdns.org", referrer: "https://XXX.duckdns.org/"

[LinuxServer-CI]

This issue has been automatically marked as stale because it has not had recent activity. This might be due to missing feedback from OP. It will be closed if no further activity occurs. Thank you for your contributions.

[Tivin-i]

Wonder why this is closed as this is still an ongoing issue.

[GuiPoM]

I can confirm the issue and this must be related to http2 support.
It should work when proxy is defining http, but I have currently none to test it.

[keesfluitman]

I've got the same error. Im using turnstile.

2024/01/08 13:25:42 [error] 965#965: *16281 lua entry thread aborted: runtime error: /usr/local/lua/crowdsec/crowdsec.lua:485: http2 requests are not supported yet
stack traceback:
coroutine 0:
	[C]: in function 'read_body'
	/usr/local/lua/crowdsec/crowdsec.lua:485: in function 'Allow'
	access_by_lua(http.d/crowdsec_nginx.conf:14):3: in main chunk while sending to client, client: 2001:1711:IPV6, server: domain.com, request: "GET /api/v2/homepage/netdata/data HTTP/2.0", host: "domain.com", referrer: "https://domain.com/"

Anything I could do?

[LaurenceJJones]

It seems they have added a check to see if the http2 request has a content length then accept the request

https://github.com/openresty/lua-nginx-module/pull/2237/files

I dont know which lua version swag is compiled with but if it becomes latest it should become supported for valid http requests with content length headers

[alleyu2]

It seems they have added a check to see if the http2 request has a content length then accept the request

https://github.com/openresty/lua-nginx-module/pull/2237/files

I dont know which lua version swag is compiled with but if it becomes latest it should become supported for valid http requests with content length headers

This is the version details for the latest SWAG container:

nginx -V nginx version: nginx/1.24.0 built with OpenSSL 3.1.3 19 Sep 2023 (running with OpenSSL 3.1.4 24 Oct 2023) TLS SNI support enabled configure arguments: --prefix=/var/lib/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib/nginx/modules --conf-path=/etc/nginx/nginx.conf --pid-path=/run/nginx/nginx.pid --lock-path=/run/nginx/nginx.lock --http-client-body-temp-path=/var/lib/nginx/tmp/client_body --http-proxy-temp-path=/var/lib/nginx/tmp/proxy --http-fastcgi-temp-path=/var/lib/nginx/tmp/fastcgi --http-uwsgi-temp-path=/var/lib/nginx/tmp/uwsgi --http-scgi-temp-path=/var/lib/nginx/tmp/scgi --with-perl_modules_path=/usr/lib/perl5/vendor_perl --user=nginx --group=nginx --with-threads --with-file-aio --without-pcre2 --with-http_ssl_module --with-http_v2_module --with-http_realip_module --with-http_addition_module --with-http_xslt_module=dynamic --with-http_image_filter_module=dynamic --with-http_geoip_module=dynamic --with-http_sub_module --with-http_dav_module --with-http_flv_module --with-http_mp4_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_auth_request_module --with-http_random_index_module --with-http_secure_link_module --with-http_degradation_module --with-http_slice_module --with-http_stub_status_module --with-http_perl_module=dynamic --with-mail=dynamic --with-mail_ssl_module --with-stream=dynamic --with-stream_ssl_module --with-stream_realip_module --with-stream_geoip_module=dynamic --with-stream_ssl_preread_module --add-dynamic-module=/home/buildozer/aports/main/nginx/src/njs-0.7.11/nginx --add-dynamic-module=/home/buildozer/aports/main/nginx/src/ngx_devel_kit-0.3.2/ --add-dynamic-module=/home/buildozer/aports/main/nginx/src/traffic-accounting-nginx-module-2.0/ --add-dynamic-module=/home/buildozer/aports/main/nginx/src/array-var-nginx-module-0.06/ --add-dynamic-module=/home/buildozer/aports/main/nginx/src/nginx-auth-jwt-0.2.1/ --add-dynamic-module=/home/buildozer/aports/main/nginx/src/ngx_brotli-1.0.0rc/ --add-dynamic-module=/home/buildozer/aports/main/nginx/src/ngx_cache_purge-2.5.3/ --add-dynamic-module=/home/buildozer/aports/main/nginx/src/nginx_cookie_flag_module-1.1.0/ --add-dynamic-module=/home/buildozer/aports/main/nginx/src/nginx-dav-ext-module-3.0.0/ --add-dynamic-module=/home/buildozer/aports/main/nginx/src/echo-nginx-module-0.63/ --add-dynamic-module=/home/buildozer/aports/main/nginx/src/encrypted-session-nginx-module-0.09/ --add-dynamic-module=/home/buildozer/aports/main/nginx/src/ngx-fancyindex-0.5.2/ --add-dynamic-module=/home/buildozer/aports/main/nginx/src/ngx_http_geoip2_module-3.4/ --add-dynamic-module=/home/buildozer/aports/main/nginx/src/headers-more-nginx-module-0.34/ --add-dynamic-module=/home/buildozer/aports/main/nginx/src/nginx-keyval-0.1.0/ --add-dynamic-module=/home/buildozer/aports/main/nginx/src/nginx-log-zmq-1.0.0/ --add-dynamic-module=/home/buildozer/aports/main/nginx/src/lua-nginx-module-0.10.24/ --add-dynamic-module=/home/buildozer/aports/main/nginx/src/lua-upstream-nginx-module-0.07/ --add-dynamic-module=/home/buildozer/aports/main/nginx/src/naxsi-1.3/naxsi_src --add-dynamic-module=/home/buildozer/aports/main/nginx/src/nchan-1.3.6/ --add-dynamic-module=/home/buildozer/aports/main/nginx/src/redis2-nginx-module-0.15/ --add-dynamic-module=/home/buildozer/aports/main/nginx/src/set-misc-nginx-module-0.33/ --add-dynamic-module=/home/buildozer/aports/main/nginx/src/nginx-http-shibboleth-2.0.1/ --add-dynamic-module=/home/buildozer/aports/main/nginx/src/ngx_http_untar_module-1.1/ --add-dynamic-module=/home/buildozer/aports/main/nginx/src/nginx-upload-module-2.3.0/ --add-dynamic-module=/home/buildozer/aports/main/nginx/src/nginx-upload-progress-module-0.9.2/ --add-dynamic-module=/home/buildozer/aports/main/nginx/src/nginx-upstream-fair-0.1.3/ --add-dynamic-module=/home/buildozer/aports/main/nginx/src/ngx_upstream_jdomain-1.4.0/ --add-dynamic-module=/home/buildozer/aports/main/nginx/src/nginx-vod-module-1.31/ --add-dynamic-module=/home/buildozer/aports/main/nginx/src/nginx-module-vts-0.2.1/ --add-dynamic-module=/home/buildozer/aports/main/nginx/src/mod_zip-1.3.0/ --add-dynamic-module=/home/buildozer/aports/main/nginx/src/nginx-rtmp-module-1.2.2/

[GuiPoM]

I was about to answer but you were faster @alleyu2

Latest NGINX version is 1.25.3 and contains many fixes for HTTP/2 and HTTP/3: https://nginx.org/en/CHANGES
I dont know what it the process to ask linuxserver to study this upgrade.

[aptalca]

We install from the alpine repo

[alleyu2]

I am still getting the error on latest SWAG:

2024/01/12 09:12:51 [error] 581#581: *134 lua entry thread aborted: runtime error: /usr/local/lua/crowdsec/crowdsec.lua:485: http2 requests are not supported yet

[LaurenceJJones]

Yes alpine edge and 3.19 have no been updated to latest and linuxserver uses 3.18 as base so one library has been updated we still need to wait for base image to be updated which I know is planned

[thespad]

We won't get nginx 1.25 until at least Alpine 3.20, but we should be able to get the lua mod for 1.24 backported to 3.19 once edge is updated.

[LinuxServer-CI]

This issue has been automatically marked as stale because it has not had recent activity. This might be due to missing feedback from OP. It will be closed if no further activity occurs. Thank you for your contributions.

[alleyu2]

Yes alpine edge and 3.19 have no been updated to latest and linuxserver uses 3.18 as base so one library has been updated we still need to wait for base image to be updated which I know is planned

Hi @thespad . Based on this "https://pkgs.alpinelinux.org/package/edge/main/x86_64/nginx-mod-http-lua" is lua updated to version v0.10.26 in alpine edge? if yes, can we backport it for SWAG?

[alleyu2]

Issue not closed

[LaurenceJJones]

I contacted the maintainer, and he said he will backport 0.10.26 to 3.18 and 3.19

[thespad]

You can track the commits here https://git.alpinelinux.org/aports/log/main/nginx?h=3.19-stable

This is the commit for Edge that bumped it to 0.10.26 a couple of weeks ago https://git.alpinelinux.org/aports/commit/main/nginx?id=239a9ba729dc42dbe264e33816f880d89cdd5398

-_add_module "http-lua" "v0.10.25" "https://github.com/openresty/lua-nginx-module" "" "$_has_luajit"
+_add_module "http-lua" "v0.10.26" "https://github.com/openresty/lua-nginx-module" "" "$_has_luajit"

[jclendineng]

Watching this closely as http2 is still broken in latest SWAG + Crowdsec

[drizuid]

Watching this closely as http2 is still broken in latest SWAG + Crowdsec

this is a broad statement, i believe only captcha is broke in crowdsec. Suggest reading the OP which explains it quite well.

to be clear, this is not something we can or will fix. once alpine backports the lua module, things will just work.

[alleyu2]

Watching this closely as http2 is still broken in latest SWAG + Crowdsec

this is a broad statement, i believe only captcha is broke in crowdsec. Suggest reading the OP which explains it quite well.

to be clear, this is not something we can or will fix. once alpine backports the lua module, things will just work.

I think it's Captcha and Appsec since both have dependency on lua module.

[jclendineng]

Watching this closely as http2 is still broken in latest SWAG + Crowdsec

this is a broad statement, i believe only captcha is broke in crowdsec. Suggest reading the OP which explains it quite well.

to be clear, this is not something we can or will fix. once alpine backports the lua module, things will just work.

Commented as I read OP and have the exact same issue ;) . Wouldn't have commented except this clearly broke with an update to SWAG as my configs have not changed, assuming a dependency bump caused it. Surprised this list a broader issue, but maybe not many people use SWAG with CS.

Edit. Removing swag-crowdsec will fix this until its officially resolved, you don't need to remove anything else.

[drizuid]

ignoring captcha, you can also set CROWDSEC_VERSION=v1.0.7 instead of using the default 1.0.8

[jclendineng]

ignoring captcha, you can also set CROWDSEC_VERSION=v1.0.7 instead of using the default 1.0.8

Awesome! Yes can confirm this works.

[LaurenceJJones]

Just to add a FYI, please if you are NOT using captcha remediation anymore please remove the configuration for it via the bouncer config. By design because CrowdSec lua acts as a MITM we have to check all requests if captcha is configured, and this may inadvertently cause this stack trace from version 1.0.8 onwards

[voc0der]

Is there a way we can see when this plugin updates? Just trying to keep tabs on when I can remove CROWDSEC_VERSION=v1.0.7 to prevent the latest issues with 500?

[drizuid]

you can monitor here https://pkgs.alpinelinux.org/packages?name=nginx-mod-http-lua&branch=v3.19&repo=&arch=&maintainer= im sure alpine also has changelogs... but this MOD itself wont show anything different. You have to track alpine.

[LaurenceJJones]

you can monitor here https://pkgs.alpinelinux.org/packages?name=nginx-mod-http-lua&branch=v3.19&repo=&arch=&maintainer= im sure alpine also has changelogs... but this MOD itself wont show anything different. You have to track alpine.

I don't know if it has been updated yet, but SWAG was based on alpine 3.18 so the packages need to be backported that far I believe

[thespad]

Yeah, we bumped to 3.19 at the end of January https://github.com/linuxserver/docker-swag?tab=readme-ov-file#versions

[LaurenceJJones]

FYI it seems the http2 content header fix broke a lot of things so they have reverted it back to the original it is not supported -> openresty/lua-nginx-module@e0d19f7

I think they are reaching the limitations of what lua can provide to Nginx so I dont know what to suggest as if the dependancy cant do it then neither can we.

So moving forward the official line from CrowdSec is if you want to use Captcha or AppSec component within Nginx then your webserver must not use http2 or higher, simply put... its out of our hands we have no control over it.

[thespad]

I'll look at adding something to that effect to the mod readme.

[alleyu2]

FYI it seems the http2 content header fix broke a lot of things so they have reverted it back to the original it is not supported -> openresty/lua-nginx-module@e0d19f7

I think they are reaching the limitations of what lua can provide to Nginx so I dont know what to suggest as if the dependancy cant do it then neither can we.

So moving forward the official line from CrowdSec is if you want to use Captcha or AppSec component within Nginx then your webserver must not use http2 or higher, simply put... its out of our hands we have no control over it.

I can close this ticket as it has reached a dead-end. Is there any alternative solution to get appsec or recpatcha working with nginx or we move on to another webserver to get these funtionlities working in crowdsec?

[drizuid]

if you are expecting an up to date reverse proxy to use http/1 i don't think you'll find it. However, this is nginx, just remove all the http2 refs... nothing magical here. honestly, i couldn't tell what your plan was anyway, it's never been our issue, never will be our issue, we were never going to do anything about it... and you closed/reopened this like 7 times anyway :p

[LaurenceJJones]

Just a further investigation, it seems they have now removed the restriction completely meaning in theory if you are using the master (or soon to be released version) it will now start working again, however, it important to read the note they added

Note: This method will block while waiting for client request body to be fully received. Block time depends on the client_body_timeout directive and maximum body size specified by the client_max_body_size directive. If read timeout occurs or client body size exceeds the defined limit, this function will not return and 408 Request Time-out or 413 Request Entity Too Large response will be returned to the client instead.

So if you are using http2/3 and the client does send a body without a content-length then you may start getting the above status codes as a response to our middleware

[thespad]

Blocking for 60 seconds per request seems like a great way to create a denial of service.

[LaurenceJJones]

Blocking for 60 seconds per request seems like a great way to create a denial of service.

Exactly, that why I said important to read the note 😆