FrameTooLargeError on batch.internet.nl with large batch requests

[stitch]

Since August 1st it's not possible to send in batch requests to a certain length to https://batch.internet.nl/api/batch/v2/requests. Sending it to the other endpoints works fine. This prevents running larger tests on the dashboard production environment.

The error received is:

jh2.exceptions.FrameTooLargeError: Cannot send frame size 16388, max frame size is 16384
urllib3.exceptions.ProtocolError: Cannot send frame size 16388, max frame size is 16384
requests.exceptions.ConnectionError: Cannot send frame size 16388, max frame size is 16384

The call is a simple batch request with just enough domains to exceed the request data limit. Code like this:

username = ""
password = ""

if __name__ == '__main__':
    url = "https://batch.internet.nl/api/batch/v2/requests"
    # url = "https://docker.batch.internet.nl/api/batch/v2/requests" -> werkt

    response = requests.post(
        url,
        json={"type": "web", "name": "", "domains": ["internet.nl"] * 2000},
        auth=HTTPBasicAuth(username, password),
        timeout=(300, 300),
    )
    print(response.json())

This issue only seems to be occurring in the production dashboard, since august 1st 2024. It can be tested with the above code with a longer list of domains setting the credentials.

[mxsasha]

So it was first seen on August 1st, but do we have an idea when it was not yet present? We don't really touch that server, so I'd like to determine a window where a change could have happened. As small as possible, of course :)

[bwbroersma]

The batch frontend shows version v1.7.2.dev1+gccd2cc1.d20240804.
So dated 2024-08-04 (Sunday almost 2 weeks ago).
Git hash ccd2cc1.

[mxsasha]

File modification times of most recently changes files in /opt/internet.nl/Internet.nl, excluding batch_results:

2023-10-10+11:56:26.4609281080	./.git/config
2023-10-10+11:56:26.4609281080	./.git/HEAD
2023-10-10+11:56:26.4609281080	./.git/logs/HEAD
2023-10-10+11:56:26.4609281080	./.git/logs/refs/heads/release/1.7.x
2023-10-10+11:56:26.4609281080	./.git/refs/heads/release/1.7.x
2023-10-10+12:00:07.6997302980	./.git/index
2024-02-14+14:56:40.6742088590	./unbound/.git/objects/pack/
<and many more unbound files>
2024-02-21+12:41:00.9485997660	./strace.out
2024-02-29+12:40:50.8957575130	./.venv/lib/python3.7/site-packages/sectxt-0.8.3.dist-info/LICENCE
2024-02-29+12:40:50.8957575130	./.venv/lib/python3.7/site-packages/sectxt-0.8.3.dist-info/METADATA
2024-02-29+12:40:50.8957575130	./.venv/lib/python3.7/site-packages/sectxt-0.8.3.dist-info/top_level.txt
2024-02-29+12:40:50.8957575130	./.venv/lib/python3.7/site-packages/sectxt-0.8.3.dist-info/WHEEL
2024-02-29+12:40:50.8957575130	./.venv/lib/python3.7/site-packages/sectxt-0.8.3.dist-info/zip-safe
2024-02-29+12:40:50.8957575130	./.venv/lib/python3.7/site-packages/sectxt/__init__.py
2024-02-29+12:40:50.8997575630	./.venv/lib/python3.7/site-packages/sectxt-0.8.3.dist-info/INSTALLER
2024-02-29+12:40:50.8997575630	./.venv/lib/python3.7/site-packages/sectxt/__pycache__/__init__.cpython-37.pyc
2024-02-29+12:40:50.9037576150	./.venv/lib/python3.7/site-packages/sectxt-0.8.3.dist-info/direct_url.json
2024-02-29+12:40:50.9037576150	./.venv/lib/python3.7/site-packages/sectxt-0.8.3.dist-info/RECORD
2024-02-29+12:40:50.9037576150	./.venv/lib/python3.7/site-packages/sectxt-0.8.3.dist-info/REQUESTED
2024-03-12+18:23:05.7895663330	./.well-known/pki-validation/EF1357377AF0CD3DE14402752774A06E.txt
2024-08-04+14:08:32.7560000000	./_unbound/etc/unbound/unbound.pid
2024-08-19+08:14:33.3866929670	./django.log
2024-08-19+14:08:41.0192627540	./remote_data/dns/root.key
2024-08-19+14:57:20.6533875590	./celerybeat-schedule

This is consistent with what I know: last commit was in October, unbound upgrade mid February, sectxt hotfix for leap day late February. The lat reboot was Aug 4, so that's the unbound pid change. My guess is that the version indicator in the frontend may include the unbound pid in the last modification date. last logs go back to Aug 2 only.

Very little happening in /etc as well this year, and no changes to Apache config since March, which probably was a cert update.

So no indicators from my end from the last months about a change that could be related to this.

I also can not reproduce it using the Python sample given above, i.e. with 2000 domains. Response I got:
{'request': {'name': '', 'submit_date': '2024-08-19T13:10:52.808776+00:00', 'finished_date': None, 'request_type': 'web', 'status': 'registering', 'request_id': 'df372c7fca844ecfb2bd463fed71add5'}, 'api_version': '2.3.0'}

[stitch]

This goes wrong after installing python -m pip install niquests

Perhaps a dependency issue of niquests

Collecting kiss-headers<4,>=2
  Downloading kiss_headers-2.4.3-py3-none-any.whl (43 kB)
     |████████████████████████████████| 43 kB 2.9 MB/s
Collecting wassima<2,>=1.0.1
  Downloading wassima-1.1.2-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (251 kB)
     |████████████████████████████████| 251 kB 51.4 MB/s
Collecting charset-normalizer<4,>=2
  Downloading charset_normalizer-3.3.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (142 kB)
     |████████████████████████████████| 142 kB 44.2 MB/s
Collecting urllib3-future<3,>=2.8.902
  Downloading urllib3_future-2.8.906-py3-none-any.whl (550 kB)
     |████████████████████████████████| 550 kB 22.9 MB/s
Requirement already satisfied: idna<4,>=2.5 in /usr/lib/python3/dist-packages (from niquests) (2.10)
Collecting h11<1.0.0,>=0.11.0
  Downloading h11-0.14.0-py3-none-any.whl (58 kB)
     |████████████████████████████████| 58 kB 8.1 MB/s
Collecting qh3<2.0.0,>=1.0.3
  Downloading qh3-1.0.9-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.9 MB)
     |████████████████████████████████| 1.9 MB 49.6 MB/s
Collecting jh2<6.0.0,>=5.0.3
  Downloading jh2-5.0.3-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.1 MB)
     |████████████████████████████████| 1.1 MB 25.3 MB/s

[bwbroersma]

There was a call with @baknu, ProLocation, @stitch and myself.
A system update of batch was done at 2024-08-04, therefor the version date changed (maybe the .d<date> should be removed from the version, since it's a bit confusing that this is the system start date).
The problem is because niquests, requires urllib3-future, which states:

⚠️ Installing urllib3.future shadows the actual urllib3 package (depending on installation order). The semver will always be like MAJOR.MINOR.9PP like 2.0.941, the patch node is always greater or equal to 900.

Therefor adding the niquest requirement to web-security-map made the normal request go over HTTP2, instead of HTTP1.1.
This (old) Apache 2.4.x has a HTTP2 MAX_FRAME_SIZE (see RFC 9113 - HTTP/2 - SETTINGS_MAX_FRAME_SIZE) of 16384 (16kB), the minimum value. While nginx uses the maximum default value of 16MB. Only in Apache 2.5.x this can be modified with H2MaxDataFrameLen.

So the shadowing isn't popular (see the issues page), another problem is that the HTTP1.1 => HTTP2 'upgrade' because of urllib3.future isn't pretty when talking to an 2.4.x Apache. I don't fully understand why the urllib3.future client doesn't have a chunked approach that respects the MAX_FRAME_SIZE.

[mxsasha]

A system update of batch was done at 2024-08-04, therefor the version date changed (maybe the .d<date> should be removed from the version, since it's a bit confusing that this is the system start date).

Per the docs, the date is only added when there have been local changes not in git. This is probably from when unbound updated files on restart. So not an issue in 1.8+

[stitch]

Fix by @Ousret applied. Thank you very much, the quick response is highly appreciated!