fix: pass headers when connecting sockets #1575

willydouhard · 2024-12-09T08:40:02Z

Fixing #1279 and closes #1359 as well. Tested it on GCP with session affinity + 2 instances with this docker file

# Build stage for Python application
FROM python:3.11-slim

# Install system dependencies
RUN apt-get update && apt-get install -y \
    git \
    nodejs \
    npm \
    && rm -rf /var/lib/apt/lists/*

# Install pnpm
RUN npm install -g pnpm

# Set working directory
WORKDIR /app

# Copy your application files
COPY app.py .

# Install Python dependencies
RUN pip install --no-cache-dir git+https://github.com/Chainlit/chainlit.git@willy/sticky-sessions#subdirectory=backend/

# Set environment variables
ENV PORT=8000
ENV HOST=0.0.0.0

# Expose the port
EXPOSE 8000

# Start the application
CMD ["chainlit", "run", "app.py", "--host", "0.0.0.0", "--port", "8000", "-h"]

willydouhard · 2024-12-09T11:23:30Z

This is failing for any origin that is the not the server's origin (typically Chainlit embedded as a copilot on a website) because withCredential does not support '*' cross origin.

Access to XMLHttpRequest at 'https://chainlit-sticky-182925357376.europe-west2.run.app/ws/socket.io/?EIO=4&transport=polling&t=PEh9E88' from origin 'http://localhost:9000/' has been blocked by CORS policy: The value of the 'Access-Control-Allow-Origin' header in the response must not be the wildcard '*' when the request's credentials mode is 'include'. The credentials mode of requests initiated by the XMLHttpRequest is controlled by the withCredentials attribute.

willydouhard · 2024-12-09T14:41:58Z

The issue is fixed. For cross origin use cases, it is now mandatory to allow it in the project config (I had to update the copilot test for example) as the wildcard "*" is not supported with credentials.

evantahler · 2024-12-09T15:49:14Z

libs/react-client/src/useChatSession.ts

@@ -100,6 +100,8 @@ const useChatSession = () => {

      const socket = io(uri, {
        path,
+        withCredentials: true,
+        transports: ['websocket'],


This change works for our use-case, but I believe adding the transports option here means that Socket.IO will not fall back to long-polling if a WS connection can't be established. There are probably still some load balancers/browsers out there that can't handle web-sockets properly. It might be helpful for these options to become chainlit initialization options that the user can select?

For instance the gcp load balancer only works with this. We can make it configurable I think.

willydouhard · 2024-12-09T16:51:04Z

libs/react-client/src/useChatSession.ts

-        }
+        withCredentials: true,
+        transports: ['websocket'],
+        auth: {


Had to move out from the extra header approach since the websocket transport does not support that.

willydouhard · 2024-12-10T08:22:12Z

backend/chainlit/config.py

@@ -311,6 +311,8 @@ class CodeSettings:
 @dataclass()
 class ProjectSettings(DataClassJsonMixin):
    allow_origins: List[str] = Field(default_factory=lambda: ["*"])
+    # Socket.io client transports option
+    transports: Optional[List[str]] = None


made the transports configurable @evantahler

willydouhard · 2024-12-10T08:46:49Z

Here is the doc PR Chainlit/docs#199

evantahler · 2024-12-10T17:33:17Z

I gave this branch a try within our application, but every request I made is rejected:

Clint WS message: {"message":"Connection rejected by server"}
Server: {"type":"LOG","log":{"level":"INFO","message":"Authentication failed"}}

We have a custom data layer, but I tried removing it with the same (failing) results. Any tips?

evantahler · 2024-12-10T17:36:06Z

Note: Poetry install line for pyproject.toml

chainlit = { git = "ssh://[email protected]/chainlit/chainlit.git", branch = "willy/sticky-sessions", subdirectory = "backend" }

dokterbob · 2024-12-10T17:51:15Z

Note: Poetry install line for pyproject.toml

chainlit = { git = "ssh://[email protected]/chainlit/chainlit.git", branch = "willy/sticky-sessions", subdirectory = "backend" }

Convenient one-liner - not requiring authentication/write access to the repo:

poetry add git+https://github.com/Chainlit/chainlit.git@willy/sticky-sessions#subdirectory=backend/

dokterbob · 2024-12-10T17:58:00Z

I gave this branch a try within our application, but every request I made is rejected:

Clint WS message: {"message":"Connection rejected by server"}

Server: {"type":"LOG","log":{"level":"INFO","message":"Authentication failed"}}

We have a custom data layer, but I tried removing it with the same (failing) results. Any tips?

Could you share a screendump of the browser's console and network requests from the debugger? Particularly the websocket initialisation process. Details on the first non-successful request from the network debugger would also help.

dokterbob · 2024-12-10T18:00:27Z

@dvdr00t This probably also solves #1407. Any chance you could give it a try? More eyes on this the better. :)

evantahler · 2024-12-10T18:02:24Z

Sure! We are using a FastAPI application & mount_chainlit

Server Logs:

10:00:54 api.1       | INFO:     127.0.0.1:57431 - "GET /api/v1/users/me HTTP/1.1" 200 OK
10:00:54 api.1       | INFO:     127.0.0.1:57431 - "GET /chat/auth/config HTTP/1.1" 200 OK
10:00:54 api.1       | {"type":"LOG","log":{"level":"WARN","message":"Translated markdown file for en-US not found. Defaulting to chainlit.md."}}
10:00:54 api.1       | INFO:     127.0.0.1:57431 - "GET /chat/auth/config HTTP/1.1" 200 OK
10:00:54 api.1       | INFO:     127.0.0.1:57434 - "GET /chat/project/settings?language=en-US HTTP/1.1" 200 OK
10:00:54 api.1       | INFO:     127.0.0.1:57434 - "GET /api/v1/chat/auth HTTP/1.1" 200 OK
10:00:55 api.1       | INFO:     127.0.0.1:57434 - "GET /api/v1/chat/auth HTTP/1.1" 200 OK
10:00:55 api.1       | {"type":"LOG","log":{"level":"WARN","message":"Translated markdown file for en-US not found. Defaulting to chainlit.md."}}
10:00:55 api.1       | INFO:     127.0.0.1:57434 - "GET /chat/auth/config HTTP/1.1" 200 OK
10:00:55 api.1       | INFO:     127.0.0.1:57431 - "GET /chat/project/settings?language=en-US HTTP/1.1" 200 OK
10:00:55 api.1       | {"type":"LOG","log":{"level":"WARN","message":"Translated markdown file for en-US not found. Defaulting to chainlit.md."}}
10:00:55 api.1       | INFO:     127.0.0.1:57434 - "GET /chat/auth/config HTTP/1.1" 200 OK
10:00:55 api.1       | INFO:     127.0.0.1:57431 - "GET /chat/project/settings?language=en-US HTTP/1.1" 200 OK
10:00:55 api.1       | INFO:     127.0.0.1:57434 - "POST /chat/project/threads HTTP/1.1" 200 OK
10:00:55 api.1       | INFO:     127.0.0.1:57431 - "POST /chat/project/threads HTTP/1.1" 200 OK
10:00:55 api.1       | INFO:     127.0.0.1:57431 - "OPTIONS /chat/ws/socket.io/?EIO=4&transport=polling&t=rp6pmpfx HTTP/1.1" 200 OK
10:00:55 api.1       | INFO:     127.0.0.1:57434 - "GET /chat/ws/socket.io/?EIO=4&transport=polling&t=rp6pmpfx HTTP/1.1" 200 OK
10:00:55 api.1       | INFO:     127.0.0.1:57434 - "OPTIONS /chat/ws/socket.io/?EIO=4&transport=polling&t=rp6qz0a3&sid=ochdon-Bc-f-hV5YAAAA HTTP/1.1" 200 OK
10:00:55 api.1       | INFO:     ('127.0.0.1', 57441) - "WebSocket /chat/ws/socket.io/?EIO=4&transport=websocket&sid=ochdon-Bc-f-hV5YAAAA" [accepted]
10:00:55 api.1       | INFO:     127.0.0.1:57431 - "OPTIONS /chat/ws/socket.io/?EIO=4&transport=polling&t=rp6r0pdd&sid=ochdon-Bc-f-hV5YAAAA HTTP/1.1" 200 OK
10:00:55 api.1       | INFO:     connection open
10:00:55 api.1       | {"type":"LOG","log":{"level":"INFO","message":"Authentication failed"}}
10:00:55 api.1       | INFO:     127.0.0.1:57443 - "POST /chat/ws/socket.io/?EIO=4&transport=polling&t=rp6qz0a3&sid=ochdon-Bc-f-hV5YAAAA HTTP/1.1" 200 OK
10:00:55 api.1       | INFO:     127.0.0.1:57434 - "GET /chat/ws/socket.io/?EIO=4&transport=polling&t=rp6r0pdd&sid=ochdon-Bc-f-hV5YAAAA HTTP/1.1" 200 OK
10:00:55 api.1       | INFO:     127.0.0.1:57434 - "OPTIONS /chat/ws/socket.io/?EIO=4&transport=polling&t=rp6td00j&sid=ochdon-Bc-f-hV5YAAAA HTTP/1.1" 200 OK
10:00:55 api.1       | INFO:     127.0.0.1:57443 - "GET /chat/ws/socket.io/?EIO=4&transport=polling&t=rp6td00j&sid=ochdon-Bc-f-hV5YAAAA HTTP/1.1" 200 OK
10:00:55 api.1       | INFO:     127.0.0.1:57443 - "OPTIONS /chat/ws/socket.io/?EIO=4&transport=polling&t=rp6v2btm&sid=ochdon-Bc-f-hV5YAAAA HTTP/1.1" 200 OK
10:00:55 api.1       | INFO:     127.0.0.1:57434 - "GET /chat/ws/socket.io/?EIO=4&transport=polling&t=rp6v2btm&sid=ochdon-Bc-f-hV5YAAAA HTTP/1.1" 200 OK
10:00:55 api.1       | INFO:     connection closed

Some screenshots from the browser console:

dokterbob · 2024-12-10T18:05:06Z

Incidentally, a prior PR for this issue was already open #1339. Should close it once we merge this.

dokterbob · 2024-12-10T18:06:49Z

Sure! We are using a FastAPI application & mount_chainlit

Server Logs:

10:00:54 api.1       | INFO:     127.0.0.1:57431 - "GET /api/v1/users/me HTTP/1.1" 200 OK
10:00:54 api.1       | INFO:     127.0.0.1:57431 - "GET /chat/auth/config HTTP/1.1" 200 OK
10:00:54 api.1       | {"type":"LOG","log":{"level":"WARN","message":"Translated markdown file for en-US not found. Defaulting to chainlit.md."}}
10:00:54 api.1       | INFO:     127.0.0.1:57431 - "GET /chat/auth/config HTTP/1.1" 200 OK
10:00:54 api.1       | INFO:     127.0.0.1:57434 - "GET /chat/project/settings?language=en-US HTTP/1.1" 200 OK
10:00:54 api.1       | INFO:     127.0.0.1:57434 - "GET /api/v1/chat/auth HTTP/1.1" 200 OK
10:00:55 api.1       | INFO:     127.0.0.1:57434 - "GET /api/v1/chat/auth HTTP/1.1" 200 OK
10:00:55 api.1       | {"type":"LOG","log":{"level":"WARN","message":"Translated markdown file for en-US not found. Defaulting to chainlit.md."}}
10:00:55 api.1       | INFO:     127.0.0.1:57434 - "GET /chat/auth/config HTTP/1.1" 200 OK
10:00:55 api.1       | INFO:     127.0.0.1:57431 - "GET /chat/project/settings?language=en-US HTTP/1.1" 200 OK
10:00:55 api.1       | {"type":"LOG","log":{"level":"WARN","message":"Translated markdown file for en-US not found. Defaulting to chainlit.md."}}
10:00:55 api.1       | INFO:     127.0.0.1:57434 - "GET /chat/auth/config HTTP/1.1" 200 OK
10:00:55 api.1       | INFO:     127.0.0.1:57431 - "GET /chat/project/settings?language=en-US HTTP/1.1" 200 OK
10:00:55 api.1       | INFO:     127.0.0.1:57434 - "POST /chat/project/threads HTTP/1.1" 200 OK
10:00:55 api.1       | INFO:     127.0.0.1:57431 - "POST /chat/project/threads HTTP/1.1" 200 OK
10:00:55 api.1       | INFO:     127.0.0.1:57431 - "OPTIONS /chat/ws/socket.io/?EIO=4&transport=polling&t=rp6pmpfx HTTP/1.1" 200 OK
10:00:55 api.1       | INFO:     127.0.0.1:57434 - "GET /chat/ws/socket.io/?EIO=4&transport=polling&t=rp6pmpfx HTTP/1.1" 200 OK
10:00:55 api.1       | INFO:     127.0.0.1:57434 - "OPTIONS /chat/ws/socket.io/?EIO=4&transport=polling&t=rp6qz0a3&sid=ochdon-Bc-f-hV5YAAAA HTTP/1.1" 200 OK
10:00:55 api.1       | INFO:     ('127.0.0.1', 57441) - "WebSocket /chat/ws/socket.io/?EIO=4&transport=websocket&sid=ochdon-Bc-f-hV5YAAAA" [accepted]
10:00:55 api.1       | INFO:     127.0.0.1:57431 - "OPTIONS /chat/ws/socket.io/?EIO=4&transport=polling&t=rp6r0pdd&sid=ochdon-Bc-f-hV5YAAAA HTTP/1.1" 200 OK
10:00:55 api.1       | INFO:     connection open
10:00:55 api.1       | {"type":"LOG","log":{"level":"INFO","message":"Authentication failed"}}
10:00:55 api.1       | INFO:     127.0.0.1:57443 - "POST /chat/ws/socket.io/?EIO=4&transport=polling&t=rp6qz0a3&sid=ochdon-Bc-f-hV5YAAAA HTTP/1.1" 200 OK
10:00:55 api.1       | INFO:     127.0.0.1:57434 - "GET /chat/ws/socket.io/?EIO=4&transport=polling&t=rp6r0pdd&sid=ochdon-Bc-f-hV5YAAAA HTTP/1.1" 200 OK
10:00:55 api.1       | INFO:     127.0.0.1:57434 - "OPTIONS /chat/ws/socket.io/?EIO=4&transport=polling&t=rp6td00j&sid=ochdon-Bc-f-hV5YAAAA HTTP/1.1" 200 OK
10:00:55 api.1       | INFO:     127.0.0.1:57443 - "GET /chat/ws/socket.io/?EIO=4&transport=polling&t=rp6td00j&sid=ochdon-Bc-f-hV5YAAAA HTTP/1.1" 200 OK
10:00:55 api.1       | INFO:     127.0.0.1:57443 - "OPTIONS /chat/ws/socket.io/?EIO=4&transport=polling&t=rp6v2btm&sid=ochdon-Bc-f-hV5YAAAA HTTP/1.1" 200 OK
10:00:55 api.1       | INFO:     127.0.0.1:57434 - "GET /chat/ws/socket.io/?EIO=4&transport=polling&t=rp6v2btm&sid=ochdon-Bc-f-hV5YAAAA HTTP/1.1" 200 OK
10:00:55 api.1       | INFO:     connection closed

Some screenshots from the browser console:

Thanks. That was quick! Could you send the browser's Console as well?

I am about to test with a similar setup for our own application RN.

dokterbob · 2024-12-10T18:09:14Z

@evantahler The auth error comes from here:

chainlit/backend/chainlit/socket.py

Line 117 in 8b2d4ba

logger.info("Authentication failed")

Could you remove the try/except there, or change it into logger.exception() to get the underlying Exception?

dokterbob · 2024-12-10T18:29:48Z

On my setup (no custom data layer, but using mount_chainlit(), with a single instance, I don't get authorization errors.

With two instances:

{"detail":"Unauthorized"}

This happens with OAuth authorisation, after logging out, when attempting to log back in. The request is handled by the same instance:

This seems a different error from the one you've been getting:

    # Check the state from the oauth provider against the browser cookie
    oauth_state = request.cookies.get("oauth_state")
    if oauth_state != state:
        raise HTTPException(
            status_code=status.HTTP_401_UNAUTHORIZED,
            detail="Unauthorized",
        )

evantahler · 2024-12-10T18:38:10Z

The browser console for this page is just tons of react errors about tooltips and mimetypes. Here's the output in a gist - https://gist.github.com/evantahler/171f6aa0a9f862c0122ec0728d8c430c

dokterbob · 2024-12-10T18:42:20Z

The browser console for this page is just tons of react errors about tooltips and mimetypes.

🤣

dokterbob · 2024-12-10T18:43:33Z

GTG now, happy to look again tomorrow. Any discriminating factors (what auth are you using) and/or feedback from other users greatly appreciated.

evantahler · 2024-12-10T18:45:39Z

@evantahler The auth error comes from here:

chainlit/backend/chainlit/socket.py

Line 117 in 8b2d4ba

logger.info("Authentication failed")

Could you remove the try/except there, or change it into logger.exception() to get the underlying Exception?

Looks like my auth is none:

10:45:00 api.1       | {"type":"LOG","log":{"level":"ERROR","message":"Authentication failed\nTraceback (most recent call last):\n
  File \"/Users/evan/Library/Caches/pypoetry/virtualenvs/app---U6Ca7R-py3.10/lib/python3.10/site-packages/socketio/async_server.py\", line 555, in _handle_connect\n
    success = await self._trigger_event(\n
  File \"/Users/evan/Library/Caches/pypoetry/virtualenvs/app---U6Ca7R-py3.10/lib/python3.10/site-packages/socketio/async_server.py\", line 637, in _trigger_event\n
    ret = await handler(*args)\n
TypeError: connect() missing 1 required positional argument: 'auth'\n\n
During handling of the above exception, another exception occurred:\n\n
Traceback (most recent call last):\n
  File \"/Users/evan/Library/Caches/pypoetry/virtualenvs/app---U6Ca7R-py3.10/lib/python3.10/site-packages/chainlit/socket.py\", line 97, in connect\n
    token = auth.get(\"token\")\n
AttributeError: 'NoneType' object has no attribute 'get'"}}

(I added some line breaks)

evantahler · 2024-12-10T18:49:06Z

Perhaps I'm still using an older version of the UI... any way to confirm that it has been re-compiled/re-built?

Edit:

I /think/ that the fact that the websocket request is not including any cookie header matches this

dokterbob · 2024-12-10T20:48:30Z

Perhaps I'm still using an older version of the UI... any way to confirm that it has been re-compiled/re-built?

Edit:

I /think/ that the fact that the websocket request is not including any cookie header matches this

Yes, that seems most likely. Can you share your Dockerfile?

Normally, poetry install should trigger npm buildUi, which builds the UI - then it copies it to chainlit/backend/chainlit/frontend/dist` from where chainlit serves it.

However, if you are running a chainlit install from the repo (as should be the case for source builds), AFAIK it serves from frontend/dist instead.

The obvious way is to add a console.log() somewhere. ;)

willydouhard · 2024-12-11T08:37:02Z

Most likely using an old version of the UI. The auth change happens in the react-client (which is supposed to be rebuilt as well by pnpm run buildUi).

dokterbob

I've just done repeated login/logout testing with 4 instances on GCP with OAuth. Used private browser and normal browser. No problems whatsoever. I think we should merge this and have users test it in a release candidate.!

dokterbob · 2024-12-09T21:07:16Z

cypress/e2e/copilot/.chainlit/config.toml

@@ -13,7 +13,7 @@ session_timeout = 3600
 cache = false

 # Authorized origins
-allow_origins = ["*"]
+allow_origins = ["http://127.0.0.1:8000"]


I love finally seeing this! :)

We should document that on a server users need to configure the origin.

dokterbob · 2024-12-09T21:08:51Z

backend/chainlit/socket.py

-    except StopIteration:
-        return str(uuid.uuid5(uuid.NAMESPACE_DNS, ip))
-
-


Happy to see this go! :)

dokterbob · 2024-12-09T21:09:44Z

backend/chainlit/socket.py

@@ -149,7 +132,7 @@ def emit_call_fn(event: Literal["ask", "call_fn"], data, timeout):
        user=user,
        token=token,
        chat_profile=chat_profile,
-        thread_id=environ.get("HTTP_X_CHAINLIT_THREAD_ID"),
+        thread_id=auth.get("threadId"),


Seems much cleaner here.

evantahler · 2024-12-12T15:51:02Z

I think we should merge this and have users test it in a release candidate!

Excellent! Can you comment here when that RC is ready to try out?

asvishnyakov · 2024-12-19T07:41:35Z

@willydouhard @dokterbob Just installed chainlit from main branch and got the same message about TypeError: connect() missing 1 required positional argument: 'auth'\n\n

UPD. I confirm downgrading to previous commit resolves the issue. poetry updates frontend in backend folder fine, I checked index file hash. Please investigate further.

UPD2. It looks like pip isn't propely downgrade packages, so if you install chainlit via git, you want to fully recreate environment in order to downgrade and fix an issue.

willydouhard · 2024-12-19T11:31:52Z

I am not able to reproduce, did you rebuild the entire UI including the react client? The auth change is actually in the react client which is a dependency of the main frontend.

willydouhard · 2024-12-19T17:40:09Z

The pre-release is available @evantahler @asvishnyakov.

pip3 install -U --pre chainlit

Let me know if it works on your end.

asvishnyakov · 2024-12-19T18:30:23Z

@willydouhard Thank you! This resolved the issue. Glad to be able use new version, it has amazing features!

evantahler · 2024-12-19T23:10:15Z

We are using a custom frontend (@chainlit/react-client), and it looks like a new version of the frontend hasn't been pushed to NPM yet https://www.npmjs.com/package/@chainlit/react-client?activeTab=versions

willydouhard · 2024-12-20T09:35:53Z

I just released it, version 0.2.1.

We are using a custom frontend (@chainlit/react-client), and it looks like a new version of the frontend hasn't been pushed to NPM yet https://www.npmjs.com/package/@chainlit/react-client?activeTab=versions

fix: pass headers when connecting sockets

cf235db

dosubot bot added the size:XS This PR changes 0-9 lines, ignoring generated files. label Dec 9, 2024

fix: remove task start/end in window_message

bae4782

dosubot bot added size:S This PR changes 10-29 lines, ignoring generated files. and removed size:XS This PR changes 0-9 lines, ignoring generated files. labels Dec 9, 2024

willydouhard added 4 commits December 9, 2024 13:45

fix: update cors

d5dc6c7

fix: cors

b50fd60

fix: update cors

96f68c9

fix: ci

b599372

fix: add transports: ['websocket']

7ad1706

evantahler reviewed Dec 9, 2024

View reviewed changes

fix: test

646d1bf

dosubot bot added size:M This PR changes 30-99 lines, ignoring generated files. and removed size:S This PR changes 10-29 lines, ignoring generated files. labels Dec 9, 2024

willydouhard commented Dec 9, 2024

View reviewed changes

willydouhard added 2 commits December 9, 2024 18:01

fix: lint

d1931bf

feat: make socket.io client transports option configurable

8272012

willydouhard commented Dec 10, 2024

View reviewed changes

dokterbob approved these changes Dec 12, 2024

View reviewed changes

dokterbob merged commit d531633 into main Dec 12, 2024
9 checks passed

dokterbob deleted the willy/sticky-sessions branch December 12, 2024 11:21

asvishnyakov mentioned this pull request Dec 19, 2024

chore: prepare release #1595

Merged

		except StopIteration:
		return str(uuid.uuid5(uuid.NAMESPACE_DNS, ip))

fix: pass headers when connecting sockets #1575

fix: pass headers when connecting sockets #1575

Conversation

willydouhard commented Dec 9, 2024 • edited by dokterbob Loading

willydouhard commented Dec 9, 2024

willydouhard commented Dec 9, 2024

evantahler Dec 9, 2024

Choose a reason for hiding this comment

willydouhard Dec 9, 2024

Choose a reason for hiding this comment

willydouhard Dec 9, 2024

Choose a reason for hiding this comment

willydouhard Dec 10, 2024

Choose a reason for hiding this comment

willydouhard commented Dec 10, 2024

evantahler commented Dec 10, 2024

evantahler commented Dec 10, 2024

dokterbob commented Dec 10, 2024

dokterbob commented Dec 10, 2024

dokterbob commented Dec 10, 2024

evantahler commented Dec 10, 2024

dokterbob commented Dec 10, 2024

dokterbob commented Dec 10, 2024

dokterbob commented Dec 10, 2024 • edited Loading

dokterbob commented Dec 10, 2024 • edited Loading

evantahler commented Dec 10, 2024

dokterbob commented Dec 10, 2024

dokterbob commented Dec 10, 2024

evantahler commented Dec 10, 2024 • edited Loading

evantahler commented Dec 10, 2024 • edited Loading

dokterbob commented Dec 10, 2024

willydouhard commented Dec 11, 2024

dokterbob left a comment

Choose a reason for hiding this comment

dokterbob Dec 9, 2024

Choose a reason for hiding this comment

dokterbob Dec 9, 2024

Choose a reason for hiding this comment

dokterbob Dec 9, 2024

Choose a reason for hiding this comment

evantahler commented Dec 12, 2024

asvishnyakov commented Dec 19, 2024 • edited Loading

willydouhard commented Dec 19, 2024

willydouhard commented Dec 19, 2024

asvishnyakov commented Dec 19, 2024 • edited Loading

evantahler commented Dec 19, 2024

willydouhard commented Dec 20, 2024

willydouhard commented Dec 9, 2024 •

edited by dokterbob

Loading

dokterbob commented Dec 10, 2024 •

edited

Loading

dokterbob commented Dec 10, 2024 •

edited

Loading

evantahler commented Dec 10, 2024 •

edited

Loading

evantahler commented Dec 10, 2024 •

edited

Loading

asvishnyakov commented Dec 19, 2024 •

edited

Loading

asvishnyakov commented Dec 19, 2024 •

edited

Loading