"Ambiguous URI path encoding" when upgrading to Jetty 12 EE8 #12162

vmassol · 2024-08-13T15:17:39Z

Jetty Version

Jetty 12.0.12

Jetty Environment
EE8

Java Version
17

Question

On XWiki, I'm trying to upgrade from Jetty 10 to Jetty 12.0.12 and one of our tests is failing when http://localhost:8080/xwiki/rest/wikis/xwiki/spaces/AttachmentsResourceIT/pages/testPUTGETAttachments/attachments/%25percent.txt is called. It leads to a 400 error with the message "Ambiguous URI path encoding".

The problem is that we've set jetty.httpConfig.uriCompliance=RFC3986 (we had that in Jetty 10 already) but it's not helping.

I've read:

I'm still puzzled and don't understand what we can do to allow % to be passed in URLs in Jetty 12. In the threads above it's mentioned that the issue is that Servlet 6 spec doesn't allow it. That's fine but I don't fully understand why the RFC3986 compliance doesn't allow it (I've also tried other values like jetty.httpConfig.uriCompliance=RFC3986,AMBIGUOUS_PATH_ENCODING,AMBIGUOUS_PATH_SEPARATOR), but more importantly since we've configured Jetty to use EE8 (ie Servlet 4), I don't understand why we would be subject to Servlet 6 rules :)

Any idea?

Thanks a lot

The text was updated successfully, but these errors were encountered:

joakime · 2024-08-13T17:20:26Z

The reason Servlet 6 rejects these sequences is that is causes bugs in Servlet land that cannot be resolved or fixed without big changes to the Servlet API.

When things are ambiguous in the path section, then things like security constraints, url-patterns, redirect location logic, and various values on HttpServletRequest are bad and you cannot tell the difference between someone using a raw % or encoded %25 or double-encoded %2525 sequence. Allowing %25 can, for example, allowed a bypass into the WEB-INF directory, via requests like /%2557EB-INF/one.js%23/ simply because of how requests are simultaneously encoded and decoded in the HttpServletRequest object depending on the API in use (fixed in Jetty by finding all WEB-INF directories in the base resource tree and rejecting access to those specific paths via the ResourceService after the requested resource has been resolved). Similar bypasses exist for security constraints and url-patterns that cannot be resolved in Servlet land. Hence the new rules in Servlet 6 to just reject those ambiguous character sequences.

These bugs also exist in all prior versions of Servlet too, and it is not possible to resolve them.
You have bugs if you accept /%25percent.txt on the path, ones that cannot be resolved while in Servlet land.

Regarding your specific case, that seems to be triggering the UriCompliance.Violation.AMBIGUOUS_PATH_ENCODING specific violation.

Using jetty.httpConfig.uriCompliance=RFC3986,AMBIGUOUS_PATH_ENCODING should be enough.

Confirmed on fresh jetty-base running Jetty 12.0.12 with http and ee8-demos installed.

Default behavior

$ java -jar ~/dists/jetty-home-12.0.12/start.jar
...(snip startup)..

# In different terminal
$ echo -e "GET /test/%25foo.txt HTTP/1.0\r\n\r\n" | nc -vvv localhost 8080
Connection to localhost (::1) 8080 port [tcp/http-alt] succeeded!
HTTP/1.1 400 Bad Request
Server: Jetty(12.0.12)
Cache-Control: must-revalidate,no-cache,no-store
Content-Type: text/html;charset=iso-8859-1
Content-Length: 452

<html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 400 Ambiguous URI path encoding</title>
</head>
<body>
<h2>HTTP ERROR 400 Ambiguous URI path encoding</h2>
<table>
<tr><th>URI:</th><td>/badURI</td></tr>
<tr><th>STATUS:</th><td>400</td></tr>
<tr><th>MESSAGE:</th><td>Ambiguous URI path encoding</td></tr>
</table>
<hr/><a href="https://jetty.org/">Powered by Jetty:// 12.0.12</a><hr/>

</body>
</html>

Using configuration to allow it.

$ java -jar ~/dists/jetty-home-12.0.12/start.jar jetty.httpConfig.uriCompliance=RFC3986,AMBIGUOUS_PATH_ENCODING
... (snip lots of output) ...

# In different terminal
$ echo -e "GET /test/%25foo.txt HTTP/1.0\r\n\r\n" | nc -vvv localhost 8080
Connection to localhost (::1) 8080 port [tcp/http-alt] succeeded!
HTTP/1.1 404 Not Found
Server: Jetty(12.0.12)
Set-Cookie: visited=yes
Cache-Control: must-revalidate,no-cache,no-store
Content-Type: text/html;charset=iso-8859-1
Content-Length: 449

<html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 404 Not Found</title>
</head>
<body><h2>HTTP ERROR 404 Not Found</h2>
<table>
<tr><th>URI:</th><td>/test/%25foo.txt</td></tr>
<tr><th>STATUS:</th><td>404</td></tr>
<tr><th>MESSAGE:</th><td>Not Found</td></tr>
<tr><th>SERVLET:</th><td>default</td></tr>
</table>
<hr/><a href="https://jetty.org/">Powered by Jetty:// 12.0.12</a><hr/>

</body>
</html>

joakime · 2024-08-13T17:21:16Z

The behavior you are experiencing is coming from org.eclipse.jetty.http.HttpURI btw.

package uri;

import java.net.URI;

import org.eclipse.jetty.http.HttpURI;
import org.eclipse.jetty.http.UriCompliance;

public class JettyUriTest
{
    public static void main(String[] args)
    {
        String name = "5boards.txt";
        dump("http://localhost/foo/%" + name);
        dump("http://localhost/foo/%25" + name);
        dump("http://localhost/foo/%2525" + name);
    }

    public static void dump(String rawinput)
    {
        System.out.println("\n--- raw input \"" + rawinput + "\" ---");
        dump(HttpURI.build(rawinput));
        dump(URI.create(rawinput));
    }

    private static void dump(HttpURI uri)
    {
        System.out.println("Jetty HttpURI = " + uri);
        System.out.println("  .toString = " + uri.toString());
//        System.out.println("  .isAbsolute = " + uri.isAbsolute());
//        System.out.println("  .scheme = " + uri.getScheme());
//        System.out.println("  .authority = " + uri.getAuthority());
//        System.out.println("  .user = " + uri.getUser());
//        System.out.println("  .host = " + uri.getHost());
//        System.out.println("  .port = " + uri.getPort());
        System.out.println("  .path: " + uri.getPath());
        System.out.println("  .decodedPath: " + uri.getDecodedPath());
        if (uri.hasViolations())
        {
            System.out.println("  -VIOLATIONS-");
            for (UriCompliance.Violation violation : uri.getViolations())
            {
                System.out.println("  * violation = " + violation);
            }
        }
    }

    private static void dump(URI uri)
    {
        System.out.println("java.net.URI = " + uri);
        System.out.println("  .toASCIIString = " + uri.toASCIIString());
//        System.out.println("  .isAbsolute = " + uri.isAbsolute());
//        System.out.println("  .scheme = " + uri.getScheme());
//        System.out.println("  .authority = " + uri.getAuthority());
//        System.out.println("  .userInfo = " + uri.getUserInfo());
//        System.out.println("  .host = " + uri.getHost());
//        System.out.println("  .port = " + uri.getPort());
        System.out.println("  .rawPath: " + uri.getRawPath());
        System.out.println("  .path = " + uri.getPath());
        System.out.println("  .schemeSpecificPart = " + uri.getSchemeSpecificPart());
        System.out.println("  .normalize = " + uri.normalize());
    }
}

With the output ...

--- raw input "http://localhost/foo/%5boards.txt" ---
Jetty HttpURI = http://localhost/foo/%5boards.txt
  .toString = http://localhost/foo/%5boards.txt
  .path: /foo/%5boards.txt
  .decodedPath: /foo/[oards.txt
java.net.URI = http://localhost/foo/%5boards.txt
  .toASCIIString = http://localhost/foo/%5boards.txt
  .rawPath: /foo/%5boards.txt
  .path = /foo/[oards.txt
  .schemeSpecificPart = //localhost/foo/[oards.txt
  .normalize = http://localhost/foo/%5boards.txt

--- raw input "http://localhost/foo/%255boards.txt" ---
Jetty HttpURI = http://localhost/foo/%255boards.txt
  .toString = http://localhost/foo/%255boards.txt
  .path: /foo/%255boards.txt
  .decodedPath: /foo/%5boards.txt
  -VIOLATIONS-
  * violation = AMBIGUOUS_PATH_ENCODING
java.net.URI = http://localhost/foo/%255boards.txt
  .toASCIIString = http://localhost/foo/%255boards.txt
  .rawPath: /foo/%255boards.txt
  .path = /foo/%5boards.txt
  .schemeSpecificPart = //localhost/foo/%5boards.txt
  .normalize = http://localhost/foo/%255boards.txt

--- raw input "http://localhost/foo/%25255boards.txt" ---
Jetty HttpURI = http://localhost/foo/%25255boards.txt
  .toString = http://localhost/foo/%25255boards.txt
  .path: /foo/%25255boards.txt
  .decodedPath: /foo/%255boards.txt
  -VIOLATIONS-
  * violation = AMBIGUOUS_PATH_ENCODING
java.net.URI = http://localhost/foo/%25255boards.txt
  .toASCIIString = http://localhost/foo/%25255boards.txt
  .rawPath: /foo/%25255boards.txt
  .path = /foo/%255boards.txt
  .schemeSpecificPart = //localhost/foo/%255boards.txt
  .normalize = http://localhost/foo/%25255boards.txt

joakime · 2024-08-13T17:24:01Z

Demo of this setup in Jetty 12 embedded.

package uri;

import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.net.Socket;
import java.net.URI;
import java.nio.charset.StandardCharsets;

import org.eclipse.jetty.http.HttpHeader;
import org.eclipse.jetty.http.UriCompliance;
import org.eclipse.jetty.io.Content;
import org.eclipse.jetty.server.Handler;
import org.eclipse.jetty.server.HttpConfiguration;
import org.eclipse.jetty.server.HttpConnectionFactory;
import org.eclipse.jetty.server.Request;
import org.eclipse.jetty.server.Response;
import org.eclipse.jetty.server.Server;
import org.eclipse.jetty.server.ServerConnector;
import org.eclipse.jetty.util.Callback;
import org.eclipse.jetty.util.IO;
import org.eclipse.jetty.util.component.LifeCycle;

public class JettyEncodedPercentDemo
{
    public static void main(String[] args) throws Exception
    {
        Server server = new Server();

        HttpConfiguration httpConfiguration = new HttpConfiguration();
        UriCompliance specViolatingCompliance = UriCompliance.from("RFC3986,AMBIGUOUS_PATH_ENCODING");
        httpConfiguration.setUriCompliance(specViolatingCompliance);
        ServerConnector connector = new ServerConnector(server, new HttpConnectionFactory(httpConfiguration));
        connector.setPort(8080);
        server.addConnector(connector);

        Handler.Abstract handler = new Handler.Abstract()
        {
            @Override
            public boolean handle(Request request, Response response, Callback callback)
            {
                response.setStatus(200);
                response.getHeaders().add(HttpHeader.CONTENT_TYPE, "text/plain;charset=utf-8");
                StringBuilder str = new StringBuilder();
                str.append("Got Raw URI: ").append(request.getHttpURI().toString());
                str.append("\nDecoded Path: ").append(request.getHttpURI().getDecodedPath());
                Content.Sink.write(response, true, str.toString(), callback);
                return true;
            }
        };

        server.setHandler(handler);
        try
        {
            server.start();

            URI serverURI = server.getURI();
            String name = "5boards.txt";
            sendRequest(serverURI.getHost(), serverURI.getPort(), "/foo/%" + name);
            sendRequest(serverURI.getHost(), serverURI.getPort(), "/foo/%25" + name);
            sendRequest(serverURI.getHost(), serverURI.getPort(), "/foo/%2525" + name);
        }
        finally
        {
            LifeCycle.stop(server);
        }
    }

    private static void sendRequest(String host, int port, String path) throws IOException
    {
        try (Socket client = new Socket(host, port);
             OutputStream out = client.getOutputStream();
             InputStream in = client.getInputStream())
        {
            String rawRequest = """
                GET %s HTTP/1.1
                Host: %s:%d
                Connection: close
                
                """.formatted(path, host, port);
            System.out.println("\n----- REQUEST -----\n" + rawRequest);
            out.write(rawRequest.getBytes(StandardCharsets.UTF_8));
            out.flush();

            String rawResponse = IO.toString(in, StandardCharsets.UTF_8);
            System.out.println("----- RESPONSE -----\n" + rawResponse);
        }
    }
}

With output ...

2024-08-13 12:23:18.933:INFO :oejs.Server:main: jetty-12.0.12; built: 2024-07-25T21:58:37.668Z; git: cc6f1b74db755fed228b50701ad967aeaa68e83f; jvm 21.0.3+9-LTS
2024-08-13 12:23:18.978:INFO :oejs.AbstractConnector:main: Started ServerConnector@1f554b06{HTTP/1.1, (http/1.1)}{0.0.0.0:8080}
2024-08-13 12:23:18.996:INFO :oejs.Server:main: Started oejs.Server@27efef64{STARTING}[12.0.12,sto=0] @416ms

----- REQUEST -----
GET /foo/%5boards.txt HTTP/1.1
Host: 192.168.1.215:8080
Connection: close


----- RESPONSE -----
HTTP/1.1 200 OK
Server: Jetty(12.0.12)
Date: Tue, 13 Aug 2024 17:23:19 GMT
Content-Type: text/plain;charset=utf-8
Content-Length: 85
Connection: close

Got Raw URI: http://192.168.1.215:8080/foo/%5boards.txt
Decoded Path: /foo/[oards.txt

----- REQUEST -----
GET /foo/%255boards.txt HTTP/1.1
Host: 192.168.1.215:8080
Connection: close


----- RESPONSE -----
HTTP/1.1 200 OK
Server: Jetty(12.0.12)
Date: Tue, 13 Aug 2024 17:23:19 GMT
Content-Type: text/plain;charset=utf-8
Content-Length: 89
Connection: close

Got Raw URI: http://192.168.1.215:8080/foo/%255boards.txt
Decoded Path: /foo/%5boards.txt

----- REQUEST -----
GET /foo/%25255boards.txt HTTP/1.1
Host: 192.168.1.215:8080
Connection: close


----- RESPONSE -----
HTTP/1.1 200 OK
Server: Jetty(12.0.12)
Date: Tue, 13 Aug 2024 17:23:19 GMT
Content-Type: text/plain;charset=utf-8
Content-Length: 93
Connection: close

Got Raw URI: http://192.168.1.215:8080/foo/%25255boards.txt
Decoded Path: /foo/%255boards.txt
2024-08-13 12:23:19.107:INFO :oejs.Server:main: Stopped oejs.Server@27efef64{STOPPING}[12.0.12,sto=0]
2024-08-13 12:23:19.111:INFO :oejs.AbstractConnector:main: Stopped ServerConnector@1f554b06{HTTP/1.1, (http/1.1)}{0.0.0.0:8080}

vmassol · 2024-08-14T11:47:13Z

Thanks a lot @joakime for your answer (and a fast one), much appreciated :)

The reason Servlet 6 rejects these sequences is that is causes bugs in Servlet land that cannot be resolved or fixed without big changes to the Servlet API.

Ok, but here since I'm using EE8, I'm on Servlet 4 so the rules for Servlet 6 shouldn't apply, right? Aren't they the same rules that were used in Jetty 10?

Regarding your specific case, that seems to be triggering the UriCompliance.Violation.AMBIGUOUS_PATH_ENCODING specific violation.

Using jetty.httpConfig.uriCompliance=RFC3986,AMBIGUOUS_PATH_ENCODING should be enough.

Unfortunately it didn't help. I had tried a lot of values, including using UNSAFE. It always fails as if the config property is not used. I checked and re-checked how the property is used but it looks good; in jetty.xml there's <Set name="uriCompliance"><Call class="org.eclipse.jetty.http.UriCompliance" name="from"><Arg><Property name="jetty.httpConfig.uriCompliance" default="DEFAULT"/></Arg></Call></Set> and then in a xwiki.ini file, I have jetty.httpConfig.uriCompliance=RFC3986,AMBIGUOUS_PATH_ENCODING (I've updated it locally to add AMBIGUOUS_PATH_ENCODING).

I'm still getting:

HTTP ERROR 400 Ambiguous URI path encoding

URI: /badURI
STATUS: 400
MESSAGE: Ambiguous URI path encoding

Any idea of what I could try?

joakime · 2024-08-14T13:26:03Z

Thanks a lot @joakime for your answer (and a fast one), much appreciated :)

twas an easy one to answer too, and I happened to have these demo projects already written. :)

The reason Servlet 6 rejects these sequences is that is causes bugs in Servlet land that cannot be resolved or fixed without big changes to the Servlet API.

Ok, but here since I'm using EE8, I'm on Servlet 4 so the rules for Servlet 6 shouldn't apply, right? Aren't they the same rules that were used in Jetty 10?

The bugs exists in the Servlet API (no specific version).
The bug is that any ambiguous encoding in the path portion of the URL/URI cannot be properly decode / resolved / made available in the APIs / re-encoded due to the API calls / etc ...

The "any ambiguous encoding in the path portion" is any encoded character that impacts URI reserved characters or the % symbol itself. (There is also an ambiguous path segment one where the path segment is empty and has nothing to do with encoding)
https://datatracker.ietf.org/doc/html/rfc3986#section-2.2

The bug is not unique to Servlet 6, it exists all the way back to Servlet 1.0 too.

Regarding your specific case, that seems to be triggering the UriCompliance.Violation.AMBIGUOUS_PATH_ENCODING specific violation.
Using jetty.httpConfig.uriCompliance=RFC3986,AMBIGUOUS_PATH_ENCODING should be enough.

Unfortunately it didn't help. I had tried a lot of values, including using UNSAFE. It always fails as if the config property is not used. I checked and re-checked how the property is used but it looks good; in jetty.xml there's <Set name="uriCompliance"><Call class="org.eclipse.jetty.http.UriCompliance" name="from"><Arg><Property name="jetty.httpConfig.uriCompliance" default="DEFAULT"/></Arg></Call></Set> and then in a xwiki.ini file, I have jetty.httpConfig.uriCompliance=RFC3986,AMBIGUOUS_PATH_ENCODING (I've updated it locally to add AMBIGUOUS_PATH_ENCODING).

I'm still getting:

HTTP ERROR 400 Ambiguous URI path encoding
URI: /badURI
STATUS: 400
MESSAGE: Ambiguous URI path encoding

Any idea of what I could try?

The "Ambiguous URI path encoding" message applies to all ambiguous path issues.
Which includes ...

The fact that you see /badURI here is because the failure is detected early, well before it is even dispatched to your webapp.

I suspect you are not setting that configuration either in the right place (so it isn't applied to the same connector you are testing), or the right time (so it isn't applied), or something else.

I was going to have you look at the HttpConfiguration at runtime, but discovered those configurations are not present in either the dump or JMX (grumble grumble), so I opened #12163
Any chance you can look at the HttpConfiguration in use via a debugger?

vmassol · 2024-08-14T14:01:31Z

Any chance you can look at the HttpConfiguration in use via a debugger?

Seems you're right and it's not set (DEFAULT is used somehow):

Now need to find why...

vmassol · 2024-08-14T14:09:44Z

Works if I modify jetty.xml from <Set name="uriCompliance"><Call class="org.eclipse.jetty.http.UriCompliance" name="from"><Arg><Property name="jetty.httpConfig.uriCompliance" default="DEFAULT"/></Arg></Call></Set> to <Set name="uriCompliance"><Call class="org.eclipse.jetty.http.UriCompliance" name="from"><Arg>UNSAFE</Arg></Call></Set>, so the issue is the passing of the property from our xwiki.ini file somehow.

vmassol · 2024-08-14T14:12:53Z

How can I know if the xwiki.ini file located in start.d is used?

vmassol · 2024-08-14T14:14:22Z

How can I know if the xwiki.ini file located in start.d is used?

I guess it's used since otherwise the --module=xwiki command wouldn't be used and the xwiki.mod wouldn't be used and Jetty wouldn't deploy XWiki.

So maybe the issue is just the syntax for defining a property in xwiki.ini?

https://github.com/xwiki/xwiki-platform/blob/477dbd0edd0572cd3102c7355308a1c7c9f58596/xwiki-platform-tools/xwiki-platform-tool-jetty/xwiki-platform-tool-jetty-resources/src/main/resources/start.d/xwiki.ini

vmassol · 2024-08-14T14:14:56Z

Knowing that this used to work in Jetty 10, maybe there's been a change in Jetty 12?

vmassol · 2024-08-14T14:23:02Z

If I remove our xwiki.ini and try to regenerate it with java --add-opens java.base/java.lang=ALL-UNNAMED --add-opens java.base/java.io=ALL-UNNAMED --add-opens java.base/java.util=ALL-UNNAMED --add-opens java.base/java.util.concurrent=ALL-UNNAMED -Djava.io.tmpdir=./tmp -Djetty.home=jetty -Djetty.base=. -Dfile.encoding=UTF8 -Djetty.http.port=8080 -jar jetty/start.jar --add-modules=xwiki jetty.http.port=8080 STOP.KEY=xwiki STOP.PORT=8079, I get some warnings/errors (no idea if that matters):

INFO  : server          transitively enabled, ini template available with --add-modules=server
INFO  : ee8-plus        transitively enabled
INFO  : ee-webapp       transitively enabled, ini template available with --add-modules=ee-webapp
INFO  : ee8-websocket-jetty transitively enabled
INFO  : threadpool      transitively enabled, ini template available with --add-modules=threadpool
INFO  : ee8-annotations transitively enabled
INFO  : deploy          transitively enabled
INFO  : security        transitively enabled
INFO  : ee8-servlet     transitively enabled
INFO  : console-capture transitively enabled, ini template available with --add-modules=console-capture
INFO  : client          transitively enabled
INFO  : bytebufferpool  transitively enabled, ini template available with --add-modules=bytebufferpool
INFO  : ext             transitively enabled
INFO  : ee8-deploy      transitively enabled, ini template available with --add-modules=ee8-deploy
INFO  : sessions        transitively enabled, ini template available with --add-modules=sessions
INFO  : ee8-webapp      transitively enabled, ini template available with --add-modules=ee8-webapp
INFO  : requestlog      transitively enabled, ini template available with --add-modules=requestlog
INFO  : ee8-apache-jsp  transitively enabled
INFO  : http-forwarded  transitively enabled, ini template available with --add-modules=http-forwarded
INFO  : resources       transitively enabled
INFO  : plus            transitively enabled
INFO  : ee8-security    transitively enabled
INFO  : xwiki-logging   transitively enabled
INFO  : ee8-websocket-javax transitively enabled
INFO  : jndi            transitively enabled
INFO  : http            transitively enabled, ini template available with --add-modules=http
INFO  : xwiki           initialized in /Users/vmassol/dev/xwiki/xwiki-platform/xwiki-platform-core/xwiki-platform-rest/xwiki-platform-rest-test/xwiki-platform-rest-test-tests/target/xwiki/start.d/xwiki.ini
INFO  : logging/slf4j   dynamic dependency of xwiki-logging
java.io.IOException: For security reasons, Jetty start is unable to process file resource not in ${jetty.base} - webapps/
        at org.eclipse.jetty.start.FileInitializer.getDestination(FileInitializer.java:114)
        at org.eclipse.jetty.start.fileinits.LocalFileInitializer.create(LocalFileInitializer.java:36)
        at org.eclipse.jetty.start.BaseBuilder.processFileResource(BaseBuilder.java:370)
        at org.eclipse.jetty.start.BaseBuilder.processFileResources(BaseBuilder.java:407)
        at org.eclipse.jetty.start.BaseBuilder.build(BaseBuilder.java:338)
        at org.eclipse.jetty.start.Main.start(Main.java:551)
        at org.eclipse.jetty.start.Main.main(Main.java:82)
java.io.IOException: For security reasons, Jetty start is unable to process file resource not in ${jetty.base} - logs/
        at org.eclipse.jetty.start.FileInitializer.getDestination(FileInitializer.java:114)
        at org.eclipse.jetty.start.fileinits.LocalFileInitializer.create(LocalFileInitializer.java:36)
        at org.eclipse.jetty.start.BaseBuilder.processFileResource(BaseBuilder.java:370)
        at org.eclipse.jetty.start.BaseBuilder.processFileResources(BaseBuilder.java:407)
        at org.eclipse.jetty.start.BaseBuilder.build(BaseBuilder.java:338)
        at org.eclipse.jetty.start.Main.start(Main.java:551)
        at org.eclipse.jetty.start.Main.main(Main.java:82)
java.io.IOException: For security reasons, Jetty start is unable to process file resource not in ${jetty.base} - lib/
        at org.eclipse.jetty.start.FileInitializer.getDestination(FileInitializer.java:114)
        at org.eclipse.jetty.start.fileinits.LocalFileInitializer.create(LocalFileInitializer.java:36)
        at org.eclipse.jetty.start.BaseBuilder.processFileResource(BaseBuilder.java:370)
        at org.eclipse.jetty.start.BaseBuilder.processFileResources(BaseBuilder.java:407)
        at org.eclipse.jetty.start.BaseBuilder.build(BaseBuilder.java:338)
        at org.eclipse.jetty.start.Main.start(Main.java:551)
        at org.eclipse.jetty.start.Main.main(Main.java:82)
java.io.IOException: For security reasons, Jetty start is unable to process file resource not in ${jetty.base} - lib/ext/
        at org.eclipse.jetty.start.FileInitializer.getDestination(FileInitializer.java:114)
        at org.eclipse.jetty.start.fileinits.LocalFileInitializer.create(LocalFileInitializer.java:36)
        at org.eclipse.jetty.start.BaseBuilder.processFileResource(BaseBuilder.java:370)
        at org.eclipse.jetty.start.BaseBuilder.processFileResources(BaseBuilder.java:407)
        at org.eclipse.jetty.start.BaseBuilder.build(BaseBuilder.java:338)
        at org.eclipse.jetty.start.Main.start(Main.java:551)
        at org.eclipse.jetty.start.Main.main(Main.java:82)
java.io.IOException: For security reasons, Jetty start is unable to process file resource not in ${jetty.base} - logs/
        at org.eclipse.jetty.start.FileInitializer.getDestination(FileInitializer.java:114)
        at org.eclipse.jetty.start.fileinits.LocalFileInitializer.create(LocalFileInitializer.java:36)
        at org.eclipse.jetty.start.BaseBuilder.processFileResource(BaseBuilder.java:370)
        at org.eclipse.jetty.start.BaseBuilder.processFileResources(BaseBuilder.java:407)
        at org.eclipse.jetty.start.BaseBuilder.build(BaseBuilder.java:338)
        at org.eclipse.jetty.start.Main.start(Main.java:551)
        at org.eclipse.jetty.start.Main.main(Main.java:82)
java.io.IOException: For security reasons, Jetty start is unable to process file resource not in ${jetty.base} - resources/
        at org.eclipse.jetty.start.FileInitializer.getDestination(FileInitializer.java:114)
        at org.eclipse.jetty.start.fileinits.LocalFileInitializer.create(LocalFileInitializer.java:36)
        at org.eclipse.jetty.start.BaseBuilder.processFileResource(BaseBuilder.java:370)
        at org.eclipse.jetty.start.BaseBuilder.processFileResources(BaseBuilder.java:407)
        at org.eclipse.jetty.start.BaseBuilder.build(BaseBuilder.java:338)
        at org.eclipse.jetty.start.Main.start(Main.java:551)
        at org.eclipse.jetty.start.Main.main(Main.java:82)
WARN  : Failed to process all file resources.
 - [IOException] For security reasons, Jetty start is unable to process file resource not in ${jetty.base} - webapps/ - webapps/
 - [IOException] For security reasons, Jetty start is unable to process file resource not in ${jetty.base} - logs/ - logs/
 - [IOException] For security reasons, Jetty start is unable to process file resource not in ${jetty.base} - lib/ - lib/
 - [IOException] For security reasons, Jetty start is unable to process file resource not in ${jetty.base} - lib/ext/ - lib/ext/
 - [IOException] For security reasons, Jetty start is unable to process file resource not in ${jetty.base} - logs/ - logs/
 - [IOException] For security reasons, Jetty start is unable to process file resource not in ${jetty.base} - resources/ - resources/
java.lang.RuntimeException: Failed to process all file resources.
 - [IOException] For security reasons, Jetty start is unable to process file resource not in ${jetty.base} - webapps/ - webapps/
 - [IOException] For security reasons, Jetty start is unable to process file resource not in ${jetty.base} - logs/ - logs/
 - [IOException] For security reasons, Jetty start is unable to process file resource not in ${jetty.base} - lib/ - lib/
 - [IOException] For security reasons, Jetty start is unable to process file resource not in ${jetty.base} - lib/ext/ - lib/ext/
 - [IOException] For security reasons, Jetty start is unable to process file resource not in ${jetty.base} - logs/ - logs/
 - [IOException] For security reasons, Jetty start is unable to process file resource not in ${jetty.base} - resources/ - resources/
        at org.eclipse.jetty.start.BaseBuilder.processFileResources(BaseBuilder.java:427)
        at org.eclipse.jetty.start.BaseBuilder.build(BaseBuilder.java:338)
        at org.eclipse.jetty.start.Main.start(Main.java:551)
        at org.eclipse.jetty.start.Main.main(Main.java:82)

Usage: java -jar $JETTY_HOME/start.jar [options] [properties] [configs]
       java -jar $JETTY_HOME/start.jar --help  # for more information

We have:

$ tree -L 1
.
├── data
├── jetty
├── logs
├── resources
├── start.d
├── start_xwiki.bat
├── start_xwiki.sh
├── start_xwiki_debug.bat
├── start_xwiki_debug.sh
├── stop_xwiki.bat
├── stop_xwiki.sh
├── tmp
└── webapps

joakime · 2024-08-14T15:34:57Z

The --add-modules=xwiki command simply adds the module to the configuration file (and ensures that it's files section is sane), it is not enabled with that command.
The --module=xwiki is the command that enables it.

You can verify the configuration with start.jar --list-config command.
That will show the list of modules enabled, and the properties present from the configuration.

-Djava.io.tmpdir=./tmp -Djetty.home=jetty -Djetty.base=.

All three of those must be absolute paths, not relative.

joakime · 2024-08-14T15:36:53Z

Also, you should never have the jetty-home or the jetty-base nested inside each other in either direction.

This is bad.

/path/to/dist/jetty-home
/path/to/dist/jetty-home/jetty-base

This is also bad.

/path/to/dist/jetty-base
/path/to/dist/jetty-base/jetty-home

It should be ...

/path/to/dist/jetty-home
/path/to/dist/jetty-base

vmassol · 2024-08-16T07:51:32Z

Thanks @joakime for the recommendations.

However, this shouldn't affect why the jetty.httpConfig.uriCompliance=RFC3986,AMBIGUOUS_PATH_ENCODING property in startd.d/xwiki/ini isn't used in etc/jetty.xml to replace <Property name="jetty.httpConfig.uriCompliance" default="DEFAULT"/> in <Set name="uriCompliance"><Call class="org.eclipse.jetty.http.UriCompliance" name="from"><Arg><Property name="jetty.httpConfig.uriCompliance" default="DEFAULT"/></Arg></Call></Set>, right?

This is what I got with --list-config, it seems the properties are read correctly. I don't know why they're listed as "ee8" properties and if that has some importance or not:

Enabled Modules:
----------------
  0) resources                 transitive provider of resources for xwiki
  1) logging/slf4j             dynamic dependency of xwiki-logging
  2) xwiki-logging             transitive provider of logging for threadpool
  3) bytebufferpool            transitive provider of bytebufferpool for server
                               ini template available with --add-modules=bytebufferpool
  4) client                    transitive provider of client for ee8-websocket-javax
  5) console-capture           transitive provider of console-capture for xwiki
                               ini template available with --add-modules=console-capture
  6) ext                       transitive provider of ext for xwiki
  7) threadpool                transitive provider of threadpool for server
                               ini template available with --add-modules=threadpool
  8) server                    transitive provider of server for xwiki
                               ini template available with --add-modules=server
  9) deploy                    transitive provider of deploy for ee8-deploy
 10) sessions                  transitive provider of sessions for ee8-servlet
                               ini template available with --add-modules=sessions
 11) ee8-servlet               transitive provider of ee8-servlet for ee8-security
 12) security                  transitive provider of security for ee8-security
 13) ee8-security              transitive provider of ee8-security for ee8-plus
 14) ee-webapp                 transitive provider of ee-webapp for ee8-webapp
                               ini template available with --add-modules=ee-webapp
 15) ee8-webapp                transitive provider of ee8-webapp for ee8-plus
                               ini template available with --add-modules=ee8-webapp
 16) plus                      transitive provider of plus for ee8-annotations
 17) jndi                      transitive provider of jndi for ee8-plus
 18) ee8-plus                  transitive provider of ee8-plus for ee8-annotations
 19) ee8-annotations           transitive provider of ee8-annotations for xwiki
 20) ee8-apache-jsp            transitive provider of ee8-apache-jsp for xwiki
 21) ee8-deploy                transitive provider of ee8-deploy for xwiki
                               ini template available with --add-modules=ee8-deploy
 22) ee8-websocket-javax       transitive provider of ee8-websocket-javax for xwiki
 23) ee8-websocket-jetty       transitive provider of ee8-websocket-jetty for xwiki
 24) http                      transitive provider of http for xwiki
                               ini template available with --add-modules=http
 25) http-forwarded            transitive provider of http-forwarded for xwiki
                               ini template available with --add-modules=http-forwarded
 26) requestlog                transitive provider of requestlog for xwiki
                               ini template available with --add-modules=requestlog
 27) xwiki                     ${jetty.base}/start.d/xwiki.ini

JVM Version & Properties:
-------------------------
 java.home = /Library/Java/JavaVirtualMachines/jdk-17.jdk/Contents/Home
 java.vm.vendor = Oracle Corporation
 java.vm.version = 17+35-LTS-2724
 java.vm.name = Java HotSpot(TM) 64-Bit Server VM
 java.vm.info = mixed mode, sharing
 java.runtime.name = Java(TM) SE Runtime Environment
 java.runtime.version = 17+35-LTS-2724
 java.io.tmpdir = ./tmp
 user.dir = /Users/vmassol/dev/xwiki/xwiki-platform/xwiki-platform-core/xwiki-platform-rest/xwiki-platform-rest-test/xwiki-platform-rest-test-tests/target/xwiki
 user.language = en
 user.country = FR

Jetty Version & Properties:
---------------------------
 jetty.version = 12.0.12
 jetty.tag.version = jetty-12.0.12
 jetty.build = cc6f1b74db755fed228b50701ad967aeaa68e83f
 jetty.home = jetty
 jetty.base = .

Config Search Order:
--------------------
 <command-line>
 ${jetty.base} -> /Users/vmassol/dev/xwiki/xwiki-platform/xwiki-platform-core/xwiki-platform-rest/xwiki-platform-rest-test/xwiki-platform-rest-test-tests/target/xwiki/.
 ${jetty.home} -> /Users/vmassol/dev/xwiki/xwiki-platform/xwiki-platform-core/xwiki-platform-rest/xwiki-platform-rest-test/xwiki-platform-rest-test-tests/target/xwiki/jetty

System Properties:
------------------
 (no system properties specified)

Properties: Jetty
-----------------
 STOP.KEY = xwiki
 STOP.PORT = 8079
 java.version = 17
 java.version.major = 17
 java.version.micro = 0
 java.version.minor = 0
 java.version.platform = 17
 jetty.http.port = 8080
 jetty.requestlog.dir = logs
 jetty.webapp.addHiddenClasses = org.eclipse.jetty.logging.,${jetty.home.uri}/lib/logging/,org.slf4j.
 runtime.feature.alpn = true
 slf4j.version = 2.0.13

Classpath: Jetty
----------------
Version Information on 17 entries in the classpath.
Note: order presented here is how they would appear on the classpath.
      changes to the --module=name command line options will be reflected here.
 0:                    (dir) | ${jetty.home}/resources
 1:                   2.0.13 | ${jetty.home}/lib/logging/slf4j-api-2.0.13.jar | http://www.opensource.org/licenses/mit-license.php
 2:                  12.0.12 | ${jetty.home}/lib/logging/jetty-slf4j-impl-12.0.12.jar | EPL-2.0 OR Apache-2.0
 3:                  12.0.12 | ${jetty.home}/lib/jetty-client-12.0.12.jar | EPL-2.0 OR Apache-2.0
 4:                   2.16.1 | ${jetty.home}/lib/ext/commons-io-2.16.1.jar | https://www.apache.org/licenses/LICENSE-2.0.txt
 5:          16.7.0-SNAPSHOT | ${jetty.home}/lib/ext/xwiki-platform-tool-jetty-listener-16.7.0-20240814.065843-28.jar | GNU LESSER GENERAL PUBLIC LICENSE Version 2.1, February 1999
 6:                  12.0.12 | ${jetty.home}/lib/jetty-http-12.0.12.jar | EPL-2.0 OR Apache-2.0
 7:                  12.0.12 | ${jetty.home}/lib/jetty-server-12.0.12.jar | EPL-2.0 OR Apache-2.0
 8:                  12.0.12 | ${jetty.home}/lib/jetty-xml-12.0.12.jar | EPL-2.0 OR Apache-2.0
 9:                  12.0.12 | ${jetty.home}/lib/jetty-util-12.0.12.jar | EPL-2.0 OR Apache-2.0
10:                  12.0.12 | ${jetty.home}/lib/jetty-io-12.0.12.jar | EPL-2.0 OR Apache-2.0
11:                  12.0.12 | ${jetty.home}/lib/jetty-deploy-12.0.12.jar | EPL-2.0 OR Apache-2.0
12:                  12.0.12 | ${jetty.home}/lib/jetty-session-12.0.12.jar | EPL-2.0 OR Apache-2.0
13:                  12.0.12 | ${jetty.home}/lib/jetty-security-12.0.12.jar | EPL-2.0 OR Apache-2.0
14:                  12.0.12 | ${jetty.home}/lib/jetty-ee-12.0.12.jar | EPL-2.0 OR Apache-2.0
15:                  12.0.12 | ${jetty.home}/lib/jetty-plus-12.0.12.jar | EPL-2.0 OR Apache-2.0
16:                  12.0.12 | ${jetty.home}/lib/jetty-jndi-12.0.12.jar | EPL-2.0 OR Apache-2.0

Active XMLs: Jetty
------------------
 ${jetty.home}/etc/jetty-bytebufferpool.xml
 ${jetty.home}/etc/console-capture.xml
 ${jetty.home}/etc/jetty-threadpool.xml
 ${jetty.home}/etc/jetty.xml
 ${jetty.home}/etc/jetty-deploy.xml
 ${jetty.home}/etc/sessions/id-manager.xml
 ${jetty.home}/etc/jetty-ee-webapp.xml
 ${jetty.home}/etc/jetty-http.xml
 ${jetty.home}/etc/jetty-http-forwarded.xml
 ${jetty.home}/etc/jetty-requestlog.xml

Properties: ee8
---------------
 contextHandlerClass = org.eclipse.jetty.ee8.webapp.WebAppContext
 ee8.asm.version = 9.7
 ee8.jakarta.annotation.api.version = 1.3.5
 ee8.jsp.impl.version = 9.0.90
 jetty.deploy.extractWars = false
 jetty.deploy.scanInterval = 0
 jetty.httpConfig.uriCompliance = RFC3986,AMBIGUOUS_PATH_ENCODING

Classpath: ee8
--------------
Version Information on 21 entries in the classpath.
Note: order presented here is how they would appear on the classpath.
      changes to the --module=name command line options will be reflected here.
 0:                    4.0.6 | ${jetty.home}/lib/jetty-servlet-api-4.0.6.jar | http://www.apache.org/licenses/LICENSE-2.0, http://www.eclipse.org/org/documents/epl-v10.php
 1:                  12.0.12 | ${jetty.home}/lib/jetty-ee8-nested-12.0.12.jar | EPL-2.0 OR Apache-2.0
 2:                  12.0.12 | ${jetty.home}/lib/jetty-ee8-servlet-12.0.12.jar | EPL-2.0 OR Apache-2.0
 3:                  12.0.12 | ${jetty.home}/lib/jetty-ee8-security-12.0.12.jar | EPL-2.0 OR Apache-2.0
 4:                  12.0.12 | ${jetty.home}/lib/jetty-ee8-webapp-12.0.12.jar | EPL-2.0 OR Apache-2.0
 5:                  12.0.12 | ${jetty.home}/lib/jetty-ee8-plus-12.0.12.jar | EPL-2.0 OR Apache-2.0
 6:                    1.3.3 | ${jetty.home}/lib/jakarta.transaction-api-1.3.3.jar | http://www.eclipse.org/legal/epl-2.0, https://www.gnu.org/software/classpath/license.html
 7:                  12.0.12 | ${jetty.home}/lib/jetty-ee8-annotations-12.0.12.jar | EPL-2.0 OR Apache-2.0
 8:                      9.7 | ${jetty.home}/lib/ee8-annotations/asm-9.7.jar | BSD-3-Clause;link=https://asm.ow2.io/LICENSE.txt
 9:                      9.7 | ${jetty.home}/lib/ee8-annotations/asm-commons-9.7.jar | BSD-3-Clause;link=https://asm.ow2.io/LICENSE.txt
10:                      9.7 | ${jetty.home}/lib/ee8-annotations/asm-tree-9.7.jar | BSD-3-Clause;link=https://asm.ow2.io/LICENSE.txt
11:                    1.3.5 | ${jetty.home}/lib/ee8-annotations/jakarta.annotation-api-1.3.5.jar | http://www.eclipse.org/legal/epl-2.0, https://www.gnu.org/software/classpath/license.html
12:                   9.0.90 | ${jetty.home}/lib/ee8-apache-jsp/apache-el-9.0.90.jar | http://www.apache.org/licenses/LICENSE-2.0
13:                  12.0.12 | ${jetty.home}/lib/jetty-websocket-core-common-12.0.12.jar | EPL-2.0 OR Apache-2.0
14:                  12.0.12 | ${jetty.home}/lib/jetty-websocket-core-client-12.0.12.jar | EPL-2.0 OR Apache-2.0
15:                  12.0.12 | ${jetty.home}/lib/jetty-websocket-core-server-12.0.12.jar | EPL-2.0 OR Apache-2.0
16:                  12.0.12 | ${jetty.home}/lib/ee8-websocket/jetty-ee8-websocket-servlet-12.0.12.jar | EPL-2.0 OR Apache-2.0
17:                    1.1.2 | ${jetty.home}/lib/ee8-websocket/jetty-javax-websocket-api-1.1.2.jar | http://www.apache.org/licenses/LICENSE-2.0, http://www.eclipse.org/org/documents/epl-v10.php
18:                  12.0.12 | ${jetty.home}/lib/ee8-websocket/jetty-ee8-websocket-javax-client-12.0.12.jar | EPL-2.0 OR Apache-2.0
19:                  12.0.12 | ${jetty.home}/lib/ee8-websocket/jetty-ee8-websocket-javax-common-12.0.12.jar | EPL-2.0 OR Apache-2.0
20:                  12.0.12 | ${jetty.home}/lib/ee8-websocket/jetty-ee8-websocket-javax-server-12.0.12.jar | EPL-2.0 OR Apache-2.0

Active XMLs: ee8
----------------
 ${jetty.home}/etc/jetty-ee8-webapp.xml
 ${jetty.home}/etc/jetty-ee8-deploy.xml
 ${jetty.home}/etc/jetty-xwiki.xml

Any idea what could be wrong?

Thx

vmassol · 2024-08-16T07:52:52Z

I don't know why they're listed as "ee8" properties and if that has some importance or not:

Probably because xwiki.ini is the match for xwiki.mod which has:

[environment]
ee8

joakime · 2024-08-17T02:42:25Z

As you can see in your output ...

Properties: ee8
---------------
 contextHandlerClass = org.eclipse.jetty.ee8.webapp.WebAppContext
 ee8.asm.version = 9.7
 ee8.jakarta.annotation.api.version = 1.3.5
 ee8.jsp.impl.version = 9.0.90
 jetty.deploy.extractWars = false
 jetty.deploy.scanInterval = 0
 jetty.httpConfig.uriCompliance = RFC3986,AMBIGUOUS_PATH_ENCODING

That property is being placed into the ee8 environment, very much due to your module definition.
So it is not available to the core level configuration for the Connectors.

Another way to see this ...

start.jar --list-config=args - shows what arguments (and properties) are used at core.
start.jar --list-config=envs - shows what each environment is setup as (look for --env <name> as separator between each environment)

vmassol · 2024-08-17T11:14:01Z

Thanks @joakime

I've now removed the following from xwiki.mod and the properties from xwiki.ini are now defined as core and thus the jetty.httpConfig.uriCompliance property is seen from jetty.xml:

[environment]
ee8

All working!

vmassol · 2024-08-17T11:14:35Z

Only remaining question I guess is whether there's a syntax to force a property to be defined as core?

joakime · 2024-08-17T16:16:14Z

Only remaining question I guess is whether there's a syntax to force a property to be defined as core?

That would mean 2 separate modules (one for server, one for ee8)
You could have ...

xwiki-server.mod - where you put the server level configuration that xwiki wants (no [environment] section)
xwiki-ee8.mod - where you put the ee8 level configuration that xwiki webapp wants (with [environment] section at ee8)
xwiki.mod - meta module that really only depends on xwiki-ee8, and xwiki-server

vmassol · 2024-08-19T08:08:26Z

Thanks @joakime

gregw · 2024-08-21T21:22:18Z

re-opening to see if we can improve documentation

janbartel · 2024-08-21T22:05:58Z

@vmassol the problem, as @joakime pointed out, is that you have defined a core property as being in the ee8 environment, which means it is not available for substitution when the jetty.xml file (a core file) is read before any of the environments are activated. The appropriate place to define these core property values is in the matching core .ini file, which in this case is server.ini. Since we moved to environments, a property can't be defined just anywhere on the command line (or in any module), it must be defined appropriately:

a) if specifying things on the command line, the property must be defined before the module or xml file that uses it
b) if using only modules, the property must be defined before or in the module that actually uses the property

I think we should update our doco to be clearer about this change.

vmassol · 2024-08-22T11:20:19Z

@janbartel Thanks a lot for the additional explanations! So far I've removed the ee8 environment from my xwiki.mod. I could add it back and instead add a servet.ini file in start.d.

TBH I'm not sure what adding the ee8 environment does. I have the feeling it's useful if you want to run the same jetty engine several times with different environments, as a way to differentiate what gets defined. In our case, we run a single environment so I guess it doesn't matter.

Thanks again

vmassol added the Question label Aug 13, 2024

joakime mentioned this issue Aug 14, 2024

HttpConfiguration dump is missing entries #12163

Closed

vmassol closed this as completed Aug 17, 2024

gregw self-assigned this Aug 21, 2024

gregw reopened this Aug 21, 2024

nscuro mentioned this issue Oct 10, 2024

ERROR 400 Ambiguous URI path separator DependencyTrack/dependency-track#4238

Closed

2 tasks

gregw mentioned this issue Nov 20, 2024

Improve documentation for ambiguous URIs #12554

Merged

gregw added this to Jetty 12.0.16 FROZEN Nov 20, 2024

gregw moved this to 🏗 In progress in Jetty 12.0.16 FROZEN Nov 20, 2024

gregw closed this as completed in #12554 Nov 28, 2024

github-project-automation bot moved this from 🏗 In progress to ✅ Done in Jetty 12.0.16 FROZEN Nov 28, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

"Ambiguous URI path encoding" when upgrading to Jetty 12 EE8 #12162

"Ambiguous URI path encoding" when upgrading to Jetty 12 EE8 #12162

vmassol commented Aug 13, 2024

joakime commented Aug 13, 2024

joakime commented Aug 13, 2024

joakime commented Aug 13, 2024

vmassol commented Aug 14, 2024

joakime commented Aug 14, 2024 •

edited

Loading

vmassol commented Aug 14, 2024

vmassol commented Aug 14, 2024

vmassol commented Aug 14, 2024

vmassol commented Aug 14, 2024

vmassol commented Aug 14, 2024

vmassol commented Aug 14, 2024 •

edited

Loading

joakime commented Aug 14, 2024

joakime commented Aug 14, 2024 •

edited

Loading

vmassol commented Aug 16, 2024

vmassol commented Aug 16, 2024

joakime commented Aug 17, 2024

vmassol commented Aug 17, 2024

vmassol commented Aug 17, 2024

joakime commented Aug 17, 2024

vmassol commented Aug 19, 2024

gregw commented Aug 21, 2024

janbartel commented Aug 21, 2024

vmassol commented Aug 22, 2024

"Ambiguous URI path encoding" when upgrading to Jetty 12 EE8 #12162

"Ambiguous URI path encoding" when upgrading to Jetty 12 EE8 #12162

Comments

vmassol commented Aug 13, 2024

joakime commented Aug 13, 2024

joakime commented Aug 13, 2024

joakime commented Aug 13, 2024

vmassol commented Aug 14, 2024

joakime commented Aug 14, 2024 • edited Loading

vmassol commented Aug 14, 2024

vmassol commented Aug 14, 2024

vmassol commented Aug 14, 2024

vmassol commented Aug 14, 2024

vmassol commented Aug 14, 2024

vmassol commented Aug 14, 2024 • edited Loading

joakime commented Aug 14, 2024

joakime commented Aug 14, 2024 • edited Loading

vmassol commented Aug 16, 2024

vmassol commented Aug 16, 2024

joakime commented Aug 17, 2024

vmassol commented Aug 17, 2024

vmassol commented Aug 17, 2024

joakime commented Aug 17, 2024

vmassol commented Aug 19, 2024

gregw commented Aug 21, 2024

janbartel commented Aug 21, 2024

vmassol commented Aug 22, 2024

joakime commented Aug 14, 2024 •

edited

Loading

vmassol commented Aug 14, 2024 •

edited

Loading

joakime commented Aug 14, 2024 •

edited

Loading