[Bug] - HTTP500 from GET list /bucket/file_not_dir/no_such_file

[Ostheer]

Describe the bug
Doing a HTTP/HEAD GET request to a path like whatever/this_is_an_existing_file_and_not_a_dir/this_file_cannot_be_here yields a 500 server error, throwing off some clients expecting a 404 (see context below).

In my versity log, I see
Internal Error, walk testbucket: stat empty-0/.zattrs/.zarray: not a directory

Note that I previously filed an issue, #942, for this. @benmcclelland quickly merged some changes that slightly but not qualitatively altered the server error message.
I created this second issue because I got no response yet on my comment on the old issue. Feel free to close this and continue work in the old issue.

To Reproduce

xr.DataArray().to_zarr("s3://testbucket/empty-0", storage_options={"key": ..., "secret": ..., "endpoint_url": "http://..../"})

Expected behavior
A 404 would seem sensible.

Server Version
Compiled from git master branch after commit 0312a1e as described in this comment

Version  : 1.0.8
Build    : a26f069
BuildTime: 2024-11-12_01:04:10PM

Linux sr-storage 6.8.4-2-pve #1 SMP PREEMPT_DYNAMIC PMX 6.8.4-2 (2024-04-10T17:36Z) x86_64 GNU/Linux

[benmcclelland]

Sorry I missed the update to the other issue. I am able to reproduce this in the list-objects API. Let's keep this issue open for tracking this one since it is slightly different than the other issue (list vs head).

% aws s3api put-object --bucket test --key file                      
{
    "ETag": "d41d8cd98f00b204e9800998ecf8427e"
}
% aws s3api list-objects-v2 --bucket test --prefix file/file/file

An error occurred (InternalError) when calling the ListObjectsV2 operation (reached max retries: 2): We encountered an internal error, please try again.

[Ostheer]

You're right! The HEAD now does return 404. The 500 is caused by a GET with parameter "list-type...":

Nov 19 09:44:10 sr-storage versitygw-measurementsgw[529]: 2024/11/19 09:44:10 Internal Error, walk testbucket: stat test-wlshift-10/.zattrs/.zarray: not a directory
Nov 19 09:44:10 sr-storage versitygw-measurementsgw[529]: 09:44:09 | 500 |     433.296µs | 10.10.10.4 | GET | /testbucket | - | list-type=2&prefix=test-wlshift-10%2F.zattrs%2F.zarray%2F&delimiter=%2F&max-keys=1&encoding-type=url

[Ostheer]

Yes! It works now :) Thanks!