s3fs rmdir command returns error when given a valid s3 path

[rjhawar]

Issue:
I have s3 uri that points to empty folder, i have verified the uri is valid by checking the path with fs.isdir and fs.info and both return valid responses. However when i go and use the rmdir command i get the following error:

Invalid bucket name: Bucket name must match the regex "^[a-zA-Z0-9.-_]{1,255}$" or be an ARN matching the regex "^arn:(aws).:(s3|s3-object-lambda):[a-z-0-9]:[0-9]{12}:accesspoint[/:][a-zA-Z0-9-.]{1,63}$|^arn:(aws).*:s3-outposts:[a-z-0-9]+:[0-9]{12}:outpost[/:][a-zA-Z0-9-]{1,63}[/:]accesspoint[/:][a-zA-Z0-9-]{1,63}$"

How is this function supposed to be used?

Version info:
Python 3.11.1
sf3s 2011.11.0

[martindurant]

Can you share the exact code you executed?

[rjhawar]

@martindurant apologies for the delay. Here is the code i used

import s3fs
def main(s3_uri):

    fs = s3fs.S3FileSystem(anon=False)

    # print(fs.info(s3_uri))
    print(fs.isdir(s3_uri))
    fs.rmdir(s3_uri)
# -----------------------------------------------------------------------------
if __name__ == '__main__':
    s3_uri = "s3://bucket-test-gh/test1/"
    main(s3_uri)
# -----------------------------------------------------------------------------

Error:
raise ParamValidationError(report=error_msg)
botocore.exceptions.ParamValidationError: Parameter validation failed:
Invalid bucket name "s3://bucket-test-gh/test1/": Bucket name must match the regex "^[a-zA-Z0-9.-_]{1,255}$" or be an ARN matching the regex "^arn:(aws).:(s3|s3-object-lambda):[a-z-0-9]:[0-9]{12}:accesspoint[/:][a-zA-Z0-9-.]{1,63}$|^arn:(aws).*:s3-outposts:[a-z-0-9]+:[0-9]{12}:outpost[/:][a-zA-Z0-9-]{1,63}[/:]accesspoint[/:][a-zA-Z0-9-]{1,63}$"

[sotosoul]

Same here. It returns the same error.

[DavidKatz-il]

The error occurs because rmdir invokes delete_bucket and passes the entire path as the name of the bucket.

    async def _rmdir(self, path):
        try:
            await self._call_s3("delete_bucket", Bucket=path)
    ...

Based on the code, it appears that rmdir is intended for deleting buckets, while rm is designed to delete the contents of a folder (but not the folder itself).

[martindurant]

Yes, agree with everything here. Something like the following then?

--- a/s3fs/core.py
+++ b/s3fs/core.py
@@ -899,6 +899,12 @@ class S3FileSystem(AsyncFileSystem):
     makedirs = sync_wrapper(_makedirs)

     async def _rmdir(self, path):
+        bucket, key = self.split_path(path)
+        if key:
+            if await self._exists(path):
+                # did you mean rm(path, recursive=True) ?
+                raise FileExistsError
+            raise FileNotFoundError
         try:
             await self._call_s3("delete_bucket", Bucket=path)

i.e., in S3, a directory rather than a bucket can only exist if it contains something, so rmdir is always an exception.
OR rmdir can be a noop if it is empty.