NSFS | S3 | Versioning: Threaded delete of multiple objects does not delete

[hseipp]

Environment info

• NooBaa Version: 5.17.0-20240904
• Platform: CentOS8, deployed using https://github.com/IBM/StorageScaleVagrant, Scale 5.2.2.0 240913.105913

Actual behavior

With the following test code, all delete_objects commands return, but the objects are still present afterwards:

    def test_versioning_concurrent_multi_object_delete():
        num_objects = 5
        num_threads = 5
        bucket_name = get_new_bucket()
    
        check_configure_versioning_retry(bucket_name, "Enabled", "Enabled")
    
        key_names = ["key_{:d}".format(x) for x in range(num_objects)]
        bucket = _create_objects(bucket_name=bucket_name, keys=key_names)
    
        client = get_client()
        versions = client.list_object_versions(Bucket=bucket_name)['Versions']
        assert len(versions) == num_objects
        objs_dict = {'Objects': [dict((k, v[k]) for k in ["Key", "VersionId"]) for v in versions]}
        results = [None] * num_threads
    
        def do_request(n):
            results[n] = client.delete_objects(Bucket=bucket_name, Delete=objs_dict)
    
        t = []
        for i in range(num_threads):
            thr = threading.Thread(target = do_request, args=[i])
            thr.start()
            t.append(thr)
        _do_wait_completion(t)
    
        for response in results:
            assert len(response['Deleted']) == num_objects
            assert 'Errors' not in response
    
        response = client.list_objects(Bucket=bucket_name)
>       assert 'Contents' not in response
E       assert 'Contents' not in {'Contents': [{'ETag': '"mtime-d48pnbhmvi0w-ino-15hc"', 'Key': 'key_0', 'LastModified': datetime.datetime(2024, 9, 17,..., 'ID': '66e85c103a034ad6a392511e'}, ...}], 'Delimiter': 'undefined', 'EncodingType': 'url', 'IsTruncated': False, ...}

Expected behavior

Test code passes and all objects get properly deleted.

Steps to reproduce

Execute the above test code as part of Ceph s3-tests.

More information - Screenshots / Logs / Other output

Noobaa log with "all" log level attached
noobaa_20240917_2.log.gz