-
Notifications
You must be signed in to change notification settings - Fork 364
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
API for conditional / exclusive write #1693
Comments
If this is only to apply to A couple of thoughts:
|
Do you know how this interacts with multi-part-uploads, where although many bytes might have been sent, the file is not really written to the remote path location until a final commit? At what point is the exists condition applied? |
I'm not sure offhand. |
Over in zarr-developers/zarr-python#2262, we'd like to write a file but only if it doesn't already exist. On a local file system, this would be
open(path, mode="xb")
, which will fail with aFileExistsError
if the file already exists.Now that S3 supports conditional writes, it should be possible to implement this for s3fs, gcsfs (
if_generation_match=0
), and adlfs (overwrite=False
).Would there be any appetite for standardizing this behavior? I'm not sure what API is best, but I lean towards something like an
overwrite: bool
parameter topipe
and similar methods. We could also try to supportmode=xb
in someopen
-like methods, but I'm less sure about that.The text was updated successfully, but these errors were encountered: