Duplicate files don't get invalidated

[mckamey]

Experiencing this on the v1.5.0-beta1 build in Homebrew.

When using s3cmd sync --cf-invalidate ..., if two files are identical it uses the server-side copying to reduce bandwidth. This seems to work well, but it doesn't include those duplicate files in the CloudFront invalidations. As such only the first gets invalidated and the others remain until the cache expires.

Expected behavior would be that all changed files are invalidated.

Thanks!

[mdomsch]

This is the same problem as #305. I would like someone to submit a patch
addressing both cleanly.
On Mar 29, 2014 4:30 PM, "Stephen McKamey" [email protected] wrote:

Experiencing this on the v1.5.0-beta1 build in Homebrew.

When using s3cmd sync --cf-invalidate ..., if two files are identical it
uses the server-side copying to reduce bandwidth. This seems to work well,
but it doesn't include those duplicate files in the CloudFront
invalidations. As such only the first gets invalidated and the others
remain until the cache expires.

Expected behavior would be that all changed files are invalidated.

Thanks!

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/309
.

[mckamey]

I would argue that while it might be fixed by the same patch, it is actually a distinct issue. For example, your comment about the actual need for invalidating deleted files (I agree BTW) doesn't apply here where newly uploaded files are not getting properly invalidated.

[mdomsch]

The logic needs to be:

1. if a file is newly uploaded, do not invalidate
2. if a file is uploaded replacing an existing file, do invalidate
3. if a file is remote copied, into a new file name, do not invalidate
4. if a file is remote copied, replacing an existing file, do invalidate

Invalidates on delete are, IMHO, not needed. Except for the case where
there are two distinct s3cmd sync runs:
A) run issues delete calls on some files
B) run issues upload calls with filenames that were deleted in run A

In this case, we would want to invalidate either on upload (which seems
weird), or on delete (which also seems weird). But we're not keeping state
across the two runs to know when we're in this case, so one or the other
should be done. If done on upload, it changes test 1 above to require
invalidation, therefore it also matches test 2, and we're back in the state
of issuing invalidates on all uploads. Tests 3 and 4 remain the same.

I'd love someone to take a crack at a patch (or series of patches) that
handles these all together cleanly, as they are closely related.

Thanks,
Matt

On Sun, Mar 30, 2014 at 2:06 PM, Stephen McKamey
[email protected]:

I would argue that while it might be fixed by the same patch, it is
actually a distinct issue. For example, your comment about the actual need
for invalidating deleted files (I agree BTW) doesn't apply here where newly
uploaded files are not getting properly invalidated.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/309#issuecomment-39035746
.

[mckamey]

And just to clarify this issue #309 is regarding case 4). In my experience 1), 2), 3) seem to be working in 1.5.0-beta1.

[mdomsch]

1 doesn't work today - we're invalidating newly uploaded files too, which
is wasteful.
2 works "on accident" because it's invalidating all uploaded files, not
just files that are replacing existing files.
3 works "on accident" because it's not calling invalidate on any remote
copied files.

On Sun, Mar 30, 2014 at 3:10 PM, Stephen McKamey
[email protected]:

And just to clarify this issue #309https://github.com/s3tools/s3cmd/issues/309is regarding case 4). In my experience 1), 2), 3) seem to be working in
1.5.0-beta1.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/309#issuecomment-39037759
.