Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a jsoncs3 share manager #3148

Merged
merged 95 commits into from
Aug 23, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
95 commits
Select commit Hold shift + click to select a range
a5d6c5b
initial jsoncs3 share manager
butonic Jul 20, 2022
ab2bfd3
explore and note thoughts
butonic Jul 20, 2022
f82c775
more thoughts
butonic Jul 25, 2022
9fe892f
Lay some groundwork
aduffeck Jul 26, 2022
ac6751c
WIP
aduffeck Jul 27, 2022
b080c70
implement getByKey and getByID
butonic Jul 27, 2022
1a70643
GetShare
butonic Jul 27, 2022
7fa2872
Unshare
butonic Jul 27, 2022
afa977e
UpdateShare
butonic Jul 27, 2022
f5e83fc
ListShares without filter
butonic Jul 27, 2022
449ee9b
DRY up tests
aduffeck Jul 28, 2022
defcb21
Extend and cleanup test suite
aduffeck Jul 28, 2022
4b800a2
Mininmal implementation for ListReceivedShares
aduffeck Jul 28, 2022
b7a937c
Add a basic cache for the received shares
aduffeck Jul 28, 2022
da4a239
UpdateReceivedShare
butonic Jul 28, 2022
0db511f
filter by id
butonic Jul 28, 2022
33ac3a2
initial group accessCache
butonic Jul 28, 2022
247070e
thoughts
butonic Jul 28, 2022
8eee472
refactor shareCache
butonic Jul 28, 2022
3757085
add todos for permission changes
butonic Jul 29, 2022
06b3f0a
add sharecache implementation
butonic Jul 29, 2022
005831b
use group cache for pending state
butonic Jul 29, 2022
af8127e
add plan for storage
butonic Jul 29, 2022
781a526
add storage.Stat() and sketch out refresh from disk
butonic Jul 29, 2022
b94855b
read user cache from storage
butonic Jul 29, 2022
f25e074
write user share cache on share
butonic Jul 29, 2022
61d7311
persist on unshare
butonic Jul 29, 2022
e24690e
extract set/removeCreatedCache
butonic Jul 29, 2022
d1f3637
replace provider space gcache with map
butonic Jul 29, 2022
35cd86a
drop gcache and unused code
butonic Jul 29, 2022
924d145
Increase test coverage
aduffeck Aug 1, 2022
61d21b7
Do not choke when the usercache is empty
aduffeck Aug 1, 2022
8635e06
Load the created.json when there's no in-memory cache yet
aduffeck Aug 1, 2022
03f9b98
Add a ProviderCache
aduffeck Aug 2, 2022
a9e4edb
Test that expired UserShareCaches are reloaded properly
aduffeck Aug 3, 2022
16347bf
Extract ShareCache into a subpackage
aduffeck Aug 3, 2022
d7e9d5a
Extract ShareCache persistence into the sharecache subpackage
aduffeck Aug 3, 2022
45bfff5
Refactor
aduffeck Aug 3, 2022
89e4b1d
Persist the provider cache
aduffeck Aug 3, 2022
02a5585
Update the space mtime when adding/removing shares
aduffeck Aug 3, 2022
55ef7ae
Fix persisting user shares
aduffeck Aug 3, 2022
9871e20
Test that GetShare syncs the cache
aduffeck Aug 3, 2022
b5642c9
Reload cache if a Share wasn't found. Fix unmarshaling shares
aduffeck Aug 5, 2022
824590d
Persist the cache when adding/removing shares
aduffeck Aug 5, 2022
1d7a73f
Implicitly persist the cache when adding/removing entries
aduffeck Aug 5, 2022
305d1dd
Also implicitly persist the ShareCache when adding/removing items
aduffeck Aug 5, 2022
c4af393
Consider the resource id when looking for shares by key
aduffeck Aug 5, 2022
2d83a26
Add more tests
aduffeck Aug 5, 2022
7efe451
Persist updates in the provider cache
aduffeck Aug 5, 2022
83cdc02
Update the MTime when persisting the caches
aduffeck Aug 5, 2022
9c16653
Set the MTime when persisting a UserCache
aduffeck Aug 8, 2022
38b24c8
Fix reloading the provider cache from disk when outdated
aduffeck Aug 8, 2022
fff4359
Cleanup
aduffeck Aug 8, 2022
e9a0eae
Fix syncing the provider cache when listing shares
aduffeck Aug 8, 2022
1c30339
Sync the providercache before listing received shares
aduffeck Aug 8, 2022
b721d5d
Add a cache for the received shares states
aduffeck Aug 8, 2022
bfd8600
Use the new userreceivedshares cache
aduffeck Aug 8, 2022
e65d31b
Increase test coverage of UpdateReceivedShare
aduffeck Aug 8, 2022
61e96bb
Fix listing group shares
aduffeck Aug 10, 2022
678387c
Get the individual shares from the cache instead of listing all of th…
aduffeck Aug 10, 2022
ea6c5ec
Fix unmarshaling group shares
aduffeck Aug 10, 2022
b78e9ae
sync the GroupReceivedCache before listing received shares
aduffeck Aug 10, 2022
1f71825
Increase test coverage regarding group shares
aduffeck Aug 10, 2022
3f58d48
Fix the path/filename of the groups received.json file in the storage
aduffeck Aug 10, 2022
d1debf2
Update docs
aduffeck Aug 10, 2022
ffffe87
Initialize storage
aduffeck Aug 11, 2022
2bb02bb
Use a different shareid format to not clash with resource ids
aduffeck Aug 12, 2022
2ba16dc
Fix group shares not being merged with the user states
aduffeck Aug 12, 2022
79882b4
Fix hound issues
aduffeck Aug 16, 2022
425d0b8
Remove TODO comments that aren't relevant anymore/yet
aduffeck Aug 16, 2022
5a3735a
Add support for the IfUnmodifiedSince upload condition
aduffeck Aug 16, 2022
2d7cb27
Do not overwrite newer data on the storage
aduffeck Aug 16, 2022
029581d
Allow for listing shares of other users as space manager
aduffeck Aug 18, 2022
77ea208
Bump go-cs3apis
aduffeck Aug 19, 2022
5b66826
Fix hound issues
aduffeck Aug 19, 2022
5de8377
Log the sync processes
aduffeck Aug 19, 2022
81a431d
Add changelog
aduffeck Aug 19, 2022
4fa5bac
Fix even more hound issues
aduffeck Aug 19, 2022
b5141d9
Remove superfluous sync
aduffeck Aug 19, 2022
6183f48
Regenerate mocks
aduffeck Aug 19, 2022
4148cdd
use utils.NowTS()
butonic Aug 19, 2022
a2fade2
cleanup cache sync logs
butonic Aug 19, 2022
2bb66db
introduce utils.TimeToTS
butonic Aug 19, 2022
851376c
convert to timestamp instead of taking time seconds
butonic Aug 19, 2022
ba1a212
prevent nil when trying to remove not existing grant
butonic Aug 19, 2022
a033534
Update local mtime after writing to the cache
aduffeck Aug 19, 2022
8bfcfa0
WIP: use IfUnmodifiedSince flag when writing caches
aduffeck Aug 19, 2022
776b30b
comment stat after upload, update docs
butonic Aug 22, 2022
9dc366d
use new mtime for if-unmodified-since
butonic Aug 22, 2022
285e35c
add timestamps to precondition failed error
butonic Aug 22, 2022
18f2d66
revert mtime change on error
butonic Aug 22, 2022
b41b20f
add missing headers
butonic Aug 22, 2022
5b7a094
Merge branch 'edge' into jsoncs3
butonic Aug 22, 2022
2e77af4
lint fixes
butonic Aug 22, 2022
033ae61
fix nextcloud mock
butonic Aug 23, 2022
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions changelog/unreleased/jsoncs3-share-manager.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
Enhancement: Add new jsoncs3 share manager

We've added a new jsoncs3 share manager which splits the json file per storage
space and caches data locally.

https://github.com/cs3org/reva/pull/3148
2 changes: 1 addition & 1 deletion go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ require (
github.com/cheggaaa/pb v1.0.29
github.com/coreos/go-oidc v2.2.1+incompatible
github.com/cs3org/cato v0.0.0-20200828125504-e418fc54dd5e
github.com/cs3org/go-cs3apis v0.0.0-20220719130120-361e9f987d64
github.com/cs3org/go-cs3apis v0.0.0-20220818202316-e92afdddac6d
github.com/cubewise-code/go-mime v0.0.0-20200519001935-8c5762b177d8
github.com/dgraph-io/ristretto v0.1.0
github.com/emvi/iso-639-1 v1.0.1
Expand Down
4 changes: 2 additions & 2 deletions go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -170,8 +170,8 @@ github.com/cpuguy83/go-md2man/v2 v2.0.0/go.mod h1:maD7wRr/U5Z6m/iR4s+kqSMx2CaBsr
github.com/creack/pty v1.1.9/go.mod h1:oKZEueFk5CKHvIhNR5MUki03XCEU+Q6VDXinZuGJ33E=
github.com/cs3org/cato v0.0.0-20200828125504-e418fc54dd5e h1:tqSPWQeueWTKnJVMJffz4pz0o1WuQxJ28+5x5JgaHD8=
github.com/cs3org/cato v0.0.0-20200828125504-e418fc54dd5e/go.mod h1:XJEZ3/EQuI3BXTp/6DUzFr850vlxq11I6satRtz0YQ4=
github.com/cs3org/go-cs3apis v0.0.0-20220719130120-361e9f987d64 h1:cFnankJOCWndnOns4sKRG7yzH61ammK2Am6rEGWCK40=
github.com/cs3org/go-cs3apis v0.0.0-20220719130120-361e9f987d64/go.mod h1:UXha4TguuB52H14EMoSsCqDj7k8a/t7g4gVP+bgY5LY=
github.com/cs3org/go-cs3apis v0.0.0-20220818202316-e92afdddac6d h1:toyZ7IsXlUdEPZ/IG8fg7hbM8HcLPY0bkX4FKBmgLVI=
github.com/cs3org/go-cs3apis v0.0.0-20220818202316-e92afdddac6d/go.mod h1:UXha4TguuB52H14EMoSsCqDj7k8a/t7g4gVP+bgY5LY=
github.com/cubewise-code/go-mime v0.0.0-20200519001935-8c5762b177d8 h1:Z9lwXumT5ACSmJ7WGnFl+OMLLjpz5uR2fyz7dC255FI=
github.com/cubewise-code/go-mime v0.0.0-20200519001935-8c5762b177d8/go.mod h1:4abs/jPXcmJzYoYGF91JF9Uq9s/KL5n1jvFDix8KcqY=
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
Expand Down
29 changes: 18 additions & 11 deletions internal/grpc/services/storageprovider/storageprovider.go
Original file line number Diff line number Diff line change
Expand Up @@ -302,26 +302,33 @@ func (s *service) InitiateFileUpload(ctx context.Context, req *provider.Initiate
if err != nil {
return nil, err
}
switch sRes.Status.Code {
case rpc.Code_CODE_OK, rpc.Code_CODE_NOT_FOUND:
// Just continue with a normal upload
default:
return &provider.InitiateFileUploadResponse{
Status: sRes.Status,
}, nil
}

metadata := map[string]string{}
ifMatch := req.GetIfMatch()
if ifMatch != "" {
switch sRes.Status.Code {
case rpc.Code_CODE_OK:
if sRes.Info.Etag != ifMatch {
return &provider.InitiateFileUploadResponse{
Status: status.NewAborted(ctx, errors.New("etag mismatch"), "etag mismatch"),
}, nil
}
case rpc.Code_CODE_NOT_FOUND:
// Just continue with a normal upload
default:
if sRes.Info.Etag != ifMatch {
return &provider.InitiateFileUploadResponse{
Status: sRes.Status,
Status: status.NewFailedPrecondition(ctx, errors.New("etag mismatch"), "etag mismatch"),
}, nil
}
metadata["if-match"] = ifMatch
}
ifUnmodifiedSince := req.GetIfUnmodifiedSince()
if ifUnmodifiedSince != nil {
if utils.LaterTS(sRes.Info.Mtime, ifUnmodifiedSince) == sRes.Info.Mtime {
return &provider.InitiateFileUploadResponse{
Status: status.NewFailedPrecondition(ctx, errors.New("resource has been modified"), "resource has been modified"),
}, nil
}
}

ctx = ctxpkg.ContextSetLockID(ctx, req.LockId)

Expand Down
Loading