Skip to content

Commit

Permalink
Add Sharded Bucket middleware
Browse files Browse the repository at this point in the history
Adds the sharded bucket middleware, which allows for splitting objects
across multiple backend buckets for a given virtual bucket. The
middleware should be configured as:
s3proxy.sharded-blobstore.<bucket name>.shards=<number of shards>
s3proxy.sharded-blobstore.<bucket name>.prefix=<prefix>.

All shards are named <prefix>-<index>, where index is an
integer from 0 to <number of shards> - 1. If the <prefix> is not
supplied, the <bucket name> is used as the prefix.

Listing the virtual bucket and multipart uploads are not supported. When
listing all containers, the shards are elided from the result.

Fixes gaul#325
Fixes gaul#351
  • Loading branch information
timuralp committed May 8, 2021
1 parent 2cefdd1 commit e98daf7
Show file tree
Hide file tree
Showing 6 changed files with 928 additions and 0 deletions.
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -90,6 +90,7 @@ S3Proxy can modify its behavior based on middlewares:
* [eventual consistency modeling](https://github.com/gaul/s3proxy/wiki/Middleware---eventual-consistency)
* [large object mocking](https://github.com/gaul/s3proxy/wiki/Middleware-large-object-mocking)
* [read-only](https://github.com/gaul/s3proxy/wiki/Middleware-read-only)
* [sharded backend containers](https://github.com/gaul/s3proxy/wiki/Middleware-sharded-backend)

## Limitations

Expand Down
10 changes: 10 additions & 0 deletions src/main/java/org/gaul/s3proxy/Main.java
Original file line number Diff line number Diff line change
Expand Up @@ -224,6 +224,16 @@ private static BlobStore parseMiddlewareProperties(BlobStore blobStore,
blobStore = ReadOnlyBlobStore.newReadOnlyBlobStore(blobStore);
}

ImmutableMap<String, Integer> shards =
ShardedBlobStore.parseBucketShards(properties);
ImmutableMap<String, String> prefixes =
ShardedBlobStore.parsePrefixes(properties);
if (!shards.isEmpty()) {
System.err.println("Using sharded buckets backend");
blobStore = ShardedBlobStore.newShardedBlobStore(blobStore,
shards, prefixes);
}

return blobStore;
}

Expand Down
3 changes: 3 additions & 0 deletions src/main/java/org/gaul/s3proxy/S3ProxyConstants.java
Original file line number Diff line number Diff line change
Expand Up @@ -84,6 +84,9 @@ public final class S3ProxyConstants {
/** Prevent mutations. */
public static final String PROPERTY_READ_ONLY_BLOBSTORE =
"s3proxy.read-only-blobstore";
/** Shard objects across a specified number of buckets. */
public static final String PROPERTY_SHARDED_BLOBSTORE =
"s3proxy.sharded-blobstore";

/** Maximum time skew allowed in signed requests. */
public static final String PROPERTY_MAXIMUM_TIME_SKEW =
Expand Down
Loading

0 comments on commit e98daf7

Please sign in to comment.