• What does this project do

  This project essentially connects to a source redis cluster, and does its best to push data to a destination Redis cluster, in this case from our datacenter to the cloud.

• Why we did it

  At the time of writing there was a requirement to get some data from a Redis Cluster in our data center to the cloud.

  The reason we have to use this synchronization app is that in our datacenter we have Oracle stored procs that get synchronized to a Redis cluster in the local Redis cluster. Having this data as reliably as we can in aws allows for more stable and quick devellopment of features for the Dev Team.

• How we containerize it

  We currently have a modern golang app, that has a pretty solid build script from the offical source. The makefile build stage essentially packages the binary in a lightweight ubuntu container, we also use the makefile to pass some environment variables to the container so we can have the flexibility of packaging the container at run time or dynamically run the container with environment variables locally if desired. These variables are essentially what is needed to point the container towards a source and destination cluster

  The final step is to tage the image to the artifactory and push it.

• Parametrizing the container

  We can currently modify how the container is built in the pipeline by modification of the environment variables in the jenkins file at the build stage

• CNF

  The cloudformation is responsible for

  • Creating the ec2 instance
  • Updating it
  • Setting up its security groups and subnets
  • Installing docker
  • Pulling and running RedisShake

========== Following is the official doc

RedisShake is mainly used to synchronize data from one redis to another.
Thanks to the Douyu's WSD team for the support.

• 中文文档
• English tutorial
• 中文使用文档
• Release package

Redis-Shake

Redis-shake is developed and maintained by NoSQL Team in Alibaba-Cloud Database department.

Redis-shake has made some improvements based on redis-port, including bug fixes, performance improvements and feature enhancements.

Main Functions

The type can be one of the followings:

• decode: Decode dumped payload to human readable format (hex-encoding).
• restore: Restore RDB file to target redis.
• dump: Dump RDB file from source redis.
• sync: Sync data from source redis to target redis by sync or psync command. Including full synchronization and incremental synchronization.
• rump: Sync data from source redis to target redis by scan command. Only support full synchronization. Plus, RedisShake also supports fetching data from given keys in the input file when scan command is not supported on the source side. This mode is usually used when sync and psync redis commands aren't supported.

Please check out the conf/redis-shake.conf to see the detailed parameters description.

Support

Redis version from 2.x to 6.x.

Supports Standalone, Cluster and some proxies type like Codis, twemproxy, Aliyun Cluster Proxy, Tencent Cloud Proxy and so on.

For codis and twemproxy, there maybe some constraints, please checkout this question.

Configuration

Redis-shake has several parameters in the configuration conf/redis-shake.conf, that maybe confusing, if this is your first time using, please visit this tutorial.

Verification

User can use RedisFullCheck to verify correctness.

Metric

Redis-shake offers metrics through restful api and log file.

• restful api: curl 127.0.0.1:9320/metric.
• log: the metric info will be printed in the log periodically if enable.
• inner routine heap: curl http://127.0.0.1:9310/debug/pprof/goroutine?debug=2

Redis Type

Both the source and target type can be standalone, opensource cluster and proxy. Although the architecture patterns of different vendors are different for the proxy architecture, we still support different cloud vendors like alibaba-cloud, tencent-cloud and so on.

If the target is open source redis cluster, redis-shake uses redis-go-cluster driver to write data. When target type is proxy, redis-shakes write data in round-robin way.

If the source is redis cluster, redis-shake launches multiple goroutines for parallel pull. User can use rdb.parallel to control the RDB syncing concurrency.

The "move slot" operations must be disabled on the source side.

Usage

download the binary

You can directly download the binary in the release package. Run through similar commands:

./redis-shake.linux -type=sync -conf=redis-shake.conf # please note: user must modify redis-shake.conf first to match needs.

build by yourself

You can also build redis-shake yourself according to the following steps:

git clone https://github.com/alibaba/RedisShake.git
cd RedisShake
sh build.sh
cd bin
./redis-shake.linux -type=sync -conf=redis-shake.conf # please note: user must modify redis-shake.conf first to match needs.

Shake series tool

We also provide some tools for synchronization in Shake series.

• MongoShake: mongodb data synchronization tool.
• RedisShake: redis data synchronization tool.
• RedisFullCheck: redis data synchronization verification tool.
• NimoShake: sync dynamodb to mongodb.

Plus, we have a DingTalk group, so that users can join and discuss.
Group code: 23165540

Code branch rules

Version rules: a.b.c.

• a: major version
• b: minor version. even number means stable version.
• c: bugfix version

branch name	rules
master	master branch, do not allowed push code. store the latest stable version. develop branch will merge into this branch once new version created.
develop(main branch)	develop branch. all the bellowing branches fork from this.
feature-*	new feature branch. forked from develop branch and then merge back after finish developing, testing, and code review.
bugfix-*	bugfix branch. forked from develop branch and then merge back after finish developing, testing, and code review.
improve-*	improvement branch. forked from develop branch and then merge back after finish developing, testing, and code review.

Tag rules:
Add tag when releasing: "release-v{version}-{date}". for example: "release-v1.0.2-20180628"
User can use -version to print the version.

Thanks

Username	Mail
ceshihao	[email protected]
wangyiyang	[email protected]
muicoder	[email protected]
zhklcf	[email protected]
shuff1e	[email protected]
xuhualin	[email protected]