使用 redis-shake 同步 Redis 数据

🏷️ Redis

使用阿里开源的 redis-shake 工具同步 Redis 数据（这个开源工具貌似暂时仅支持单向同步，我记得阿里云提供的工具里貌似是支持双向同步的）。

这里是同步（ sync ）的示例，另外还支持从备份文件恢复（ restore ）数据。

官方仓库的 README 里有详细的介绍，这里仅记录下自己的操作步骤，供以后参考。

1. 下载

   从 Releases 页面下载后解压（这里使用的是 3.1.5 版本）：

   tar -xzvf redis-shake-linux-amd64.tar.gz

2. 同步的配置文件 sync-prod-k8s-uat.toml

   这里是在默认提供的 sync.toml 基础上修改了些配置（源和目标服务器的版本、地址和 ncpu）。
   配置项不多，备注也写的很详细。

   type = "sync"
   
   [source]
   version = 6.0 # redis version, such as 2.8, 4.0, 5.0, 6.0, 6.2, 7.0, ...
   address = "10.0.1.14:6379"
   username = "" # keep empty if not using ACL
   password = "" # keep empty if no authentication is required
   tls = false
   elasticache_psync = "" # using when source is ElastiCache. ref: https://github.com/alibaba/RedisShake/issues/373
   
   [target]
   type = "standalone" # "standalone" or "cluster"
   version = 6.0 # redis version, such as 2.8, 4.0, 5.0, 6.0, 6.2, 7.0, ...
   # When the target is a cluster, write the address of one of the nodes.
   # redis-shake will obtain other nodes through the `cluster nodes` command.
   address = "10.0.1.96:6379"
   username = "" # keep empty if not using ACL
   password = "" # keep empty if no authentication is required
   tls = false
   
   [advanced]
   dir = "data"
   
   # runtime.GOMAXPROCS, 0 means use runtime.NumCPU() cpu cores
   ncpu = 0
   
   # pprof port, 0 means disable
   pprof_port = 0
   
   # metric port, 0 means disable
   metrics_port = 0
   
   # log
   log_file = "redis-shake.log"
   log_level = "info" # debug, info or warn
   log_interval = 5 # in seconds
   
   # redis-shake gets key and value from rdb file, and uses RESTORE command to
   # create the key in target redis. Redis RESTORE will return a "Target key name
   # is busy" error when key already exists. You can use this configuration item
   # to change the default behavior of restore:
   # panic:   redis-shake will stop when meet "Target key name is busy" error.
   # rewrite: redis-shake will replace the key with new value.
   # ignore:  redis-shake will skip restore the key when meet "Target key name is busy" error.
   rdb_restore_command_behavior = "rewrite" # panic, rewrite or skip
   
   # pipeline
   pipeline_count_limit = 1024
   
   # Client query buffers accumulate new commands. They are limited to a fixed
   # amount by default. This amount is normally 1gb.
   target_redis_client_max_querybuf_len = 1024_000_000
   
   # In the Redis protocol, bulk requests, that are, elements representing single
   # strings, are normally limited to 512 mb.
   target_redis_proto_max_bulk_len = 512_000_000

3. 执行同步命令

   ./redis-shake sync-prod-k8s-uat.toml

4. 日志 data/redis-shake.log

   {"level":"info","time":"2022-10-08T14:52:41+08:00","message":"GOOS: linux, GOARCH: amd64"}
   {"level":"info","time":"2022-10-08T14:52:41+08:00","message":"Ncpu: 0, GOMAXPROCS: 2"}
   {"level":"info","time":"2022-10-08T14:52:41+08:00","message":"pid: 31083"}
   {"level":"info","time":"2022-10-08T14:52:41+08:00","message":"pprof_port: 0"}
   {"level":"info","time":"2022-10-08T14:52:41+08:00","message":"No lua file specified, will not filter any cmd."}
   {"level":"info","time":"2022-10-08T14:52:41+08:00","message":"auth successful. address=[10.0.1.96:6379]"}
   {"level":"info","time":"2022-10-08T14:52:41+08:00","message":"redisWriter connected to redis successful. address=[10.0.1.96:6379]"}
   {"level":"info","time":"2022-10-08T14:52:41+08:00","message":"auth successful. address=[10.0.1.14:6379]"}
   {"level":"info","time":"2022-10-08T14:52:41+08:00","message":"psyncReader connected to redis successful. address=[10.0.1.14:6379]"}
   {"level":"info","time":"2022-10-08T14:52:41+08:00","message":"start save RDB. address=[10.0.1.14:6379]"}
   {"level":"info","time":"2022-10-08T14:52:41+08:00","message":"send [replconf listening-port 10007]"}
   {"level":"info","time":"2022-10-08T14:52:41+08:00","message":"send [PSYNC ? -1]"}
   {"level":"info","time":"2022-10-08T14:52:41+08:00","message":"receive [FULLRESYNC 0d23bc30bd360693a56e78e71f6eee3f7c920dd5 0]"}
   {"level":"info","time":"2022-10-08T14:52:41+08:00","message":"source db is doing bgsave. address=[10.0.1.14:6379]"}
   {"level":"info","time":"2022-10-08T14:52:41+08:00","message":"source db bgsave finished. timeUsed=[0.83]s, address=[10.0.1.14:6379]"}
   {"level":"info","time":"2022-10-08T14:52:41+08:00","message":"received rdb length. length=[41477926]"}
   {"level":"info","time":"2022-10-08T14:52:41+08:00","message":"create dump.rdb file. filename_path=[dump.rdb]"}
   {"level":"info","time":"2022-10-08T14:52:42+08:00","message":"save RDB finished. address=[10.0.1.14:6379], total_bytes=[41477926]"}
   {"level":"info","time":"2022-10-08T14:52:42+08:00","message":"start send RDB. address=[10.0.1.14:6379]"}
   {"level":"info","time":"2022-10-08T14:52:42+08:00","message":"RDB version: 9"}
   {"level":"info","time":"2022-10-08T14:52:42+08:00","message":"RDB AUX fields. key=[redis-ver], value=[6.0.3]"}
   {"level":"info","time":"2022-10-08T14:52:42+08:00","message":"RDB AUX fields. key=[redis-bits], value=[64]"}
   {"level":"info","time":"2022-10-08T14:52:42+08:00","message":"RDB AUX fields. key=[ctime], value=[1665211961]"}
   {"level":"info","time":"2022-10-08T14:52:42+08:00","message":"RDB AUX fields. key=[used-mem], value=[74568984]"}
   {"level":"info","time":"2022-10-08T14:52:42+08:00","message":"RDB repl-stream-db: 0"}
   {"level":"info","time":"2022-10-08T14:52:42+08:00","message":"RDB AUX fields. key=[repl-id], value=[0d23bc30bd360693a56e78e71f6eee3f7c920dd5]"}
   {"level":"info","time":"2022-10-08T14:52:42+08:00","message":"RDB AUX fields. key=[repl-offset], value=[0]"}
   {"level":"info","time":"2022-10-08T14:52:42+08:00","message":"RDB AUX fields. key=[aof-preamble], value=[0]"}
   {"level":"info","time":"2022-10-08T14:52:42+08:00","message":"RDB resize db. db_size=[553969], expire_size=[1708]"}
   {"level":"info","time":"2022-10-08T14:52:42+08:00","message":"start save AOF. address=[10.0.1.14:6379]"}
   {"level":"info","time":"2022-10-08T14:52:42+08:00","message":"AOFWriter open file. filename=[0.aof]"}
   {"level":"info","time":"2022-10-08T14:52:45+08:00","message":"LUA script: [if (redis.call('hexists', KEYS[1], ARGV[3]) == 0) then return nil;end; local counter = redis.call('hincrby', KEYS[1], ARGV[3], -1); if (counter > 0) then redis.call('pexpire', KEYS[1], ARGV[2]); return 0; else redis.call('del', KEYS[1]); redis.call('publish', KEYS[2], ARGV[1]); return 1; end; return nil;]"}
   {"level":"info","time":"2022-10-08T14:52:45+08:00","message":"LUA script: [if (redis.call('exists', KEYS[1]) == 0) then redis.call('hset', KEYS[1], ARGV[2], 1); redis.call('pexpire', KEYS[1], ARGV[1]); return nil; end; if (redis.call('hexists', KEYS[1], ARGV[2]) == 1) then redis.call('hincrby', KEYS[1], ARGV[2], 1); redis.call('pexpire', KEYS[1], ARGV[1]); return nil; end; return redis.call('pttl', KEYS[1]);]"}
   {"level":"info","time":"2022-10-08T14:52:45+08:00","message":"send RDB finished. address=[10.0.1.14:6379], repl-stream-db=[0]"}
   {"level":"info","time":"2022-10-08T14:52:46+08:00","message":"syncing aof. allowOps=[110794.20], disallowOps=[0.00], entryId=[553970], InQueueEntriesCount=[0], unansweredBytesCount=[0]bytes, diff=[14], aofReceivedOffset=[14], aofAppliedOffset=[0]"}
   {"level":"info","time":"2022-10-08T14:52:46+08:00","message":"AOFReader open file. aof_filename=[0.aof]"}
   {"level":"info","time":"2022-10-08T14:52:51+08:00","message":"syncing aof. allowOps=[110796.80], disallowOps=[0.00], entryId=[553983], InQueueEntriesCount=[0], unansweredBytesCount=[0]bytes, diff=[0], aofReceivedOffset=[144991], aofAppliedOffset=[144991]"}