PreView
Redis进阶-Redis键值设计及BigKey问题粗略的讨论了BigKey,这里我们继续再来研究下BigKey
Redis Version : 5.0.3
模拟写入一个BigKey
如何发现BigKey
总结下无非几个点
- 应用异常日志 (没法直接从日志关键字中看出来,因为这个bigkey阻塞以后,会引发一些奇怪的问题,比如从连接池获取不到连接等等)
- redis-cli --bigkeys
- 监控系统上报
- 其他…
redis-cli --bigkeys
先看下 redis-cli 都支持什么参数
[redis@artisan bin]$ ./redis-cli --help redis-cli 5.0.3 Usage: redis-cli [OPTIONS] [cmd [arg [arg ...]]] -h <hostname> Server hostname (default: 127.0.0.1). -p <port> Server port (default: 6379). -s <socket> Server socket (overrides hostname and port). -a <password> Password to use when connecting to the server. You can also use the REDISCLI_AUTH environment variable to pass this password more safely (if both are used, this argument takes predecence). -u <uri> Server URI. -r <repeat> Execute specified command N times. -i <interval> When -r is used, waits <interval> seconds per command. It is possible to specify sub-second times like -i 0.1. -n <db> Database number. -x Read last argument from STDIN. -d <delimiter> Multi-bulk delimiter in for raw formatting (default: \n). -c Enable cluster mode (follow -ASK and -MOVED redirections). --raw Use raw formatting for replies (default when STDOUT is not a tty). --no-raw Force formatted output even when STDOUT is not a tty. --csv Output in CSV format. --stat Print rolling stats about server: mem, clients, ... --latency Enter a special mode continuously sampling latency. If you use this mode in an interactive session it runs forever displaying real-time stats. Otherwise if --raw or --csv is specified, or if you redirect the output to a non TTY, it samples the latency for 1 second (you can use -i to change the interval), then produces a single output and exits. --latency-history Like --latency but tracking latency changes over time. Default time interval is 15 sec. Change it using -i. --latency-dist Shows latency as a spectrum, requires xterm 256 colors. Default time interval is 1 sec. Change it using -i. --lru-test <keys> Simulate a cache workload with an 80-20 distribution. --replica Simulate a replica showing commands received from the master. --rdb <filename> Transfer an RDB dump from remote server to local file. --pipe Transfer raw Redis protocol from stdin to server. --pipe-timeout <n> In --pipe mode, abort with error if after sending all data. no reply is received within <n> seconds. Default timeout: 30. Use 0 to wait forever. --bigkeys Sample Redis keys looking for big keys. --hotkeys Sample Redis keys looking for hot keys. only works when maxmemory-policy is *lfu. --scan List all keys using the SCAN command. --pattern <pat> Useful with --scan to specify a SCAN pattern. --intrinsic-latency <sec> Run a test to measure intrinsic system latency. The test will run for the specified amount of seconds. --eval <file> Send an EVAL command using the Lua script at <file>. --ldb Used with --eval enable the Redis Lua debugger. --ldb-sync-mode Like --ldb but uses the synchronous Lua debugger, in this mode the server is blocked and script changes are not rolled back from the server memory. --cluster <command> [args...] [opts...] Cluster Manager command and arguments (see below). --verbose Verbose mode. --no-auth-warning Don't show warning message when using password on command line interface. --help Output this help and exit. --version Output version and exit. Cluster Manager Commands: Use --cluster help to list all available cluster manager commands. Examples: cat /etc/passwd | redis-cli -x set mypasswd redis-cli get mypasswd redis-cli -r 100 lpush mylist x redis-cli -r 100 -i 1 info | grep used_memory_human: redis-cli --eval myscript.lua key1 key2 , arg1 arg2 arg3 redis-cli --scan --pattern '*:12345*' (Note: when using --eval the comma separates KEYS[] from ARGV[] items) When no command is given, redis-cli starts in interactive mode. Type "help" in interactive mode for information on available commands and settings. [redis@artisan bin]$
官方提供的工具
执行检测
[redis@artisan bin]$ ./redis-cli --bigkeys # Scanning the entire keyspace to find biggest keys as well as # average sizes per key type. You can use -i 0.1 to sleep 0.1 sec # per 100 SCAN commands (not usually needed). [00.00%] Biggest string found so far 'artisan43206' with 12 bytes [37.40%] Biggest string found so far 'artisanBIG' with 26890 bytes -------- summary ------- Sampled 100004 keys in the keyspace! Total key length in bytes is 1188921 (avg len 11.89) Biggest string found 'artisanBIG' has 26890 bytes 100004 strings with 1215805 bytes (100.00% of keys, avg size 12.16) 0 lists with 0 items (00.00% of keys, avg size 0.00) 0 sets with 0 members (00.00% of keys, avg size 0.00) 0 hashs with 0 fields (00.00% of keys, avg size 0.00) 0 zsets with 0 members (00.00% of keys, avg size 0.00) 0 streams with 0 entries (00.00% of keys, avg size 0.00) [redis@artisan bin]$
redis 也是通过scan方式对key进行统计,无需担心对redis造成阻塞 ,从结果上看 发现了 artisanBIG 这个bigkey
debug object
这个命令本身会阻塞Redis,如果 O(N) ,需要关注N的大小。
# 查看artisanBIG序列化后的长度 [redis@artisan bin]$ ./redis-cli debug object artisanBIG Value at:0x7fa527edd0b0 refcount:1 encoding:raw serializedlength:5044 lru:7863297 lru_seconds_idle:112
输出参数说明:
- Value at:key的内存地址
- refcount:引用次数
- encoding:编码类型
- serializedlength:序列化长度 5044 —> 5KB ,并不代表真实的大小,仅供参考
- lru_seconds_idle:空闲时间
- …
object命令: https://redis.io/commands/object
如何优雅的删除BigKey (lazy delete)
- 传统的方式: DEL 呗
DEL 毫无疑问,阻塞命令,如果 O(N) N 很大,需要考虑。
- 另外一点需要考虑的: 有些key 设置了过期时间、rename操作等,这些key redis 自己维护 ,那处理这些BigKey , 同样的会阻塞。 并且这行操作不会记录到慢查询中,你都发现不了。 (慢查询只记录客户端的行为,Redis server本身的行为并不会被记录
- 推荐: Redis 4.0 提供了 lazy delete (unlink命令)
那4.0 一下的版本,怎么处理这些bigKey呢? -----》 scan , 分批次删除
我们重点来看 lazy delete -------------> UNLINK 命令
UNLINK 官方文档: https://redis.io/commands/unlink
127.0.0.1:6379> UNLINK artisanBIG (integer) 1 127.0.0.1:6379>
关于lazyfree的几个配置项
- lazyfree-lazy-eviction:是否异步驱逐key,当内存达到上限,分配失败后
- lazyfree-lazy-expire:是否异步进行key过期事件的处理
- lazyfree-lazy-server-del:del命令是否异步执行删除操作,类似unlink
- replica-lazy-flush:replica client做全同步的时候,是否异步flush本地db
默认都是no,按需开启
如何优化bigKey
- 优化数据结构 : 字符串? hash? 。。。。
- 合理拆分key : 1个大key 拆分成N个小key
- 命令优化 : getall–> mget
- 物理隔离或者升级网卡(治标不治本)