redis-server进程CPU百分百问题

本文涉及的产品
云数据库 Tair(兼容Redis),内存型 2GB
Redis 开源版,标准版 2GB
推荐场景:
搭建游戏排行榜
简介: 结论: 待确认是否为redis的BUG,原因是进程实际占用的内存远小于配置的最大内存,所以不会是内存不够需要淘汰。 CPU百分百redis-server进程集群状态: slave 解决办法: 使用gdb将d.ht[0].used的值改为0 问题原因: dictGetRandomKey()过程中, 无法走到分支“if (dictSize(d) == 0) return NULL;”, 导致函数dbRandomKey()进入死循环。
结论:
待确认是否为redis的BUG,原因是进程实际占用的内存远小于配置的最大内存,所以不会是内存不够需要淘汰。

CPU百分百redis-server进程集群状态:
slave

临时解决办法:
使用gdb将d.ht[0].used的值改为0

问题原因:
dictGetRandomKey()过程中,
无法走到分支“if (dictSize(d) == 0) return NULL;”,
导致函数dbRandomKey()进入死循环。

版本:
Redis server v=3.2.0 sha=00000000:0 malloc=jemalloc-4.0.3 bits=64 build=9894db3ef433c070

现象1:CPU百分百
PID   USER  PR NI VIRT  RES  SHR  S %CPU  %MEM TIME+   COMMAND                                                                                                                
25636 redis 20 0  38492 4096 1360 R 100.0 0.0  2578:10 redis-server

现象2:大量CLOSE_WAIT状态连接:
tcp     2417      0 1.49.26.98:11382      1.49.26.98:37268      CLOSE_WAIT  -                   
tcp     2521      0 1.49.26.98:11382      1.49.26.98:35141      CLOSE_WAIT  -                   
tcp     2521      0 1.49.26.98:11382      1.49.26.98:57181      CLOSE_WAIT  -

进程状态:
redis 25636 30.0 0.0 38492  4096 ? Rsl 3月23 2579:55 /data/redis/bin/redis-server *:1382 [cluster]

最大内存配置(1G):
maxmemory 1073741824

运行日志:
25636:S 28 Mar 00:21:24.526 - 1 clients connected (0 slaves), 1312384 bytes in use
25636:S 28 Mar 00:21:29.531 - DB 0: 1 keys (1 volatile) in 8 slots HT.
25636:S 28 Mar 00:21:29.531 - 1 clients connected (0 slaves), 1312384 bytes in use
25636:S 28 Mar 00:21:32.585 - Accepted 1.118.14.7:58132

调用栈:
#0  dictGenHashFunction (key=<optimized out>, len=5) at dict.c:123
#1  0x00000000004232e6 in dictFind (d=0x7f71c2a17240, key=key@entry=0x7f71c2a15001) at dict.c:499
#2  0x000000000043a00a in dbRandomKey (db=0x7f71c2a24800) at db.c:176
#3  0x000000000043a0a2 in randomkeyCommand (c=0x7f71c2aae1c0) at db.c:355
#4  0x0000000000426b95 in call (c=c@entry=0x7f71c2aae1c0, flags=flags@entry=15) at server.c:2221
#5  0x0000000000429ba7 in processCommand (c=0x7f71c2aae1c0) at server.c:2500
#6  0x0000000000436515 in processInputBuffer (c=0x7f71c2aae1c0) at networking.c:1296
#7  0x0000000000421338 in aeProcessEvents (eventLoop=eventLoop@entry=0x7f71c2a2e050, flags=flags@entry=3) at ae.c:412
#8  0x00000000004215eb in aeMain (eventLoop=0x7f71c2a2e050) at ae.c:455
#9  0x000000000041e5df in main (argc=2, argv=0x7ffef34b2418) at server.c:4079

#0  0x00007f71c2fbc3a2 in random () from /lib64/libc.so.6
#1  0x0000000000423745 in dictGetRandomKey (d=0x7f71c2a171e0) at dict.c:646
#2  0x0000000000439fc0 in dbRandomKey (db=0x7f71c2a24800) at db.c:171
#3  0x000000000043a0a2 in randomkeyCommand (c=0x7f71c2aae1c0) at db.c:355
#4  0x0000000000426b95 in call (c=c@entry=0x7f71c2aae1c0, flags=flags@entry=15) at server.c:2221
#5  0x0000000000429ba7 in processCommand (c=0x7f71c2aae1c0) at server.c:2500
#6  0x0000000000436515 in processInputBuffer (c=0x7f71c2aae1c0) at networking.c:1296
#7  0x0000000000421338 in aeProcessEvents (eventLoop=eventLoop@entry=0x7f71c2a2e050, flags=flags@entry=3) at ae.c:412
#8  0x00000000004215eb in aeMain (eventLoop=0x7f71c2a2e050) at ae.c:455
#9  0x000000000041e5df in main (argc=2, argv=0x7ffef34b2418) at server.c:4079

#0  0x00007f71c30e17e4 in __memcmp_sse4_1 () from /lib64/libc.so.6
#1  0x0000000000424219 in dictSdsKeyCompare (privdata=<optimized out>, key1=<optimized out>, key2=<optimized out>) at server.c:445
#2  0x000000000042331d in dictFind (d=0x7f71c2a17240, key=0x7f71c2a27e73) at dict.c:504
#3  0x0000000000439494 in getExpire (db=0x7f71c2a24800, key=0x7f71c2a27e60) at db.c:824
#4  0x0000000000439c4f in expireIfNeeded (db=0x7f71c2a24800, key=0x7f71c2a27e60) at db.c:858
#5  0x000000000043a01a in dbRandomKey (db=0x7f71c2a24800) at db.c:177
#6  0x000000000043a0a2 in randomkeyCommand (c=0x7f71c2aae1c0) at db.c:355
#7  0x0000000000426b95 in call (c=c@entry=0x7f71c2aae1c0, flags=flags@entry=15) at server.c:2221
#8  0x0000000000429ba7 in processCommand (c=0x7f71c2aae1c0) at server.c:2500
#9  0x0000000000436515 in processInputBuffer (c=0x7f71c2aae1c0) at networking.c:1296
#10 0x0000000000421338 in aeProcessEvents (eventLoop=eventLoop@entry=0x7f71c2a2e050, flags=flags@entry=3) at ae.c:412
#11 0x00000000004215eb in aeMain (eventLoop=0x7f71c2a2e050) at ae.c:455
#12 0x000000000041e5df in main (argc=2, argv=0x7ffef34b2418) at server.c:4079

#0  dictGetRandomKey (d=<optimized out>) at dict.c:663
#1  0x0000000000439fc0 in dbRandomKey (db=0x7f71c2a24800) at db.c:171
#2  0x000000000043a0a2 in randomkeyCommand (c=0x7f71c2aae1c0) at db.c:355
#3  0x0000000000426b95 in call (c=c@entry=0x7f71c2aae1c0, flags=flags@entry=15) at server.c:2221
#4  0x0000000000429ba7 in processCommand (c=0x7f71c2aae1c0) at server.c:2500
#5  0x0000000000436515 in processInputBuffer (c=0x7f71c2aae1c0) at networking.c:1296
#6  0x0000000000421338 in aeProcessEvents (eventLoop=eventLoop@entry=0x7f71c2a2e050, flags=flags@entry=3) at ae.c:412
#7  0x00000000004215eb in aeMain (eventLoop=0x7f71c2a2e050) at ae.c:455
#8  0x000000000041e5df in main (argc=2, argv=0x7ffef34b2418) at server.c:4079

猜测:
达到最大内存,进入淘汰keys逻辑,但没有keys符合淘汰,从而死循环。

相关代码:

/* Return a random key from the currently selected database. */
void randomkeyCommand(client *c) {
    robj *key;

    if ((key = dbRandomKey(c->db)) == NULL) {
        addReply(c,shared.nullbulk);
        return;
    }

    addReplyBulk(c,key);
    decrRefCount(key);
}

/* Return a random key, in form of a Redis object.
 * If there are no keys, NULL is returned.
 *
 * The function makes sure to return keys not already expired. */
robj *dbRandomKey(redisDb *db) {
    dictEntry *de;

    while(1) { // CPU百分百的原因,是这里死循环了
        sds key;
        robj *keyobj;

        de = dictGetRandomKey(db->dict);
        if (de == NULL) return NULL;

        key = dictGetKey(de);
        keyobj = createStringObject(key,sdslen(key));
        if (dictFind(db->expires,key)) {
            if (expireIfNeeded(db,keyobj)) {
                decrRefCount(keyobj);
                continue; /* search for another key. This expired. */
            }
        }
        return keyobj;
    }
}

void call(client *c, int flags) {
    long long dirty, start, duration;
    int client_old_flags = c->flags;

    /* Sent the command to clients in MONITOR mode, only if the commands are
     * not generated from reading an AOF. */
    if (listLength(server.monitors) &&
        !server.loading &&
        !(c->cmd->flags & (CMD_SKIP_MONITOR|CMD_ADMIN)))
    {
        replicationFeedMonitors(c,server.monitors,c->db->id,c->argv,c->argc);
    }

    /* Initialization: clear the flags that must be set by the command on
     * demand, and initialize the array for additional commands propagation. */
    c->flags &= ~(CLIENT_FORCE_AOF|CLIENT_FORCE_REPL|CLIENT_PREVENT_PROP);
    redisOpArrayInit(&server.also_propagate);

    /* Call the command. */
    dirty = server.dirty;
    start = ustime();
    c->cmd->proc(c);
    duration = ustime()-start;
    dirty = server.dirty-dirty;
    if (dirty < 0) dirty = 0;
    。。。。。。
}

/* With multiplexing we need to take per-client state.
 * Clients are taken in a linked list. */
typedef struct client {
    。。。。。。
    struct redisCommand *cmd, *lastcmd;  /* Last command executed. */
    。。。。。。
};

typedef void redisCommandProc(client *c);
typedef int *redisGetKeysProc(struct redisCommand *cmd, robj **argv, int argc, int *numkeys);
struct redisCommand {
    char *name;
    redisCommandProc *proc;
    int arity;
    char *sflags; /* Flags as string representation, one char per flag. */
    int flags;    /* The actual flags, obtained from the 'sflags' field. */
    /* Use a function to determine keys arguments in a command line.
     * Used for Redis Cluster redirect. */
    redisGetKeysProc *getkeys_proc;
    /* What keys should be loaded in background when calling this command? */
    int firstkey; /* The first argument that's a key (0 = no keys) */
    int lastkey;  /* The last argument that's a key */
    int keystep;  /* The step between first and last key */
    long long microseconds, calls;
};

/* This is our hash table structure. Every dictionary has two of this as we
 * implement incremental rehashing, for the old to the new table. */
typedef struct dictht {
    dictEntry **table;
    unsigned long size;
    unsigned long sizemask;
    unsigned long used;
} dictht;

typedef struct dict {
    dictType *type;
    void *privdata;
    dictht ht[2];
    long rehashidx; /* rehashing not in progress if rehashidx == -1 */
    int iterators; /* number of iterators currently running */
} dict;

/* Return a random entry from the hash table. Useful to
 * implement randomized algorithms */
dictEntry *dictGetRandomKey(dict *d)
{
    dictEntry *he, *orighe;
    unsigned int h;
    int listlen, listele;

    // (gdb) p *d
    // $1 = {type = 0x71d940 <dbDictType>, privdata = 0x0, ht = {{table = 0x7f71c2a1e480, size = 8, sizemask = 7, used = 1}, {table = 0x0, size = 0, sizemask = 0, used = 0}}, rehashidx = -1, iterators = 0}
    //
    // (gdb) p d.ht[0] 
    // $3 = {table = 0x7f71c2a1e480, size = 8, sizemask = 7, used = 1}
    // (gdb) p d.ht[1]
    // $4 = {table = 0x0, size = 0, sizemask = 0, used = 0}
    //
    // (gdb) set variable d.ht[0].used=0
    // (gdb) p d.ht[0].used      
    // $7 = 0

    // #define dictSize(d) ((d)->ht[0].used+(d)->ht[1].used)
    if (dictSize(d) == 0) return NULL;
    if (dictIsRehashing(d)) _dictRehashStep(d);
    if (dictIsRehashing(d)) {
        do {
            /* We are sure there are no elements in indexes from 0
             * to rehashidx-1 */
            h = d->rehashidx + (random() % (d->ht[0].size +
                                            d->ht[1].size -
                                            d->rehashidx));
            he = (h >= d->ht[0].size) ? d->ht[1].table[h - d->ht[0].size] :
                                      d->ht[0].table[h];
        } while(he == NULL);
    } else {
        do {
            h = random() & d->ht[0].sizemask;
            he = d->ht[0].table[h];
        } while(he == NULL);
    }

    /* Now we found a non empty bucket, but it is a linked
     * list and we need to get a random element from the list.
     * The only sane way to do so is counting the elements and
     * select a random index. */
    listlen = 0;
    orighe = he;
    while(he) {
        he = he->next;
        listlen++;
    }
    listele = random() % listlen;
    he = orighe;
    while(listele--) he = he->next;
    return he;
}

/* This function performs just a step of rehashing, and only if there are
 * no safe iterators bound to our hash table. When we have iterators in the
 * middle of a rehashing we can't mess with the two hash tables otherwise
 * some element can be missed or duplicated.
 *
 * This function is called by common lookup or update operations in the
 * dictionary so that the hash table automatically migrates from H1 to H2
 * while it is actively used. */
static void _dictRehashStep(dict *d) {
    if (d->iterators == 0) dictRehash(d,1);
}

进程内存(问题解决,退出死循环后才能看到,但结果和ps看到一致):
# Memory
used_memory:1375320
used_memory_human:1.31M
used_memory_rss:4321280
used_memory_rss_human:4.12M
used_memory_peak:2468448
used_memory_peak_human:2.35M
total_system_memory:33453797376
total_system_memory_human:31.16G
used_memory_lua:34816
used_memory_lua_human:34.00K
maxmemory:1073741824
maxmemory_human:1.00G
maxmemory_policy:allkeys-lru
mem_fragmentation_ratio:3.14
mem_allocator:jemalloc-4.0.3


相关实践学习
基于Redis实现在线游戏积分排行榜
本场景将介绍如何基于Redis数据库实现在线游戏中的游戏玩家积分排行榜功能。
云数据库 Redis 版使用教程
云数据库Redis版是兼容Redis协议标准的、提供持久化的内存数据库服务,基于高可靠双机热备架构及可无缝扩展的集群架构,满足高读写性能场景及容量需弹性变配的业务需求。 产品详情:https://www.aliyun.com/product/kvstore &nbsp; &nbsp; ------------------------------------------------------------------------- 阿里云数据库体验:数据库上云实战 开发者云会免费提供一台带自建MySQL的源数据库&nbsp;ECS 实例和一台目标数据库&nbsp;RDS实例。跟着指引,您可以一步步实现将ECS自建数据库迁移到目标数据库RDS。 点击下方链接,领取免费ECS&amp;RDS资源,30分钟完成数据库上云实战!https://developer.aliyun.com/adc/scenario/51eefbd1894e42f6bb9acacadd3f9121?spm=a2c6h.13788135.J_3257954370.9.4ba85f24utseFl
相关文章
|
29天前
|
存储 监控
【Azure Cloud Service】在Azure云服务中收集CPU监控指标和IIS进程的DUMP方法
在使用Cloud Service服务时,发现服务的CPU占用很高,在业务请求并不大的情况下,需要直到到底是什么进程占用了大量的CPU资源,已经如何获取IIS进程(w3wp.exe)的DUMP文件?
|
2月前
|
NoSQL Redis
Redis AOFRW主进程问题之更新manifest文件如何解决
Redis AOFRW主进程问题之更新manifest文件如何解决
|
3月前
|
弹性计算 Linux 区块链
Linux系统CPU异常占用(minerd 、tplink等挖矿进程)
Linux系统CPU异常占用(minerd 、tplink等挖矿进程)
94 4
Linux系统CPU异常占用(minerd 、tplink等挖矿进程)
|
2月前
|
Linux Python
在Linux中,如何查找系统中占用CPU最高的进程?
在Linux中,如何查找系统中占用CPU最高的进程?
|
2月前
|
NoSQL Redis
Redis AOF重写问题之减少CPU和fork开销如何解决
Redis AOF重写问题之减少CPU和fork开销如何解决
|
2月前
|
NoSQL 分布式数据库 Redis
Redis 进程问题之主进程更新manifest文件如何解决
Redis 进程问题之主进程更新manifest文件如何解决
|
2月前
|
Linux
在Linux中,如何查看占用CPU最多的进程?
在Linux中,如何查看占用CPU最多的进程?
|
2月前
|
数据安全/隐私保护 异构计算 Windows
【Azure 环境】 介绍两种常规的方法来监视Window系统的CPU高时的进程信息: Performance Monitor 和 Powershell Get-Counter
【Azure 环境】 介绍两种常规的方法来监视Window系统的CPU高时的进程信息: Performance Monitor 和 Powershell Get-Counter
|
2月前
|
缓存 NoSQL Redis
【Azure Redis 缓存】Redis的指标显示CPU为70%,而Service Load却达到了100%。这两个指标意义的解释及如何缓解呢?
【Azure Redis 缓存】Redis的指标显示CPU为70%,而Service Load却达到了100%。这两个指标意义的解释及如何缓解呢?
|
2月前
|
Linux
在Linux中,如何找出占用CPU或内存最多的进程?
在Linux中,如何找出占用CPU或内存最多的进程?