ceph 故障解决备忘

参考 ceph 环境

    cluster dc4f91c1-8792-4948-b68f-2fcea75f53b9
     health HEALTH_WARN 15 requests are blocked > 32 sec; clock skew detected on mon.hh-yun-ceph-cinder025-128075
     monmap e3: 5 mons at {hh-yun-ceph-cinder015-128055=240.30.128.55:6789/0,hh-yun-ceph-cinder017-128057=240.30.128.57:6789/0
,hh-yun-ceph-cinder024-128074=240.30.128.74:6789/0,hh-yun-ceph-cinder025-128075=240.30.128.75:6789/0,hh-yun-ceph-cinder026-128
076=240.30.128.76:6789/0}, election epoch 168, quorum 0,1,2,3,4 hh-yun-ceph-cinder015-128055,hh-yun-ceph-cinder017-128057,hh-y
un-ceph-cinder024-128074,hh-yun-ceph-cinder025-128075,hh-yun-ceph-cinder026-128076
     osdmap e27430: 100 osds: 100 up, 100 in
      pgmap v11224834: 20544 pgs, 2 pools, 70255 GB data, 17678 kobjects
            205 TB used, 157 TB / 363 TB avail
               20540 active+clean
                   4 active+clean+scrubbing+deep
  client io 57251 kB/s rd, 44602 kB/s wr, 3797 op/s

参考 ceph health detail 返回结果

1.  mon.hh-yun-ceph-cinder025-128075 addr 240.30.128.75:6789/0 clock skew 0.919947s > max 0.05s (latency 0.000544046s)
2.  15 requests are blocked

这里是具有两个常见错误
1. 时间不同步导致 mon 报警
2. 由于有硬件故障, 网络延时, 或其他原因导致客户端访问 ceph 存储超时

问题解决 (时间同步)
当前系统中的环境设定

[root@hh-yun-ceph-cinder015-128055 ceph]# ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok config show  | grep clock
  "mon_clock_drift_allowed": "0.05",   <- 当 mon 时间偏移 0.05 秒则不正常
  "mon_clock_drift_warn_backoff": "5",    <- 当出现 5 次偏移, 则报警
  "clock_offset": "0",                  <- mon 节点的时间偏移默认值

问题定位
检测各个机器的时间, 发现 hh-yun-ceph-cinder025-128075 节点时间偏移
修正方法

systemctl stop chronyd
ntpdate 10.199.129.21
systemctl start chronyd

当同步了时间并验证后, 需重启 mon 节点

/etc/init.d/ceph stop mon
/etc/init.d/ceph start mon

因为 mon 节点与客户非常连接, 因此, 在确保 mon 节点具有冗余情况下, 可以在生产时间段进行快速重启

问题解决 (15 requests are blocked)

参考信息

ceph health detail
HEALTH_WARN 14 requests are blocked > 32 sec; 11 osds have slow requests
7 ops are blocked > 536871 sec
2 ops are blocked > 268435 sec
2 ops are blocked > 67108.9 sec
3 ops are blocked > 33554.4 sec
1 ops are blocked > 536871 sec on osd.0
1 ops are blocked > 536871 sec on osd.10
2 ops are blocked > 536871 sec on osd.12
2 ops are blocked > 268435 sec on osd.18
1 ops are blocked > 536871 sec on osd.31
1 ops are blocked > 536871 sec on osd.38
1 ops are blocked > 67108.9 sec on osd.38
1 ops are blocked > 33554.4 sec on osd.48
1 ops are blocked > 67108.9 sec on osd.52
1 ops are blocked > 536871 sec on osd.63
1 ops are blocked > 33554.4 sec on osd.64
1 ops are blocked > 33554.4 sec on osd.69
11 osds have slow requests

上述信息, 发现, 访问被 block 而且时间很久,
解决方法

对上述 osd 进行一个一个的重启,  切记,  一个重启后,  数据 recovery 正常后才可以执行下一次的 osd 重启

/etc/init.d/ceph stop osd.0
/etc/init.d/ceph start osd.0

待数据恢复后才能够执行下一个 OSD 重启

恢复后解决

    cluster dc4f91c1-8792-4948-b68f-2fcea75f53b9
     health HEALTH_OK
     monmap e3: 5 mons at {hh-yun-ceph-cinder015-128055=240.30.128.55:6789/0,hh-yun-ceph-cinder017-128057=240.30.128.57:6789/0
,hh-yun-ceph-cinder024-128074=240.30.128.74:6789/0,hh-yun-ceph-cinder025-128075=240.30.128.75:6789/0,hh-yun-ceph-cinder026-128
076=240.30.128.76:6789/0}, election epoch 170, quorum 0,1,2,3,4 hh-yun-ceph-cinder015-128055,hh-yun-ceph-cinder017-128057,hh-y
un-ceph-cinder024-128074,hh-yun-ceph-cinder025-128075,hh-yun-ceph-cinder026-128076
     osdmap e27495: 100 osds: 100 up, 100 in
      pgmap v11231620: 20544 pgs, 2 pools, 70294 GB data, 17688 kobjects
            206 TB used, 157 TB / 363 TB avail
               20539 active+clean
                   5 active+clean+scrubbing+deep
  client io 973 kB/s rd, 22936 kB/s wr, 1334 op/s