发现问题
CDH-4.7.1 NameNode is down
启动NameNode报错如下，无法创建新的线程，可能是使用的线程数超过max user processes设定的阈值
2018-08-26 08:44:00,532 INFO org.apache.hadoop.http.HttpServer: Jetty bound to port 50070 2018-08-26 08:44:00,532 INFO org.mortbay.log: jetty-6.1.26.cloudera.4 2018-08-26 08:44:00,773 WARN org.apache.hadoop.security.authentication.server.AuthenticationFilter: 'signature.secret' configuration not set, using a random value as secret 2018-08-26 08:44:00,812 INFO org.mortbay.log: Started SelectChannelConnector@alish1-dataservice-01.mypna.cn:50070 2018-08-26 08:44:00,813 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Web-server up at: alish1-dataservice-01.mypna.cn:50070 2018-08-26 08:44:00,814 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting 2018-08-26 08:44:00,815 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 8020: starting 2018-08-26 08:44:00,828 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting 2018-08-26 08:44:00,828 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 8022: starting 2018-08-26 08:44:00,839 FATAL org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join java.lang.OutOfMemoryError: unable to create new native thread at java.lang.Thread.start0(Native Method) at java.lang.Thread.start(Thread.java:714) at org.apache.hadoop.ipc.Server.start(Server.java:2057) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.start(NameNodeRpcServer.java:303) at org.apache.hadoop.hdfs.server.namenode.NameNode.startCommonServices(NameNode.java:497) at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:459) at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:621) at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:606) at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1177) at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1241) 2018-08-26 08:44:00,851 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 1
日志内容如下，检查DNS没有问题，这里没有太多参考意义
#cat /var/log/cloudera-scm-agent/cloudera-scm-agent.log [26/Aug/2018 07:30:23 +0000] 4589 MainThread agent INFO PID '19586' associated with process '1724-hdfs-NAMENODE' with payload 'processname:1724-hdfs-NAMENODE groupname:1724-hdfs-NAMENODE from_state:RUNNING expected:0 pid:19586' exited unexpectedly [26/Aug/2018 07:45:06 +0000] 4589 Monitor-HostMonitor throttling_logger ERROR (29 skipped) Failed to collect java-based DNS names Traceback (most recent call last): File "/usr/lib64/cmf/agent/src/cmf/monitor/host/dns_names.py", line 53, in collect result, stdout, stderr = self._subprocess_with_timeout(args, self._poll_timeout) File "/usr/lib64/cmf/agent/src/cmf/monitor/host/dns_names.py", line 42, in _subprocess_with_timeout return subprocess_with_timeout(args, timeout) File "/usr/lib64/cmf/agent/src/cmf/monitor/host/subprocess_timeout.py", line 40, in subprocess_with_timeout close_fds=True) File "/usr/lib64/python2.6/subprocess.py", line 642, in __init__ errread, errwrite) File "/usr/lib64/python2.6/subprocess.py", line 1234, in _execute_child child_exception = pickle.loads(data) OSError: [Errno 2] No such file or directory
故障排查
这里设置的max user processes为65535已经非常大了，一般来说是达不到这个瓶颈的
# ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 127452 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 65535 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 10240 cpu time (seconds, -t) unlimited max user processes (-u) 65535 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited
现在系统的总进程数仅仅一百多个，我们要检查每个进程对应有多少个线程
# ps -ef|wc -l
169
已知这台服务器上主要跑的是java进程，所以重点查看java进程对应的线程数，找到30315这个进程对应约32110个线程，在加上其他进程和线程数，总数超过65535，NameNode无法在申请到多余的线程，所以报错
# pgrep java
1680
5482
19662
28770
30315
35902
# for i in `pgrep java`; do ps -T -p $i |wc -l; done
15
49
30
53
32110
114
# ps -T -p 30315|wc -l
32110
或者通过top -H 命令查看
# top -H
top - 10:44:58 up 779 days, 19:34, 3 users, load average: 0.01, 0.05, 0.05
Tasks: 32621 total, 1 running, 32620 sleeping, 0 stopped, 0 zombie
Cpu(s): 2.8%us, 4.1%sy, 0.0%ni, 93.1%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 16334284k total, 15879392k used, 454892k free, 381132k buffers
Swap: 4194296k total, 0k used, 4194296k free, 8304400k cached
解决方法
找到了问题的原因，我们可以重新设定max user processes的值为100000，再次启动NameNode成功
#echo "100000" > /proc/sys/kernel/threads-max
#echo "100000" > /proc/sys/kernel/pid_max （默认32768）
#echo "200000" > /proc/sys/vm/max_map_count （默认65530）
#vim /etc/security/limits.d/90-nproc.conf
* soft nproc unlimited
root soft nproc unlimited
#vim /etc/security/limits.conf
* soft nofile 65535
* hard nofile 65535
* hard nproc 100000
* soft nproc 100000
# ulimit -u
100000
