本篇博文为整理网络上Hadoop-HA搭建后出来的博客,参考网址为:http://blog.chinaunix.net/uid-196700-id-5751309.html
3. 部署
共5台机器(zookeeper部署在这5台机器上),部署如下表所示:
NameNode |
JournalNode |
DataNode |
ZooKeeper |
192.168.106.91 192.168.106.92 |
192.168.106.91 192.168.106.92 192.168.106.93 |
192.168.106.93 192.168.106.94 192.168.106.95 |
192.168.106.101 192.168.106.102 192.168.106.103 |
机器IP |
对应的主机名 |
192.168.106.91 |
hadoop1 |
192.168.106.92 |
hadoop2 |
192.168.106.93 |
hadoop3 |
192.168.106.94 |
hadoop4 |
192.168.106.95 |
hadoop5 |
192.168.106.101 |
hadoop11 |
192.168.106.102 |
hadoop12 |
192.168.106.103 |
hadoop13 |
注意主机名不能有下划线,否则启动时,SecondaryNameNode节点会报错
3.2.2. 永久修改主机名(注意:这一步一定要做)
不同的Linux发行版本,对应的系统配置文件可能不同,Centos6.7通过/etc/hosts:
[root@hadoop1 hadoop]# hostname hadoop1 |
查看主机配置信息cat /etc/hosts:
[root@hadoop1 hadoop]# cat /etc/hosts 192.168.106.83 hadoop-father 192.168.106.91 hadoop1 192.168.106.92 hadoop2 192.168.106.93 hadoop3 192.168.106.94 hadoop4 192.168.106.95 hadoop5 192.168.106.96 hadoop6 192.168.106.97 hadoop7 192.168.106.98 hadoop8 192.168.106.100 hadoop10 192.168.106.101 hadoop11 192.168.106.102 hadoop12 192.168.106.103 hadoop13 |
关于虚拟机克隆,hostname修改,ip修改,参考博文:
http://blog.csdn.net/tototuzuoquan/article/details/53999173
win10下虚拟机联网问题,参考:
http://blog.csdn.net/tototuzuoquan/article/details/53900836
3.3. 免密码登录范围
要求能通过免登录包括使用IP和主机名都能免密码登录:
1) NameNode能免密码登录所有的DataNode
2) 各NameNode能免密码登录自己
3) 各NameNode间能免密码互登录
4) DataNode能免密码登录自己
5) DataNode不需要配置免密码登录NameNode和其它DataNode。
注:免密码登录不是必须的,如果不使用hadoop-daemons.sh等需要ssh、scp的脚本。
以下的免密登录方案是针对Centos6.7上情况:
hadoop1下:
ssh-keygen -t rsa (直接打Enter,直到执行完毕)
然后执行:
ssh-copy-id hadoop1 ssh-copy-id hadoop2 ssh-copy-id hadoop3 ssh-copy-id hadoop4 ssh-copy-id hadoop5 |
通过上面的配置,可以在hadoop1上免密登录hadoop1、hadoop2、hadoop3、hadoop4、hadoop5
hadoop2下:
ssh-keygen -t rsa (直接打Enter,直到执行完毕)
然后执行:
ssh-copy-id hadoop1 ssh-copy-id hadoop2 ssh-copy-id hadoop3 ssh-copy-id hadoop4 ssh-copy-id hadoop5 |
通过上面的配置,可以在hadoop1上免密登录hadoop1、hadoop2、hadoop3、hadoop4、hadoop5
hadoop3下:
ssh-keygen -t rsa (直接打Enter,直到执行完毕)
然后执行:
ssh-copy-id hadoop3 |
hadoop4下:
ssh-keygen -t rsa (直接打Enter,直到执行完毕)
然后执行:
ssh-copy-id hadoop4 |
hadoop5下:
ssh-keygen -t rsa (直接打Enter,直到执行完毕)
然后执行:
ssh-copy-id hadoop5 |
下面是其它关于免密登录的资料
建议生成的私钥和公钥文件名都带上自己的IP,否则会有些混乱。
按照中免密码登录范围的说明,配置好所有的免密码登录。更多关于免密码登录说明,请浏览技术博客:
1) http://blog.chinaunix.net/uid-20682147-id-4212099.html(两个SSH2间免密码登录)
2) http://blog.chinaunix.net/uid-20682147-id-4212097.html(SSH2免密码登录OpenSSH)
3) http://blog.chinaunix.net/uid-20682147-id-4212094.html(OpenSSH免密码登录SSH2)
4) http://blog.chinaunix.net/uid-20682147-id-5520240.html(两个openssh间免密码登录)
为便于讲解,本文约定Hadoop、JDK安装目录如下:
Jdk |
/usr/local/jdk1.8.0_73 |
hadoop |
/home/tuzq/software/hadoop-2.8.0 |
在实际安装部署时,可以根据实际进行修改。
为运行Hadoop(HDFS、YARN和MapReduce)需要完成的工作详单:
Hadoop是Java语言开发的,所以需要。 |
|
NameNode控制SecondaryNameNode和DataNode使用了ssh和scp命令,需要无密码执行。 |
|
Hadoop安装和配置 |
这里指的是HDFS、YARN和MapReduce,不包含HBase、Hive等的安装。 |
本文安装的JDK1.8.0_73版本。关于JDK的安装,参考:http://blog.csdn.net/tototuzuoquan/article/details/18188109
此处略。
最后配置的java的环境变量是:
export JAVA_HOME=/usr/local/jdk1.8.0_73 export CLASSPATH=$JAVA_HOME/lib/tools.jar export PATH=$JAVA_HOME/bin:$PATH |
完成这项操作之后,需要重新登录,或source一下profile文件,以便环境变量生效,当然也可以手工运行一下,以即时生效。如果还不放心,可以运行下java或javac,看看命令是否可执行。如果在安装JDK之前,已经可执行了,则表示不用安装JDK。
本部分仅包括HDFS、MapReduce和Yarn的安装,不包括HBase、Hive等的安装。
此处略,直接进入官网,下载hadoop-2.8.0.tar.gz。关于源码编译的可以参考:
源码编译:
http://blog.csdn.net/tototuzuoquan/article/details/72796632
hadoop伪分布式集群安装:
http://blog.csdn.net/tototuzuoquan/article/details/72798435
1) 将下载好的hadoop安装包hadoop-2.8.0.tar.gz上传到/home/tuzq/software目录下
2) 进入/home/tuzq/software目录
3) 在/home/tuzq/software目录下,解压安装包hadoop-2.8.0.tar.gz:tar xzf hadoop-2.8.0.tar.gz,将自己在Linux上编译好的hadoop中的lib/native替换刚刚解压好的hadoop-2.8.0/lib/native中的内容。
4) 修改用户主目录下的文件.profile(当然也可以是/etc/profile或其它同等效果的文件),设置Hadoop环境变量:
export JAVA_HOME=/usr/local/jdk1.8.0_73 export HADOOP_HOME=/home/tuzq/software/hadoop-2.8.0 export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop export PATH=$PATH:$HADOOP_HOME/bin |
需要重新登录以生效,或者在终端上执行source /etc/profile,让export HADOOP_HOME=/home/tuzq/software/hadoop-2.8.0即时生效。
8.3. 修改hadoop-env.sh(hadoop1机器上为例)
修改所有节点上的$HADOOP_HOME/etc/hadoop/hadoop-env.sh文件,在靠近文件头部分加入:export JAVA_HOME=/usr/local/jdk1.8.0_73
特别说明一下:虽然在/etc/profile已经添加了JAVA_HOME,但仍然得修改所有节点上的hadoop-env.sh,否则启动时,报如下所示的错误:
ip: Error: JAVA_HOME is not set and could not be found. ip: Error: JAVA_HOME is not set and could not be found. ip: Error: JAVA_HOME is not set and could not be found. ip: Error: JAVA_HOME is not set and could not be found. ip: Error: JAVA_HOME is not set and could not be found. ip: Error: JAVA_HOME is not set and could not be found. |
除JAVA_HOME之外,再添加:
export HADOOP_HOME=/home/tuzq/software/hadoop-2.8.0
export HADOOP_CONF_DIR=${HADOOP_HOME}/etc/hadoop
效果如下:
同时,建议将下列添加到/etc/profile或~/.profile中:
export JAVA_HOME=/usr/local/jdk1.8.0_73
export HADOOP_HOME=/home/tuzq/software/hadoop-2.8.0
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
slaves即为HDFS的DataNode节点。当使用脚本start-dfs.sh来启动hdfs时,会使用到这个文件,以无密码登录方式到各slaves上启动DataNode。
修改主NameNode和备NameNode上的$HADOOP_HOME/etc/hadoop/slaves文件,将slaves的节点IP(也可以是相应的主机名)一个个加进去,一行一个IP,如下所示:
>cat slaves hadoop3 hadoop4 hadoop5 |
配置文件放在$HADOOP_HOME/etc/hadoop目录下,对于Hadoop 2.3.0、Hadoop 2.7.2和Hadoop 2.7.2版本,该目录下的core-site.xml、yarn-site.xml、hdfs-site.xml和mapred-site.xml都是空的。如果不配置好就启动,如执行start-dfs.sh,则会遇到各种错误。
可从$HADOOP_HOME/share/hadoop目录下拷贝一份到/etc/hadoop目录,然后在此基础上进行修改(以下内容可以直接拷贝执行,2.3.0版本中各default.xml文件路径不同于2.7.2版本):
# 进入$HADOOP_HOME目录 cd $HADOOP_HOME cp ./share/doc/hadoop/hadoop-project-dist/hadoop-common/core-default.xml ./etc/hadoop/core-site.xml cp ./share/doc/hadoop/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml ./etc/hadoop/hdfs-site.xml cp ./share/doc/hadoop/hadoop-yarn/hadoop-yarn-common/yarn-default.xml ./etc/hadoop/yarn-site.xml cp ./share/doc/hadoop/hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-default.xml ./etc/hadoop/mapred-site.xml |
接下来,需要对默认的core-site.xml、yarn-site.xml、hdfs-site.xml和mapred-site.xml进行适当的修改,否则仍然无法启动成功。
QJM的配置参照的官方文档:
对core-site.xml文件的修改,涉及下表中的属性:
属性名 |
属性值 |
说明 |
fs.defaultFS |
hdfs://mycluster |
|
fs.default.name |
hdfs://mycluster |
按理应当不用填写这个参数,因为fs.defaultFS已取代它,但启动时报错: fs.defaultFS is file:/// |
hadoop.tmp.dir |
/home/tuzq/software/hadoop-2.8.0/tmp |
|
ha.zookeeper.quorum |
hadoop11:2181,hadoop12:2181,hadoop13:2181 |
|
ha.zookeeper.parent-znode |
/mycluster/hadoop-ha |
|
io.seqfile.local.dir |
|
默认值为${hadoop.tmp.dir}/io/local |
fs.s3.buffer.dir |
|
默认值为${hadoop.tmp.dir}/s3 |
fs.s3a.buffer.dir |
|
默认值为${hadoop.tmp.dir}/s3a |
实际部署的时候的一个参考配置文件如下:
<?xml version="1.0" encoding="UTF-8"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration> <property> <name>fs.defaultFS</name> <value>hdfs://mycluster</value> </property> <property> <name>fs.default.name</name> <value>hdfs://mycluster</value> </property> <property> <name>hadoop.tmp.dir</name> <value>/home/tuzq/software/hadoop-2.8.0/tmp</value> </property>
<property> <name>ha.zookeeper.quorum</name> <value>hadoop11:2181,hadoop12:2181,hadoop13:2181</value> </property>
<property> <name>ha.zookeeper.parent-znode</name> <value>/mycluster/hadoop-ha</value> </property> </configuration> |
注意启动之前,需要将配置的目录创建好,如创建好/home/tuzq/software/current/tmp目录。详细可参考:
http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/core-default.xm。
8.7.1. dfs.namenode.rpc-address
如果没有配置,则启动时报如下错误:
Incorrect configuration: namenode address dfs.namenode.servicerpc-address or dfs.namenode.rpc-address is not configured. |
这里需要指定IP和端口,如果只指定了IP,如192.168.106.91,则启动时输出如下:
Starting namenodes on [] |
改成“hadoop1:8020”后,则启动时输出为:
Starting namenodes on [192.168.106.91] |
8.8. 修改hdfs-site.xml
对hdfs-site.xml文件的修改,涉及下表中的属性:
属性名 |
属性值 |
说明 |
dfs.nameservices |
mycluster |
|
dfs.ha.namenodes.mycluster |
nn1,nn2 |
同一个nameservice下,只能配置一个或两个,也就是说不能有nn3 |
dfs.namenode.rpc-address.mycluster.nn1 |
hadoop1:8020 |
|
dfs.namenode.rpc-address.mycluster.nn2 |
Hadoop2:8020 |
|
dfs.namenode.http-address.mycluster.nn1 |
hadoop1:50070 |
|
dfs.namenode.http-address.mycluster.nn2 |
hadoop2:50070 |
|
dfs.namenode.shared.edits.dir |
qjournal://hadoop1:8485;hadoop2:8485;hadoop3:8485/mycluster |
至少三台Quorum Journal节点配置 |
|
|
|
dfs.client.failover.proxy.provider.mycluster |
org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider |
客户端通过它来找NameNode |
|
|
|
dfs.ha.fencing.methods |
sshfence
如果配置为sshfence,当主NameNode异常时,使用ssh登录到主NameNode,然后使用fuser将主NameNode杀死,因此需要确保所有NameNode上可以使用fuser。 |
用来保证同一时刻只有一个主NameNode,以防止脑裂。可带用户名和端口参数,格式示例:sshfence([[username][:port]]);值还可以为shell脚本,格式示例: shell(/path/to/my/script.sh arg1 arg2 ...),如: shell(/bin/true) |
dfs.ha.fencing.ssh.private-key-files |
/root/.ssh/id_rsa |
指定私钥,如果是OpenSSL,则值为/root/.ssh/id_rsa |
dfs.ha.fencing.ssh.connect-timeout |
30000 |
可选的配置 |
dfs.journalnode.edits.dir |
/home/tuzq/software/hadoop-2.8.0/journal |
JournalNode存储其本地状态的位置,在JournalNode机器上的绝对路径,JNs的edits和其它本地状态被存储在此处 |
dfs.datanode.data.dir |
/home/tuzq/software/hadoop-2.8.0/data/data |
|
dfs.namenode.name.dir |
/home/tuzq/software/hadoop-2.8.0/data/name |
NameNode元数据存放目录,默认值为file://${hadoop.tmp.dir}/dfs/name,也就是在临时目录下,可以考虑放到数据目录下 |
dfs.namenode.checkpoint.dir |
|
默认值为file://${hadoop.tmp.dir}/dfs/namesecondary,但如果没有启用SecondaryNameNode,则不需要 |
dfs.ha.automatic-failover.enabled |
true |
自动主备切换 |
|
|
|
dfs.datanode.max.xcievers |
4096 |
可选修改,类似于linux的最大可打开的文件个数,默认为256,建议设置成大一点。同时,需要保证系统可打开的文件个数足够(可通过ulimit命令查看)。该错误会导致hbase报“notservingregionexception”。 |
dfs.journalnode.rpc-address |
0.0.0.0:8485 |
配置JournalNode的RPC端口号,默认为0.0.0.0:8485,可以不用修改 |
详细配置可参考:
http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml。
实际部署的时候的一个参考配置文件:
<?xml version="1.0" encoding="UTF-8"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration> <property> <name>dfs.nameservices</name> <value>mycluster</value> </property> <!--同一nameservice下,只能配置一或两个NameNode,也就是说不能有nn3,分別是nn1,nn2--> <property> <name>dfs.ha.namenodes.mycluster</name> <value>nn1,nn2</value> </property> <!-- nn1的RPC通信地址 --> <property> <name>dfs.namenode.rpc-address.mycluster.nn1</name> <value>hadoop1:8020</value> </property> <!--nn1的http通信地址--> <property> <name>dfs.namenode.http-address.mycluster.nn1</name> <value>hadoop1:50070</value> </property> <!--nn2的RPC通信地址--> <property> <name>dfs.namenode.rpc-address.mycluster.nn2</name> <value>hadoop2:8020</value> </property> <!--nn2的http通信地址--> <property> <name>dfs.namenode.http-address.mycluster.nn2</name> <value>hadoop2:50070</value> </property>
<!--指定NameNode的edits元数据在JournalNode上的存放位置,这也是一个集群,至少3台Quorum Journal节点配置--> <property> <name>dfs.namenode.shared.edits.dir</name> <value>qjournal://hadoop1:8485;hadoop2:8485;hadoop3:8485/mycluster</value> </property>
<!-- JournalNode存储其本地状态的位置,在JournalNode机器上的绝对路径,JNs的edits 和其他本地状态将被存储在此处 --> <property> <name>dfs.journalnode.edits.dir</name> <value>/home/tuzq/software/hadoop-2.8.0/journal</value> </property>
<!--配置JournalNode的RPC端口号,默认为0.0.0.0:8485,可以不用修改--> <!-- <property> <name>dfs.journalnode.rpc-address</name> <property>0.0.0.0:8485</property> </property> -->
<!--开启NameNode失败自动切换,自动主备切换--> <property> <name>dfs.ha.automatic-failover.enabled</name> <value>true</value> </property>
<!-- 配置失败自动切换实现方式,切换的时候用哪种控制器,不同的名称服务可以有不同的自动切换方式, 客户端通过它来找主NameNode --> <property> <name>dfs.client.failover.proxy.provider.mycluster</name> <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value> </property>
<!-- 配置隔离机制方法,多个机制用换行分割,即每个机制暂用一行
如果配置为sshfence,当主NameNode异常时,使用ssh登录到主NameNode,然后使用fuser将主NameNode杀死,因此需要确保所有NameNode 上可以使用fuser;
用来保证同一时刻只有一个主NameNode,以防止脑裂。可带用户名和端口参数,格式示例:sshfence([[username][:port]]);值 还可以为shell脚本,格式示例:shell(/bin/true),如果sshd不是默认的22端口时,就需要指定。 --> <property> <name>dfs.ha.fencing.methods</name> <value> sshfence shell(/bin/true) </value> </property>
<!-- 使用sshfence隔离机制时需要ssh免登陆,指定私钥,下面是OpenSSL --> <property> <name>dfs.ha.fencing.ssh.private-key-files</name> <value>/root/.ssh/id_rsa</value> </property>
<!-- 配置sshfence隔离机制超时时间 --> <property> <name>dfs.ha.fencing.ssh.connect-timeout</name> <value>30000</value> </property>
<!-- NameNode元数据存放目录,默认值为file://${hadoop.tmp.dir}/dfs/name, 也就是在临时目录下,可以考虑放到数据目录下 --> <property> <name>dfs.datanode.data.dir</name> <value>/home/tuzq/software/hadoop-2.8.0/data/data</value> </property>
<property> <name>dfs.namenode.name.dir</name> <value>/home/tuzq/software/hadoop-2.8.0/data/name</value> </property>
<!-- 可选修改,类似于Linux的最大可打开的文件个数,默认为256,建议设置成大一点。同时, 需要保证系统可打开的文件个数足够(可通过ulimit命令查看),该错误会导致hbase报 "notservingregionexception" --> <property> <name>dfs.datanode.max.xcievers</name> <value>4096</value> </property>
</configuration> |
8.9. 修改mapred-site.xml
对hdfs-site.xml文件的修改,涉及下表中的属性:
属性名 |
属性值 |
涉及范围 |
mapreduce.framework.name |
yarn |
所有mapreduce节点 |
实际部署中的一个参考配置如下:
<?xml version="1.0"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration> <property> <name>mapreduce.framework.name</name> <value>yarn</value> </property> </configuration> |
详细配置可参考:
对yarn-site.xml文件的修改,涉及下表中的属性:
属性名 |
属性值 |
涉及范围 |
|
|
HA模式可不配置,但由于其它配置项可能有引用它,建议保持值为0.0.0.0,如果没有被引用到,则可不配置。 |
yarn.nodemanager.hostname |
0.0.0.0 |
|
yarn.nodemanager.aux-services |
mapreduce_shuffle |
|
以下为HA相关的配置,包括自动切换(可仅可在ResourceManager节点上配置) |
||
yarn.resourcemanager.ha.enabled |
true |
启用HA |
yarn.resourcemanager.cluster-id |
yarn-cluster |
可不同于HDFS的 |
yarn.resourcemanager.ha.rm-ids |
rm1,rm2 |
注意NodeManager要和ResourceManager一样配置 |
yarn.resourcemanager.hostname.rm1 |
hadoop1 |
|
yarn.resourcemanager.hostname.rm2 |
hadoop2 |
|
yarn.resourcemanager.webapp.address.rm1 |
hadoop1:8088 |
在浏览器上访问:http://hadoop1:8088,可以看到yarn的信息 |
yarn.resourcemanager.webapp.address.rm2 |
hadoop2:8088 |
在浏览器上访问:http://hadoop2:8088,可以看到yarn的信息 |
yarn.resourcemanager.zk-address |
hadoop11:2181,hadoop12:2182,hadoop13:2181 |
|
yarn.resourcemanager.ha.automatic-failover.enable |
true |
可不配置,因为当yarn.resourcemanager.ha.enabled为true时,它的默认值即为true |
以下为NodeManager配置 |
||
yarn.nodemanager.vmem-pmem-ratio |
|
每使用1MB物理内存,最多可用的虚拟内存数,默认值为2.1,在运行spark-sql时如果遇到“Yarn application has already exited with state FINISHED”,则应当检查NodeManager的日志,以查看是否该配置偏小原因 |
yarn.nodemanager.resource.cpu-vcores |
|
NodeManager总的可用虚拟CPU个数,默认值为8 |
yarn.nodemanager.resource.memory-mb |
|
该节点上YARN可使用的物理内存总量,默认是8192(MB) |
yarn.nodemanager.pmem-check-enabled |
|
是否启动一个线程检查每个任务正使用的物理内存量,如果任务超出分配值,则直接将其杀掉,默认是true |
yarn.nodemanager.vmem-check-enabled |
|
是否启动一个线程检查每个任务正使用的虚拟内存量,如果任务超出分配值,则直接将其杀掉,默认是true |
以下为ResourceManager配置 |
||
yarn.scheduler.minimum-allocation-mb |
|
单个容器可申请的最小内存 |
yarn.scheduler.maximum-allocation-mb |
|
单个容器可申请的最大内存 |
实际部署的时候一个参考配置:
<?xml version="1.0"?>
<configuration> <!--启用HA--> <property> <name>yarn.resourcemanager.ha.enabled</name> <value>true</value> </property>
<!--指定RM的cluster id--> <property> <name>yarn.resourcemanager.cluster-id</name> <value>yarn-cluster</value> </property>
<!-- 指定RM的名字 --> <property> <name>yarn.resourcemanager.ha.rm-ids</name> <value>rm1,rm2</value> </property>
<!--分别指定RM的地址--> <property> <name>yarn.resourcemanager.hostname.rm1</name> <value>hadoop1</value> </property> <property> <name>yarn.resourcemanager.hostname.rm2</name> <value>hadoop2</value> </property> <property> <name>yarn.resourcemanager.webapp.address.rm1</name> <value>hadoop1:8088</value> </property> <property> <name>yarn.resourcemanager.webapp.address.rm2</name> <value>hadoop2:8088</value> </property> <!--指定zk集群地址--> <property> <name>yarn.resourcemanager.zk-address</name> <value>hadoop11:2181,hadoop12:2182,hadoop13:2181</value> </property>
<!-- yarn中的nodemanager是否要提供一些辅助的服务 --> <property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> </property>
</configuration> |
yarn.nodemanager.hostname如果配置成具体的IP,则会导致每个NodeManager的配置不同。详细配置可参考:
http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-common/yarn-default.xml。
Yarn HA的配置可以参考:
https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/ResourceManagerHA.html。
在hadoop1上配置完成后执行:
scp -r /home/toto/software/hadoop-2.8.0/etc/hadoop/* root@hadoop2:/home/tuzq/software/hadoop-2.8.0/etc/hadoop
scp -r /home/toto/software/hadoop-2.8.0/etc/hadoop/* root@hadoop3:/home/tuzq/software/hadoop-2.8.0/etc/hadoop
scp -r /home/toto/software/hadoop-2.8.0/etc/hadoop/* root@hadoop4:/home/tuzq/software/hadoop-2.8.0/etc/hadoop
scp -r /home/toto/software/hadoop-2.8.0/etc/hadoop/* root@hadoop5:/home/tuzq/software/hadoop-2.8.0/etc/hadoop
Zookeeper -> JournalNode -> 格式化NameNode -> 初始化JournalNode
-> 创建命名空间(zkfc) -> NameNode -> DataNode -> ResourceManager -> NodeManager。
但请注意首次启动NameNode之前,得先做format,也请注意备NameNode的启动方法。
在启动HDFS之前,需要先完成对NameNode的格式化。
mkdir -p /home/tuzq/software/hadoop-2.8.0/tmp/dfs/name
./zkServer.sh start
注意在启动其它之前先启动zookeeper。
在其中一个namenode(hadoop1)上执行:
bin/hdfs zkfc -formatZK
NameNode将元数据操作日志记录在JournalNode上,主备NameNode通过记录在JouralNode上的日志完成元数据同步。
在所有JournalNode上执行:
sbin/hadoop-daemon.sh start journalnode
注意,在执行“hdfs namenode -format”之前,必须先启动好JournalNode,而format又必须在启动namenode之前。
10.5初始化namenode
进入hadoop1接着执行下面的命令(初始化namenode,如果之前已经初始化过了,此时不需要再次重新初始化namenode):
hdfs namenode -format
如果是非HA转HA才需要这一步,在其中一个JournalNode(可以是在hadoop1)上执行:
bin/hdfs namenode -initializeSharedEdits
此命令默认是交互式的,加上参数-force转成非交互式。
在所有JournalNode创建如下目录:
mkdir -p /home/tuzq/software/hadoop-2.8.0/journal/mycluster/current
10.7. 启动主NameNode
下面进入的是hadoop1这台机器。关于启动hadoop2上的namenode在下面的博文中有介绍。
1) 进入$HADOOP_HOME/sbin目录
2) 启动主NameNode:
sbin/hadoop-daemon.sh start namenode
启动时,遇到如下所示的错误,则表示NameNode不能免密码登录自己。如果之前使用IP可以免密码登录自己,则原因一般是因为没有使用主机名登录过自己,因此解决办法是使用主机名SSH一下
10.8. 启动备NameNode
进入hadoop2,执行以下命令
1) bin/hdfs namenode –bootstrapStandby
2) sbin/hadoop-daemon.sh start namenode
如果没有执行第1步,直接启动会遇到如下错误:
No valid image files found
或者在该NameNode日志会发现如下错误:
2016-04-08 14:08:39,745 WARN org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Encountered exception loading fsimage
java.io.IOException: NameNode is not formatted.
在所有NameNode(即hadoop1和hadoop2上都执行命令)上启动主备切换进程:
sbin/hadoop-daemon.sh start zkfc
只有启动了DFSZKFailoverController进程,HDFS才能自动切换主备。
注:zkfc是zookeeper failover controller的缩写。
在各个DataNode上分别执行(即hadoop3,hadoop4,hadoop5上):
sbin/hadoop-daemon.sh start datanode
如果有发现DataNode进程并没有起来,可以试试删除logs目录下的DataNode日志,再得启看看。
1) 使用JDK提供的jps命令,查看相应的进程是否已启动
2) 检查$HADOOP_HOME/logs目录下的log和out文件,看看是否有异常信息。
启动后nn1和nn2都处于备机状态,将nn1切换为主机(下面的命令在hadoop1上执行):
bin/hdfs haadmin -transitionToActive nn1
执行jps命令(注:jps是jdk中的一个命令,不是jre中的命令),可看到DataNode进程:
$ jps 18669 DataNode 24542 Jps |
执行jps命令,可看到NameNode进程:
$ jps 18669 NameNode 24542 Jps |
执行HDFS命令,以进一步检验是否已经安装成功和配置好。关于HDFS命令的用法,直接运行命令hdfs或hdfs dfs,即可看到相关的用法说明。
hdfs dfsadmin -report
注意如果core-site.xml中的配置项fs.default.name的值为file:///,则会报:
report: FileSystem file:/// is not an HDFS file system
Usage: hdfs dfsadmin [-report] [-live] [-dead] [-decommissioning]
解决这个问题,只需要将fs.default.name的值设置为和fs.defaultFS相同的值。
10.12.2启动hdfs和yarn
进入hadoop1机器,执行命令:
[root@hadoop1sbin]# sbin/start-dfs.sh
[root@hadoop1sbin]# ./start-yarn.sh
在浏览器上访问:http://hadoop1:50070/,界面如下:
上面显示的是主的,是active状态。
再在浏览器上访问:http://hadoop2:50070/
通过上面,发现hadoop2是一种备用状态。
访问yarn(访问地址可以在yarn-site.xml中查找到),访问之后的效果如下http://hadoop1:8088/cluster:
如查看NameNode1和NameNode2分别是主还是备:
$ hdfs haadmin -getServiceState nn1 standby $ hdfs haadmin -getServiceState nn2 active |
10.12.3. hdfs dfs ls
注意:下面的命令只有在启动了yarn之后才会可用
“hdfs dfs -ls”带一个参数,如果参数以“hdfs://URI”打头表示访问HDFS,否则相当于ls。其中URI为NameNode的IP或主机名,可以包含端口号,即hdfs-site.xml中“dfs.namenode.rpc-address”指定的值。
“hdfs dfs -ls”要求默认端口为8020,如果配置成9000,则需要指定端口号,否则不用指定端口,这一点类似于浏览器访问一个URL。示例:
> hdfs dfs -ls hdfs://hadoop1:8020/ |
8020后面的斜杠/是和必须的,否则被当作文件。如果不指定端口号8020,则使用默认的8020,“hadoop1:8020”由hdfs-site.xml中“dfs.namenode.rpc-address”指定。
不难看出“hdfs dfs -ls”可以操作不同的HDFS集群,只需要指定不同的URI。
文件上传后,被存储在DataNode的data目录下(由DataNode的hdfs-site.xml中的属性“dfs.datanode.data.dir”指定),
如:$HADOOP_HOME/data/data/current/BP-472842913-192.168.106.91-1497065109036/current/finalized/subdir0/subdir0/blk_1073741825
文件名中的“blk”是block,即块的意思,默认情况下blk_1073741825即为文件的一个完整块,Hadoop未对它进额外处理。
上传文件命令,示例:
> hdfs dfs -put /etc/SuSE-release hdfs://192.168.106.91/ |
删除文件命令,示例:
> hdfs dfs -rm hdfs://192.168.106.91/SuSE-release Deleted hdfs://192.168.106.91/SuSE-release |
当有NameNode机器损坏时,必然存在新NameNode来替代。把配置修改成指向新NameNode,然后以备机形式启动新NameNode,这样新的NameNode即加入到Cluster中:
1) bin/hdfs namenode -bootstrapStandby 2) sbin/hadoop-daemon.sh start namenode |
10.12.7. HDFS只允许有一主一备两个NameNode
如果试图配置三个NameNode,如:
dfs.ha.namenodes.test nm1,nm2,nm3
The prefix for a given nameservice, contains a comma-separated list of namenodes for a given nameservice (eg EXAMPLENAMESERVICE).
|
则运行“hdfs namenode -bootstrapStandby”时会报如下错误,表示在同一NameSpace内不能超过2个NameNode:
16/04/11 09:51:57 ERROR namenode.NameNode: Failed to start namenode. java.io.IOException: java.lang.IllegalArgumentException: Expected exactly 2 NameNodes in namespace 'test'. Instead, got only 3 (NN ids were 'nm1','nm2','nm3' at org.apache.hadoop.hdfs.server.namenode.ha.BootstrapStandby.run(BootstrapStandby.java:425) at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1454) at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1554) Caused by: java.lang.IllegalArgumentException: Expected exactly 2 NameNodes in namespace 'test'. Instead, got only 3 (NN ids were 'nm1','nm2','nm3' at com.google.common.base.Preconditions.checkArgument(Preconditions.java:115) |
10.12.8. 存储均衡start-balancer.sh
示例:start-balancer.sh –t 10%
10%表示机器与机器之间磁盘使用率偏差小于10%时认为均衡,否则做均衡搬动。“start-balancer.sh”调用“hdfs start balancer”来做均衡,可以调用stop-balancer.sh停止均衡。
均衡过程非常慢,但是均衡过程中,仍能够正常访问HDFS,包括往HDFS上传文件。
[VM2016@hadoop-030 /data4/hadoop/sbin]$ hdfs balancer # 可以改为调用start-balancer.sh 16/04/08 14:26:55 INFO balancer.Balancer: namenodes = [hdfs://test] // test为HDFS的cluster名 16/04/08 14:26:55 INFO balancer.Balancer: parameters = Balancer.Parameters[BalancingPolicy.Node, threshold=10.0, max idle iteration = 5, number of nodes to be excluded = 0, number of nodes to be included = 0] Time Stamp Iteration# Bytes Already Moved Bytes Left To Move Bytes Being Moved 16/04/08 14:26:56 INFO net.NetworkTopology: Adding a new node: /default-rack/192.168.1.231:50010 16/04/08 14:26:56 INFO net.NetworkTopology: Adding a new node: /default-rack/192.168.1.229:50010 16/04/08 14:26:56 INFO net.NetworkTopology: Adding a new node: /default-rack/192.168.1.213:50010 16/04/08 14:26:56 INFO net.NetworkTopology: Adding a new node: /default-rack/192.168.1.208:50010 16/04/08 14:26:56 INFO net.NetworkTopology: Adding a new node: /default-rack/192.168.1.232:50010 16/04/08 14:26:56 INFO net.NetworkTopology: Adding a new node: /default-rack/192.168.1.207:50010 16/04/08 14:26:56 INFO balancer.Balancer: 5 over-utilized: [192.168.1.231:50010:DISK, 192.168.1.229:50010:DISK, 192.168.1.213:50010:DISK, 192.168.1.208:50010:DISK, 192.168.1.232:50010:DISK] 16/04/08 14:26:56 INFO balancer.Balancer: 1 underutilized(未充分利用的): [192.168.1.207:50010:DISK] # 数据将移向该节点 16/04/08 14:26:56 INFO balancer.Balancer: Need to move 816.01 GB to make the cluster balanced. # 需要移动816.01G数据达到平衡 16/04/08 14:26:56 INFO balancer.Balancer: Decided to move 10 GB bytes from 192.168.1.231:50010:DISK to 192.168.1.207:50010:DISK # 从192.168.1.231移动10G数据到192.168.1.207 16/04/08 14:26:56 INFO balancer.Balancer: Will move 10 GB in this iteration
16/04/08 14:32:58 INFO balancer.Dispatcher: Successfully moved blk_1073749366_8542 with size=77829046 from 192.168.1.231:50010:DISK to 192.168.1.207:50010:DISK through 192.168.1.213:50010 16/04/08 14:32:59 INFO balancer.Dispatcher: Successfully moved blk_1073749386_8562 with size=77829046 from 192.168.1.231:50010:DISK to 192.168.1.207:50010:DISK through 192.168.1.231:50010 16/04/08 14:33:34 INFO balancer.Dispatcher: Successfully moved blk_1073749378_8554 with size=77829046 from 192.168.1.231:50010:DISK to 192.168.1.207:50010:DISK through 192.168.1.231:50010 16/04/08 14:34:38 INFO balancer.Dispatcher: Successfully moved blk_1073749371_8547 with size=134217728 from 192.168.1.231:50010:DISK to 192.168.1.207:50010:DISK through 192.168.1.213:50010 16/04/08 14:34:54 INFO balancer.Dispatcher: Successfully moved blk_1073749395_8571 with size=134217728 from 192.168.1.231:50010:DISK to 192.168.1.207:50010:DISK through 192.168.1.231:50010 Apr 8, 2016 2:35:01 PM 0 478.67 MB 816.01 GB 10 GB 16/04/08 14:35:10 INFO net.NetworkTopology: Adding a new node: /default-rack/192.168.1.213:50010 16/04/08 14:35:10 INFO net.NetworkTopology: Adding a new node: /default-rack/192.168.1.229:50010 16/04/08 14:35:10 INFO net.NetworkTopology: Adding a new node: /default-rack/192.168.1.232:50010 16/04/08 14:35:10 INFO net.NetworkTopology: Adding a new node: /default-rack/192.168.1.231:50010 16/04/08 14:35:10 INFO net.NetworkTopology: Adding a new node: /default-rack/192.168.1.208:50010 16/04/08 14:35:10 INFO net.NetworkTopology: Adding a new node: /default-rack/192.168.1.207:50010 16/04/08 14:35:10 INFO balancer.Balancer: 5 over-utilized: [192.168.1.213:50010:DISK, 192.168.1.229:50010:DISK, 192.168.1.232:50010:DISK, 192.168.1.231:50010:DISK, 192.168.1.208:50010:DISK] 16/04/08 14:35:10 INFO balancer.Balancer: 1 underutilized(未充分利用的): [192.168.1.207:50010:DISK] 16/04/08 14:35:10 INFO balancer.Balancer: Need to move 815.45 GB to make the cluster balanced. 16/04/08 14:35:10 INFO balancer.Balancer: Decided to move 10 GB bytes from 192.168.1.213:50010:DISK to 192.168.1.207:50010:DISK 16/04/08 14:35:10 INFO balancer.Balancer: Will move 10 GB in this iteration
16/04/08 14:41:18 INFO balancer.Dispatcher: Successfully moved blk_1073760371_19547 with size=77829046 from 192.168.1.213:50010:DISK to 192.168.1.207:50010:DISK through 192.168.1.213:50010 16/04/08 14:41:19 INFO balancer.Dispatcher: Successfully moved blk_1073760385_19561 with size=77829046 from 192.168.1.213:50010:DISK to 192.168.1.207:50010:DISK through 192.168.1.213:50010 16/04/08 14:41:22 INFO balancer.Dispatcher: Successfully moved blk_1073760393_19569 with size=77829046 from 192.168.1.213:50010:DISK to 192.168.1.207:50010:DISK through 192.168.1.213:50010 16/04/08 14:41:23 INFO balancer.Dispatcher: Successfully moved blk_1073760363_19539 with size=77829046 from 192.168.1.213:50010:DISK to 192.168.1.207:50010:DISK through 192.168.1.213:50010 |
找一台已有JournalNode节点,修改它的hdfs-site.xml,将新增的Journal包含进来,如在
qjournal://hadoop1:8485;hadoop2:8485;hadoop3:8485/mycluster
的基础上新增hadoop6和hadoop7两个JournalNode:
qjournal://hadoop1:8485;hadoop2:8485;hadoop3:8485;hadoop6:8485;hadoop7:8485/mycluster
然后将安装目录和数据目录(hdfs-site.xml中的dfs.journalnode.edits.dir指定的目录)都复制到新的节点。
如果不复制JournalNode的数据目录,则新节点上的JournalNode会报错“Journal Storage Directory /data/journal/test not formatted”,将来的版本可能会实现自动同步。
接下来,就可以在新节点上启动好JournalNode(不需要做什么初始化),并重启下NameNode。注意观察JournalNode日志,查看是否启动成功,当日志显示为以下这样的INFO级别日志则表示启动成功:
2016-04-26 10:31:11,160 INFO org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits file /data/journal/test/current/edits_inprogress_0000000000000194269 -> /data/journal/test/current/edits_0000000000000194269-0000000000000194270
11. 启动YARN
1) 进入$HADOOP_HOME/sbin目录
2) 在主备两台都执行:start-yarn.sh,即开始启动YARN
若启动成功,则在Master节点执行jps,可以看到ResourceManager:
> jps 24689 NameNode 30156 Jps 28861 ResourceManager |
在Slaves节点执行jps,可以看到NodeManager:
$ jps 14019 NodeManager 23257 DataNode 15115 Jps |
如果只需要单独启动指定节点上的ResourceManager,这样:
sbin/yarn-daemon.sh start resourcemanager
对于NodeManager,则是这样:
sbin/yarn-daemon.sh start nodemanager
列举YARN集群中的所有NodeManager,如(注意参数间的空格,直接执行yarn可以看到使用帮助):
[root@hadoop1sbin]# yarn node –list
查看指定NodeManager的状态(通过上面查出来的结果进行查询),如:
[root@hadoop1 hadoop]# yarn node -status hadoop5:59894 Node Report : Node-Id : hadoop5:59894 Rack : /default-rack Node-State : RUNNING Node-Http-Address : hadoop5:8042 Last-Health-Update : 星期六 10/六月/17 12:30:38:20CST Health-Report : Containers : 0 Memory-Used : 0MB Memory-Capacity : 8192MB CPU-Used : 0 vcores CPU-Capacity : 8 vcores Node-Labels : Resource Utilization by Node : PMem:733 MB, VMem:733 MB, VCores:0.0 Resource Utilization by Containers : PMem:0 MB, VMem:0 MB, VCores:0.0
[root@hadoop1 hadoop]# |
11.2.3. yarn rmadmin -getServiceState rm1
查看rm1的主备状态,即查看它是主(active)还是备(standby)。
11.2.4. yarn rmadmin -transitionToStandby rm1
将rm1从主切为备。
更多的yarn命令可以参考:
https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YarnCommands.html。
在安装目录的share/hadoop/mapreduce子目录下,有现存的示例程序:
hadoop@VM-40-171-sles10-64:~/hadoop> ls share/hadoop/mapreduce hadoop-mapreduce-client-app-2.7.2.jar hadoop-mapreduce-client-jobclient-2.7.2-tests.jar hadoop-mapreduce-client-common-2.7.2.jar hadoop-mapreduce-client-shuffle-2.7.2.jar hadoop-mapreduce-client-core-2.7.2.jar hadoop-mapreduce-examples-2.7.2.jar hadoop-mapreduce-client-hs-2.7.2.jar lib hadoop-mapreduce-client-hs-plugins-2.7.2.jar lib-examples hadoop-mapreduce-client-jobclient-2.7.2.jar sources |
跑一个示例程序试试:
hdfs dfs -put /etc/hosts hdfs://test/in/ hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.2.jar wordcount hdfs://test/in/ hdfs://test/out/ |
运行过程中,使用java的jps命令,可以看到yarn启动了名为YarnChild的进程。
wordcount运行完成后,结果会保存在out目录下,保存结果的文件名类似于“part-r-00000”。另外,跑这个示例程序有两个需求注意的点:
1) in目录下要有文本文件,或in即为被统计的文本文件,可以为HDFS上的文件或目录,也可以为本地文件或目录
2) out目录不能存在,程序会自动去创建它,如果已经存在则会报错。
包hadoop-mapreduce-examples-2.7.2.jar中含有多个示例程序,不带参数运行,即可看到用法:
> hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.2.jar wordcount Usage: wordcount
> hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.2.jar An example program must be given as the first argument. Valid program names are: aggregatewordcount: An Aggregate based map/reduce program that counts the words in the input files. aggregatewordhist: An Aggregate based map/reduce program that computes the histogram of the words in the input files. bbp: A map/reduce program that uses Bailey-Borwein-Plouffe to compute exact digits of Pi. dbcount: An example job that count the pageview counts from a database. distbbp: A map/reduce program that uses a BBP-type formula to compute exact bits of Pi. grep: A map/reduce program that counts the matches of a regex in the input. join: A job that effects a join over sorted, equally partitioned datasets multifilewc: A job that counts words from several files. pentomino: A map/reduce tile laying program to find solutions to pentomino problems. pi: A map/reduce program that estimates Pi using a quasi-Monte Carlo method. randomtextwriter: A map/reduce program that writes 10GB of random textual data per node. randomwriter: A map/reduce program that writes 10GB of random data per node. secondarysort: An example defining a secondary sort to the reduce. sort: A map/reduce program that sorts the data written by the random writer. sudoku: A sudoku solver. teragen: Generate data for the terasort terasort: Run the terasort teravalidate: Checking results of terasort wordcount: A map/reduce program that counts the words in the input files. wordmean: A map/reduce program that counts the average length of the words in the input files. wordmedian: A map/reduce program that counts the median length of the words in the input files. wordstandarddeviation: A map/reduce program that counts the standard deviation of the length of the words in the input files. |
修改日志级别为DEBBUG,并打屏:
export HADOOP_ROOT_LOGGER=DEBUG,console13. HDFS权限配置
dfs.permissions.enabled = true dfs.permissions.superusergroup = supergroup dfs.cluster.administrators = ACL-for-admins dfs.namenode.acls.enabled = true dfs.web.ugi = webuser,webgroup |
fs.permissions.umask-mode = 022 hadoop.security.authentication = simple 安全验证规则,可为simple或kerberos |
// g++ -g -o x x.cpp -L$JAVA_HOME/lib/amd64/jli -ljli -L$JAVA_HOME/jre/lib/amd64/server -ljvm -I$HADOOP_HOME/include $HADOOP_HOME/lib/native/libhdfs.a -lpthread -ldl #include "hdfs.h" #include #include #include
int main(int argc, char **argv) { #if 0 hdfsFS fs = hdfsConnect("default", 0); // HA方式 const char* writePath = "hdfs://mycluster/tmp/testfile.txt"; hdfsFile writeFile = hdfsOpenFile(fs, writePath, O_WRONLY |O_CREAT, 0, 0, 0); if(!writeFile) { fprintf(stderr, "Failed to open %s for writing!\n", writePath); exit(-1); } const char* buffer = "Hello, World!\n"; tSize num_written_bytes = hdfsWrite(fs, writeFile, (void*)buffer, strlen(buffer)+1); if (hdfsFlush(fs, writeFile)) { fprintf(stderr, "Failed to 'flush' %s\n", writePath); exit(-1); } hdfsCloseFile(fs, writeFile); #else struct hdfsBuilder* bld = hdfsNewBuilder(); hdfsBuilderSetNameNode(bld, "default"); // HA方式 hdfsFS fs = hdfsBuilderConnect(bld); if (NULL == fs) { fprintf(stderr, "Failed to connect hdfs\n"); exit(-1); } int num_entries = 0; hdfsFileInfo* entries; if (argc < 2) entries = hdfsListDirectory(fs, "/", &num_entries); else entries = hdfsListDirectory(fs, argv[1], &num_entries); fprintf(stdout, "num_entries: %d\n", num_entries); for (int i=0; i<num_entries; ++i) </num_entries; ++i)<> { fprintf(stdout, "%s\n", entries[i].mName); } hdfsFreeFileInfo(entries, num_entries); hdfsDisconnect(fs); //hdfsFreeBuilder(bld); #endif return 0; } |
运行之前需要设置好CLASSPATH,如果设置不当,可能会遇到不少困难,比如期望操作HDFS上的文件和目录,却变成了本地的文件和目录,如者诸于“java.net.UnknownHostException”类的错误等。
为避免出现错误,强烈建议使用命令“hadoop classpath --glob”取得正确的CLASSPATH值。
另外还需要设置好libjli.so和libjvm.so两个库的LD_LIBRARY_PATH,如:
export LD_LIBRARY_PATH=$JAVA_HOME/lib/amd64/jli:$JAVA_HOME/jre/lib/amd64/server:$LD_LIBRARY_PATH |
15.1. 执行“hdfs dfs -ls”时报ConnectException
原因可能是指定的端口号9000不对,该端口号由hdfs-site.xml中的属性“dfs.namenode.rpc-address”指定,即为NameNode的RPC服务端口号。
文件上传后,被存储在DataNode的data(由DataNode的hdfs-site.xml中的属性“dfs.datanode.data.dir”指定)目录下,如:
$HADOOP_HOME/data/current/BP-139798373-192.168.106.91-1397735615751/current/finalized/blk_1073741825
文件名中的“blk”是block,即块的意思,默认情况下blk_1073741825即为文件的一个完整块,Hadoop未对它进额外处理。
hdfs dfs -ls hdfs://192.168.106.91:9000 14/04/17 12:04:02 WARN conf.Configuration: mapred-site.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring. 14/04/17 12:04:02 WARN conf.Configuration: mapred-site.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring. 14/04/17 12:04:02 WARN conf.Configuration: mapred-site.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring. 14/04/17 12:04:02 WARN conf.Configuration: mapred-site.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring. 14/04/17 12:04:02 WARN conf.Configuration: mapred-site.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring. 14/04/17 12:04:02 WARN conf.Configuration: mapred-site.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring. Java HotSpot(TM) 64-Bit Server VM warning: You have loaded library /home/tuzq/software/hadoop-2.8.0/lib/native/libhadoop.so.1.0.0 which might have disabled stack guard. The VM will try to fix the stack guard now. It's highly recommended that you fix the library with 'execstack -c ', or link it with '-z noexecstack'. 14/04/17 12:04:02 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 14/04/17 12:04:03 WARN conf.Configuration: mapred-site.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring. 14/04/17 12:04:03 WARN conf.Configuration: mapred-site.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring. ls: Call From VM-40-171-sles10-64/192.168.106.91 to VM-40-171-sles10-64:9000 failed on connection exception: java.net.ConnectException: 拒绝连接; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused |
15.2. Initialization failed for Block pool
可能是因为对NameNode做format之前,没有清空DataNode的data目录。
“Incompatible clusterIDs”的错误原因是在执行“hdfs namenode -format”之前,没有清空DataNode节点的data目录。
网上一些文章和帖子说是tmp目录,它本身也是没问题的,但Hadoop 2.7.2是data目录,实际上这个信息已经由日志的“/home/tuzq/software/hadoop-2.8.0/data”指出,所以不能死死的参照网上的解决办法,遇到问题时多仔细观察。
从上述描述不难看出,解决办法就是清空所有DataNode的data目录,但注意不要将data目录本身给删除了。 data目录由core-site.xml文件中的属性“dfs.datanode.data.dir”指定。
2014-04-17 19:30:33,075 INFO org.apache.hadoop.hdfs.server.common.Storage: Lock on /home/tuzq/software/hadoop-2.8.0/data/in_use.lock acquired by nodename 28326@localhost 2014-04-17 19:30:33,078 FATAL org.apache.hadoop.hdfs.server.datanode.DataNode: Initialization failed for block pool Block pool (Datanode Uuid unassigned) service to /192.168.106.91:9001 java.io.IOException: Incompatible clusterIDs in /home/tuzq/software/hadoop-2.8.0/data: namenode clusterID = CID-50401d89-a33e-47bf-9d14-914d8f1c4862; datanode clusterID = CID-153d6fcb-d037-4156-b63a-10d6be224091 at org.apache.hadoop.hdfs.server.datanode.DataStorage.doTransition(DataStorage.java:472) at org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:225) at org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:249) at org.apache.hadoop.hdfs.server.datanode.DataNode.initStorage(DataNode.java:929) at org.apache.hadoop.hdfs.server.datanode.DataNode.initBlockPool(DataNode.java:900) at org.apache.hadoop.hdfs.server.datanode.BPOfferService.verifyAndSetNamespaceInfo(BPOfferService.java:274) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:220) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:815) at java.lang.Thread.run(Thread.java:744) 2014-04-17 19:30:33,081 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Ending block pool service for: Block pool (Datanode Uuid unassigned) service to /192.168.106.91:9001 2014-04-17 19:30:33,184 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Block pool ID needed, but service not yet registered with NN java.lang.Exception: trace at org.apache.hadoop.hdfs.server.datanode.BPOfferService.getBlockPoolId(BPOfferService.java:143) at org.apache.hadoop.hdfs.server.datanode.BlockPoolManager.remove(BlockPoolManager.java:91) at org.apache.hadoop.hdfs.server.datanode.DataNode.shutdownBlockPool(DataNode.java:859) at org.apache.hadoop.hdfs.server.datanode.BPOfferService.shutdownActor(BPOfferService.java:350) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.cleanUp(BPServiceActor.java:619) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:837) at java.lang.Thread.run(Thread.java:744) 2014-04-17 19:30:33,184 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Removed Block pool (Datanode Uuid unassigned) 2014-04-17 19:30:33,184 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Block pool ID needed, but service not yet registered with NN java.lang.Exception: trace at org.apache.hadoop.hdfs.server.datanode.BPOfferService.getBlockPoolId(BPOfferService.java:143) at org.apache.hadoop.hdfs.server.datanode.DataNode.shutdownBlockPool(DataNode.java:861) at org.apache.hadoop.hdfs.server.datanode.BPOfferService.shutdownActor(BPOfferService.java:350) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.cleanUp(BPServiceActor.java:619) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:837) at java.lang.Thread.run(Thread.java:744) 2014-04-17 19:30:35,185 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Exiting Datanode 2014-04-17 19:30:35,187 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 0 2014-04-17 19:30:35,189 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: SHUTDOWN_MSG: /************************************************************ SHUTDOWN_MSG: Shutting down DataNode at localhost/127.0.0.1 ************************************************************/ |
15.4. Inconsistent checkpoint fields
SecondaryNameNode中的“Inconsistent checkpoint fields”错误原因,可能是因为没有设置好SecondaryNameNode上core-site.xml文件中的“hadoop.tmp.dir”。
2014-04-17 11:42:18,189 INFO org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: Log Size Trigger :1000000 txns 2014-04-17 11:43:18,365 ERROR org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: Exception in doCheckpoint java.io.IOException: Inconsistent checkpoint fields. LV = -56 namespaceID = 1384221685 cTime = 0 ; clusterId = CID-319b9698-c88d-4fe2-8cb2-c4f440f690d4 ; blockpoolId = BP-1627258458-192.168.106.91-1397735061985. Expecting respectively: -56; 476845826; 0; CID-50401d89-a33e-47bf-9d14-914d8f1c4862; BP-2131387753-192.168.106.91-1397730036484. at org.apache.hadoop.hdfs.server.namenode.CheckpointSignature.validateStorageInfo(CheckpointSignature.java:135) at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doCheckpoint(SecondaryNameNode.java:518) at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doWork(SecondaryNameNode.java:383) at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$1.run(SecondaryNameNode.java:349) at org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:415) at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.run(SecondaryNameNode.java:345) at java.lang.Thread.run(Thread.java:744)
另外,也请配置好SecondaryNameNode上hdfs-site.xml中的“dfs.datanode.data.dir”为合适的值: hadoop.tmp.dir /home/tuzq/software/current/tmp A base for other temporary directories. |
15.5. fs.defaultFS is file:///
在core-site.xml中,当只填写了fs.defaultFS,而fs.default.name为默认的file:///时,会报此错误。解决方法是设置成相同的值。
15.6. a shared edits dir must not be specified if HA is not enabled
该错误可能是因为hdfs-site.xml中没有配置dfs.nameservices或dfs.ha.namenodes.mycluster。
15.7. /tmp/dfs/name is in an inconsistent state: storage directory does not exist or is not accessible.
只需按日志中提示的,创建好相应的目录。
15.8. The auxService:mapreduce_shuffle does not exist
问题原因是没有配置yarn-site.xml中的“yarn.nodemanager.aux-services”,将它的值配置为mapreduce_shuffle,然后重启yarn问题即解决。记住所有yarn节点都需要修改,包括ResourceManager和NodeManager,如果NodeManager上的没有修改,仍然会报这个错误。
15.9. org.apache.hadoop.ipc.Client: Retrying connect to server
该问题,有可能是因为NodeManager中的yarn-site.xml和ResourceManager上的不一致,比如NodeManager没有配置yarn.resourcemanager.ha.rm-ids。
15.10. mapreduce.Job: Running job: job_1445931397013_0001
Hadoop提交mapreduce任务时,卡在mapreduce.Job: Running job: job_1445931397013_0001处。
问题原因可能是因为yarn的NodeManager没起来,可以用jdk的jps确认下。
该问题也有可能是因为NodeManager中的yarn-site.xml和ResourceManager上的不一致,比如NodeManager没有配置yarn.resourcemanager.ha.rm-ids。
15.11. Could not format one or more JournalNodes
执行“./hdfs namenode -format”时报“Could not format one or more JournalNodes”。
可能是hdfs-site.xml中的dfs.namenode.shared.edits.dir配置错误,比如重复了,如:
qjournal://hadoop-168-254:8485;hadoop-168-254:8485;hadoop-168-253:8485;hadoop-168-252:8485;hadoop-168-251:8485/mycluster
修复后,重启JournalNode,问题可能就解决了。
15.12. org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Already in standby state
遇到这个错误,可能是yarn-site.xml中的yarn.resourcemanager.webapp.address配置错误,比如配置成了两个yarn.resourcemanager.webapp.address.rm1,实际应当是yarn.resourcemanager.webapp.address.rm1和yarn.resourcemanager.webapp.address.rm2。
15.13. No valid image files found
如果是备NameNode,执行下“hdfs namenode -bootstrapStandby”再启动。
2015-12-01 15:24:39,535 ERROR org.apache.hadoop.hdfs.server.namenode.NameNode: Failed to start namenode.
java.io.FileNotFoundException: No valid image files found
at org.apache.hadoop.hdfs.server.namenode.FSImageTransactionalStorageInspector.getLatestImages(FSImageTransactionalStorageInspector.java:165)
at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:623)
at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:294)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:975)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:681)
at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:584)
at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:644)
at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:811)
at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:795)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1488)
at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1554)
2015-12-01 15:24:39,536 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 1
2015-12-01 15:24:39,539 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
15.14. xceivercount 4097 exceeds the limit of concurrent xcievers 4096
此错误的原因是hdfs-site.xml中的配置项“dfs.datanode.max.xcievers”值4096过小,需要改大一点。该错误会导致hbase报“notservingregionexception”。
16/04/06 14:30:34 ERROR namenode.NameNode: Failed to start namenode.
15.15. java.lang.IllegalArgumentException: Unable to construct journal, qjournal://hadoop-030:8485;hadoop-031:8454;hadoop-032
执行“hdfs namenode -format”遇到上述错误时,是因为hdfs-site.xml中的配置dfs.namenode.shared.edits.dir配置错误,其中的hadoop-032省了“:8454”部分。
15.16. Bad URI 'qjournal://hadoop-030:8485;hadoop-031:8454;hadoop-032:8454': must identify journal in path component
是因为配置hdfs-site.xml中的“dfs.namenode.shared.edits.dir”时,路径少带了cluster名。
15.17. 16/04/06 14:48:19 INFO ipc.Client: Retrying connect to server: hadoop-032/10.143.136.211:8454. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
检查hdfs-site.xml中的“dfs.namenode.shared.edits.dir”值,JournalNode默认端口是8485,不是8454,确认是否有写错。JournalNode端口由hdfs-site.xml中的配置项dfs.journalnode.rpc-address决定。
15.18. Exception in thread "main" org.apache.hadoop.HadoopIllegalArgumentException: Could not get the namenode ID of this node. You may run zkfc on the node other than namenode.
执行“hdfs zkfc -formatZK”遇到上面这个错误,是因为还没有执行“hdfs namenode -format”。NameNode ID是在“hdfs namenode -format”时生成的。
15.19. 2016-04-06 17:08:07,690 INFO org.apache.hadoop.hdfs.server.common.Storage: Storage directory [DISK]file:/data3/datanode/data/ has already been used.
以非root用户启动DataNode,但启动不了,在它的日志文件中发现如下错误信息:
2016-04-06 17:08:07,707 INFO org.apache.hadoop.hdfs.server.common.Storage: Analyzing storage directories for bpid BP-418073539-10.143.136.207-1459927327462
2016-04-06 17:08:07,707 WARN org.apache.hadoop.hdfs.server.common.Storage: Failed to analyze storage directories for block pool BP-418073539-10.143.136.207-1459927327462
java.io.IOException: BlockPoolSliceStorage.recoverTransitionRead: attempt to load an used block storage: /data3/datanode/data/current/BP-418073539-10.143.136.207-1459927327462
继续寻找,会发现还存在如何错误提示:
Invalid dfs.datanode.data.dir /data3/datanode/data:
EPERM: Operation not permitted
使用命令“ls -l”检查目录/data3/datanode/data的权限设置,发现owner为root,原因是因为之前使用root启动过DataNode,将owner改过来即可解决此问题。
15.20. 2016-04-06 18:00:26,939 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Problem connecting to server: hadoop-031/10.143.136.208:8020
DataNode的日志文件不停地记录如下日志,是因为DataNode将作为主NameNode,但实际上10.143.136.208并没有启动,主NameNode不是它。这个并不表示DataNode没有起来,而是因为DataNode会同时和主NameNode和备NameNode建立心跳,当备NameNode没有起来时,有这些日志是正常现象。
2016-04-06 18:00:32,940 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-031/10.143.136.208:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2016-04-06 17:55:44,555 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Namenode Block pool BP-418073539-10.143.136.207-1459927327462 (Datanode Uuid 2d115d45-fd48-4e86-97b1-e74a1f87e1ca) service to hadoop-030/10.143.136.207:8020 trying to claim ACTIVE state with txid=1
“trying to claim ACTIVE state”出自于hadoop/hdfs/server/datanode/BPOfferService.java中的updateActorStatesFromHeartbeat()。
2016-04-06 17:55:49,893 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-031/10.143.136.208:8020. Already tried 5 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
“Retrying connect to server”出自于hadoop/ipc/Client.java中的handleConnectionTimeout()和handleConnectionFailure()。
15.21. ERROR cluster.YarnClientSchedulerBackend: Yarn application has already exited with state FINISHED!
如果遇到这个错误,请检查NodeManager日志,如果发现有如下所示信息:
WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Container [pid=26665,containerID=container_1461657380500_0020_02_000001] is running beyond virtual memory limits. Current usage: 345.0 MB of 1 GB physical memory used; 2.2 GB of 2.1 GB virtual memory used. Killing container.
则表示需要增大yarn-site.xmk的配置项yarn.nodemanager.vmem-pmem-ratio的值,该配置项默认值为2.1。
16/10/13 10:23:19 ERROR client.TransportClient: Failed to send RPC 7614640087981520382 to /10.143.136.231:34800: java.nio.channels.ClosedChannelException
java.nio.channels.ClosedChannelException
16/10/13 10:23:19 ERROR cluster.YarnSchedulerBackend$YarnSchedulerEndpoint: Sending RequestExecutors(0,0,Map()) to AM was unsuccessful
java.io.IOException: Failed to send RPC 7614640087981520382 to /10.143.136.231:34800: java.nio.channels.ClosedChannelException
at org.apache.spark.network.client.TransportClient$3.operationComplete(TransportClient.java:249)
at org.apache.spark.network.client.TransportClient$3.operationComplete(TransportClient.java:233)
at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:680)
at io.netty.util.concurrent.DefaultPromise$LateListeners.run(DefaultPromise.java:845)
at io.netty.util.concurrent.DefaultPromise$LateListenerNotifier.run(DefaultPromise.java:873)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:357)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357)
at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
at java.lang.Thread.run(Thread.java:745)
《HBase-0.98.0分布式安装指南》
《Hive 0.12.0安装指南》
《ZooKeeper-3.4.6分布式安装指南》
《Hadoop 2.3.0源码反向工程》
《在Linux上编译Hadoop-2.7.2》
《Accumulo-1.5.1安装指南》
《Drill 1.0.0安装指南》
《Shark 0.9.1安装指南》