E-MapReduce集群启停HDFS/YARN服务-阿里云开发者社区

开发者社区> 阿里云EMR> 正文
登录阅读全文

E-MapReduce集群启停HDFS/YARN服务

简介: 该文章意在帮助大家,在E-MapReduce环境中停止启动yarn,hdfs服务。

启停HDFS服务

HDFS服务启停,下面脚本需要在master节点运行,切需要在hdfs账号下。su hdfs

启动HDFS

执行下面脚本

#!/bin/bash

worker_cnt=`cat /etc/hosts | grep emr-worker | grep cluster | wc -l`
master_cnt=1
ha_flag=`grep -r high_availability_enable=true /usr/local/emr/emr-bin/script/`
nn_file=/usr/local/emr/emr-bin/script/hdfs/pre_start.sh
dn_file=/usr/local/emr/emr-bin/script/hdfs/start_up.sh
if [[ ! -z $ha_flag ]];then
  master_cnt=2
  nn_file=/usr/local/emr/emr-bin/script/ha_hdfs/pre_start.sh
  dn_file=/usr/local/emr/emr-bin/script/ha_hdfs/start_up.sh
fi

nn_cmd="export app_yarn_home=/usr/lib/hadoop-current;"\ `cat $nn_file | grep -v 'echo' | grep 'start namenode' | head -n 1 | awk -F '"' '{print $2;}'`
dn_cmd="export app_yarn_home=/usr/lib/hadoop-current;"\ `cat $dn_file | grep -v 'echo' | grep 'start datanode' | head -n 1 | awk -F '"' '{print $2;}'`

#start namenode
for ((i=1; i<=$master_cnt; i++));
do
    echo master--$i
    echo "$nn_cmd"
    if [ $i -eq 2 ];then
        ssh emr-header-$i "/usr/lib/hadoop-current/bin/hdfs namenode -bootstrapStandby <<< N"
    fi
    ssh emr-header-$i "$nn_cmd"
done

# start datanode
for ((i=1; i<=$worker_cnt; i++));
do
    echo "$dn_cmd"
    ssh emr-worker-$i "$dn_cmd"
done

停止HDFS服务

执行下面脚本

#!/bin/bash

worker_cnt=`cat /etc/hosts | grep emr-worker | grep cluster | wc -l`
master_cnt=1
ha_flag=`grep -r high_availability_enable=true /usr/local/emr/emr-bin/script/`
if [[ ! -z $ha_flag ]];then
  master_cnt=2
fi

nn_cmd='/usr/lib/hadoop-current/sbin/hadoop-daemon.sh stop namenode'
dn_cmd='/usr/lib/hadoop-current/sbin/hadoop-daemon.sh stop datanode'

#stop namenode
for ((i=1; i<=$master_cnt; i++));
do
    ssh emr-header-$i "$nn_cmd"
done

# stop datanode
for ((i=1; i<=$worker_cnt; i++));
do
    ssh emr-worker-$i "$dn_cmd"
done

启停YARN服务

启停YARN服务,下面的脚本需要在master节点运行,且需要在hadoop账号下,su hadoop。

启动YARN服务

执行下面脚本

#!/bin/bash

worker_cnt=`cat /etc/hosts | grep emr-worker | grep cluster | wc -l`
master_cnt=1
ha_flag=`grep -r high_availability_enable=true /usr/local/emr/emr-bin/script/`
yarn_file=/usr/local/emr/emr-bin/script/yarn/start_up.sh
if [[ ! -z $ha_flag ]];then
  master_cnt=2
  yarn_file=/usr/local/emr/emr-bin/script/ha_yarn/start_up.sh
fi

rm_cmd="export app_yarn_home=/usr/lib/hadoop-current;"\ `cat $yarn_file | grep -v 'echo' | grep  'start resourcemanager' | head -n 1 | awk -F '"' '{print $2;}'`
nm_cmd="export app_yarn_home=/usr/lib/hadoop-current;"\ `cat $yarn_file | grep -v 'echo' | grep  'start nodemanager' | head -n 1 | awk -F '"' '{print $2;}'`

#start resourcemanager
for ((i=1; i<=$master_cnt; i++));
do
    ssh emr-header-$i "$rm_cmd"
done

# start nodemanager
for ((i=1; i<=$worker_cnt; i++));
do
    ssh emr-worker-$i "$nm_cmd"
done

停止YARN服务

执行下面脚本

#!/bin/bash

worker_cnt=`cat /etc/hosts | grep emr-worker | grep cluster | wc -l`
master_cnt=1
ha_flag=`grep -r high_availability_enable=true /usr/local/emr/emr-bin/script/`
if [[ ! -z $ha_flag ]];then
  master_cnt=2
fi

nn_cmd='/usr/lib/hadoop-current/sbin/hadoop-daemon.sh stop namenode'
dn_cmd='/usr/lib/hadoop-current/sbin/hadoop-daemon.sh stop datanode'

#stop resourcemanager
for ((i=1; i<=$master_cnt; i++));
do
    ssh emr-header-$i "$nn_cmd"
done

# stop nodemanager
for ((i=1; i<=$worker_cnt; i++));
do
    ssh emr-worker-$i "$dn_cmd"
done

版权声明:本文内容由阿里云实名注册用户自发贡献,版权归原作者所有,阿里云开发者社区不拥有其著作权,亦不承担相应法律责任。具体规则请查看《阿里云开发者社区用户服务协议》和《阿里云开发者社区知识产权保护指引》。如果您发现本社区中有涉嫌抄袭的内容,填写侵权投诉表单进行举报,一经查实,本社区将立刻删除涉嫌侵权内容。

分享:
+ 订阅

阿里巴巴开源大数据技术团队成立阿里云EMR技术圈, 每周推送前沿技术文章,直播分享经典案例、在线答疑,营造纯粹的开源大数据氛围,欢迎加入!加入钉钉群聊阿里云E-MapReduce交流2群,点击进入查看详情 https://qr.dingtalk.com/action/joingroup?code=v1,k1,cNBcqHn4TvG0iHpN3cSc1B86D1831SGMdvGu7PW+sm4=&_dt_no_comment=1&origin=11

官方博客
最新文章
相关文章
官网链接