作者：陈云亮，系深圳某数据治理厂商，有10余年IT从业经验，在数据仓库，传统数据库和大数据领域有丰富的实战经验。
在阿里云某私有云的项目中，遇到了需要在E-MapReduce集群节点外使用客户端的需求，而目前在集群管理中只提供了通过Gateway新建一台ECS来满足。但是，由于客户方ECS资源已经申请，所以无法使用Gateway，基于此，通过对E-MapReduce的官方文档和社群资源学习，深入集群了解，特开发一套部署客户端程序。
（PS：在华为FIHD大数据平台，星环和CDH等大数据平台的管理界面都提供了下载客户端安装包并能实现一键安装部署。希望下个版本能有这样的功能。）
E-MapReduce客户端安装和卸载工具，主要用于集群外的ECS（已经存在）中需要使用hadoop，hive等客户端命令工具。
目前云平台提供的gateway只能重新创建ECS进行部署，而不支持在已经创建的Ecs中部署。
主要涉及的程序：
[root@iZ1cr01rx0b6j12dir0qaxZ chyl20]# ls -ltr
total 12
-rw-r--r-- 1 root root 13 Oct 10 14:38 pwd.txt
-rwxr-xr-x 1 root root 1195 Oct 11 10:32 uninstall_emr_client.sh
-rwxr-xr-x 1 root root 3337 Oct 11 10:32 install_emr_client.sh其中install_emr_client.sh的程序内容：#!/usr/bin/bash
if [ $# != 2 ]
then
echo "Usage: sh $0 master_ip master_password_file(store root password plaintext)"
echo "such as: sh $0 172.30.XXX.XXX /opt/chyl20/pwd.txt"
exit 1;
fi
masterip=${1}
masterpwdfile=${2}
client_dir="/opt/emr_client/"
if ! type sshpass >/dev/null 2>&1; then
yum install -y sshpass
fi
if ! type java >/dev/null 2>&1; then
yum install -y java-1.8.0-openjdk
fi
[ -d ${client_dir} ] && echo "Please uninstall the installed client!" && exit 1
echo "Start to create file path..."
mkdir -p ${client_dir}
mkdir -p /etc/ecm
echo "Start to copy package from ${masterip} to local client(${client_dir})..."
echo " -copying hadoop"
sshpass -f ${masterpwdfile} scp -r -o 'StrictHostKeyChecking no' root@${masterip}:/usr/lib/hadoop-current ${client_dir}
echo " -copying zookeeper"
sshpass -f ${masterpwdfile} scp -r root@${masterip}:/usr/lib/zookeeper-current ${client_dir}
echo " -copying hive"
sshpass -f ${masterpwdfile} scp -r root@${masterip}:/usr/lib/hive-current ${client_dir}
echo " -copying spark"
sshpass -f ${masterpwdfile} scp -r root@${masterip}:/usr/lib/spark-current ${client_dir}
echo " -extra-jars"
sshpass -f ${masterpwdfile} scp -r root@${masterip}:/opt/apps/extra-jars ${client_dir}
echo "Start to link /usr/lib/\${app}-current to ${client_dir}\${app}..."
[ -L /usr/lib/hadoop-current ] && unlink /usr/lib/hadoop-current
ln -s ${client_dir}hadoop-current /usr/lib/hadoop-current
[ -L /usr/lib/zookeeper-current ] && unlink /usr/lib/zookeeper-current
ln -s ${client_dir}zookeeper-current /usr/lib/zookeeper-current
[ -L /usr/lib/hive-current ] && unlink /usr/lib/hive-current
ln -s ${client_dir}hive-current /usr/lib/hive-current
[ -L /usr/lib/spark-current ] && unlink /usr/lib/spark-current
ln -s ${client_dir}spark-current /usr/lib/spark-current
echo "Start to copy conf from ${masterip} to local client(/etc/ecm)..."
sshpass -f ${masterpwdfile} scp -r root@${masterip}:/etc/ecm/hadoop-conf /etc/ecm/hadoop-conf
sshpass -f ${masterpwdfile} scp -r root@${masterip}:/etc/ecm/zookeeper-conf /etc/ecm/zookeeper-conf
sshpass -f ${masterpwdfile} scp -r root@${masterip}:/etc/ecm/hive-conf /etc/ecm/hive-conf
sshpass -f ${masterpwdfile} scp -r root@${masterip}:/etc/ecm/spark-conf /etc/ecm/spark-conf
echo "Start to copy environment from ${masterip} to local client(/etc/profile.d)..."
sshpass -f ${masterpwdfile} scp root@${masterip}:/etc/profile.d/hdfs.sh /etc/profile.d/
sshpass -f ${masterpwdfile} scp root@${masterip}:/etc/profile.d/zookeeper.sh /etc/profile.d/
sshpass -f ${masterpwdfile} scp root@${masterip}:/etc/profile.d/yarn.sh /etc/profile.d/
sshpass -f ${masterpwdfile} scp root@${masterip}:/etc/profile.d/hive.sh /etc/profile.d/
sshpass -f ${masterpwdfile} scp root@${masterip}:/etc/profile.d/spark.sh /etc/profile.d/
if [ -L /usr/lib/jvm/java ]
then
unlink /usr/lib/jvm/java
fi
echo "" >>/etc/profile.d/hdfs.sh
echo export JAVA_HOME=/usr/lib/jvm/jre-1.8.0 >>/etc/profile.d/hdfs.sh
echo "Start to copy host info from ${masterip} to local client(/etc/hosts)"
sshpass -f ${masterpwdfile} scp root@${masterip}:/etc/hosts /etc/hosts_bak
cat /etc/hosts_bak | grep emr | grep cluster >>/etc/hosts
! egrep "^hadoop" /etc/group >& /dev/null && groupadd hadoop
! egrep "^hadoop" /etc/passwd >& /dev/null && useradd -g hadoop hadoop
echo "Install EMR client successfully!"uninstall_emr_client.sh的程序内容：
#!/usr/bin/bash
if [ $# != 0 ]
then
echo "Usage: sh $0 "
exit 1;
fi
client_dir="/opt/emr_client/"
echo "Start to delete path..."
[ -d ${client_dir} ] && rm -rf ${client_dir}
echo "Start to delete link /usr/lib/\${app}-current..."
[ -L /usr/lib/hadoop-current ] && unlink /usr/lib/hadoop-current
[ -L /usr/lib/zookeeper-current ] && unlink /usr/lib/zookeeper-current
[ -L /usr/lib/hive-current ] && unlink /usr/lib/hive-current
[ -L /usr/lib/spark-current ] && unlink /usr/lib/spark-current
echo "Start to delete conf..."
[ -d /etc/ecm ] && rm -rf /etc/ecm
echo "Start to delete environment..."
[ -f /etc/profile.d/hdfs.sh ] && rm -rf /etc/profile.d/hdfs.sh
[ -f /etc/profile.d/zookeeper.sh ] && rm -rf /etc/profile.d/zookeeper.sh
[ -f /etc/profile.d/yarn.sh ] && rm -rf /etc/profile.d/yarn.sh
[ -f /etc/profile.d/hive.sh ] && rm -rf /etc/profile.d/hive.sh
[ -f /etc/profile.d/spark.sh ] && rm -rf /etc/profile.d/spark.sh
echo "Start to remove host info..."
sed -i "/emr/d" /etc/hosts
echo "Start to remove user..."
egrep "^hadoop" /etc/passwd >& /dev/null && userdel -r hadoop
egrep "^hadoop" /etc/group >& /dev/null && groupdel hadoop
echo "Uninstall EMR client successfully!"pwd.txt中存放密码明文即可。
创建后给程序授权755
安装执行：
sh install_emr_client.sh 172.30.1*2.1*3 /opt/chyl20/pwd.txt卸载执行：
sh uninstall_emr_client.sh安装后重新连接会话，或重新打开一个PTY
验证：
运行hive
beeline -u jdbc:hive2://emr-header-1:10000 -e "show tables; " -n hadoop -p hadoop
运行hadoop
hadoop jar /usr/lib/hadoop-current/share/hadoop/mapreduce/hadoop-mapreduce-examples*.jar pi 10 10运行spark
spark-submit --name spark-pi --class org.apache.spark.examples.SparkPi --master yarn /usr/lib/spark-current/examples/jars/spark-examples_2.11-2.4.3.jar
