centos7脚本离线化安装CDH6.3版本(ansible闪亮登场~~~)
CDH好不好,毋庸置疑,具体体现在该平台是闭源的。逻辑是这样的:既然敢闭源,那么,说明这平台确实好用,闭源都不怕挣不到钱。当然,作为大数据平台管理平台之一的CDH好像也有底气闭源啦。
软件开或者闭源,都只是它的一种运营模式,在此,不过多评价。
CDH有三种使用形式,第一,白嫖,功能砍了很多,第二,免费试用期限,免费期内和收费的版本一样功能,过期了砍成白嫖功能。第三,付费版本,所有功能都开放,如何选择,不在本文讨论范围,文章末尾会有给出详细图解。
那么,本文主要是如何自动化的脚本安装CDH6.3版本,在使用脚本前,我想,有些工作还是手动做吧,毕竟脚本不能写的太庞大,臃肿了。
基本的服务器信息概述:
说明:CDH安装的时候非常吃内存,8G是勉强可以跑,推荐是16G,虚拟机设置16G的话,宿主机可能先不行了。硬盘大一些,是怕安装的时候磁盘满造成意外停止。
文件下载地址:
链接:https://pan.baidu.com/s/12wNH03PZhqtSHNIwP8LgfA
提取码:cdhf
一,CDH脚本安装前的环境搭建工作
(1)关闭防火墙和selinux,以防止不必要的干扰。
关闭防火墙:systemctl disable firewalld && systemctl stop firewalld
关闭selinux: vim /etc/selinux/config 文件,修改成disabled
(2)ntpd时间服务器的搭建(时间服务器搭建完毕并不能立即使用,需要等待若干分钟,搭建写在脚本内并不合适)
(3)服务器之间的免密登陆
(4)本地仓库的搭建(基础软件安装仓库,一般使用centos7的系统安装ISO格式文件搭建,基础命令指得是gcc,openssl-devel这些基本的软件)
(5)编译安装MySQL5.7
编译安装MySQL也是脚本自动安装,请参照https://blog.csdn.net/alwaysbefine/article/details/118440336安装,并将后续初始化部分执行完成。
(6)进入MySQL,执行以下SQL语句,建立库和用户
CREATE DATABASE scm DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
grant all privileges on scm.* to scm@'localhost' identified by 'scm';
grant all privileges on scm.* to scm@'%' identified by 'scm';
flush privileges;
注意,我这里MySQL的scm用户设置的密码是scm,当然不建议这么简单的密码,请按实际需要更改,脚本内初始化数据库也要做相应更改哦。
(7)执行cdh安装脚本,该脚本放置在主服务器192.168.88.11上。
安装cdh的脚本:
#!/bin/bash # author zsk_john NGINX_HOME=`find / -name ansible_offline.tar.gz` tar zxf $NGINX_HOME rpm -ivh ansible_offline/* --nodeps cp -f /root/CDH-offline/ansible-hosts /etc/ansible/hosts ansible all -m copy -a'src=~/host.txt dest=~/' ansible all -m copy -a'src=~/freelogin.sh dest=~/' ansible all -m shell -a 'bash ~/freelogin.sh' JAVA_HOME=/usr/java/jdk1.8.0_181-cloudera/ PATH=.$PATH:$JAVA_HOME/bin CLASSPATH=$JAVA_HOME/jre/lib/ext:$JAVA_HOME/lib/tools.jar export CLASSPATH PATH JAVA_HOME export PATH=./usr/lib64/qt-3.3/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin:/usr/java/jdk1.8.0_181-cloudera//bin:/usr/local/src/cmake/bin:/usr/local/mysql/bin ansible all -m copy -a 'src=~/CDH-offline/oracle-j2sdk1.8-1.8.0+update181-1.x86_64.rpm dest=~/oracle-j2sdk1.8-1.8.0+update181-1.x86_64.rpm' ansible all -m copy -a 'src=~/CDH-offline/enterprise-debuginfo-6.3.1-1466458.el7.x86_64.rpm dest=~/enterprise-debuginfo-6.3.1-1466458.el7.x86_64.rpm' ansible all -m copy -a 'src=~/CDH-offline/cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm dest=~/cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm' ansible all -m copy -a 'src=~/CDH-offline/cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm dest=~/cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm' ansible all -m yum -a 'state=installed name=MySQL-python,bind-utils,psmisc,cyrus-sasl-plain,cyrus-sasl-gssapi,portmap,lsb,httpd,mod_ssl,openssl-devel,python-psycopg2' sleep 120 ansible all -m shell -a'rpm -ivh ~/oracle-j2sdk1.8-1.8.0+update181-1.x86_64.rpm' ansible all -m shell -a'rpm -ivh ~/enterprise-debuginfo-6.3.1-1466458.el7.x86_64.rpm' ansible all -m shell -a'rpm -ivh ~/cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm' ansible all -m shell -a'rpm -ivh ~/cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm' ansible all -m shell -a 'systemctl enable cloudera-scm-agent' ansible all -m shell -a 'systemctl start cloudera-scm-agent' ansible all -m copy -a 'src=/etc/cloudera-scm-agent/config.ini dest=/etc/cloudera-scm-agent/config.ini' ansible all -m shell -a 'mkdir -p /usr/share/java' ansible all -m copy -a 'src=~/CDH-offline/mysql-connector-java-5.1.47.jar dest=/usr/share/java/mysql-connector-java.jar' rpm -ivh cloudera-manager-server-6.3.1-1466458.el7.x86_64.rpm systemctl enable cloudera-scm-server systemctl start cloudera-scm-server ansible all -m shell -a 'systemctl restart cloudera-scm-agent' #初始化数据库,最后那个scm 是前面指定的scm用户密码,这里写成固定的,可根据需要更改哦 /opt/cloudera/cm/schema/scm_prepare_database.sh mysql scm scm scm systemctl restart cloudera-scm-server mv ~/CDH-offline/CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel* /opt/cloudera/parcel-repo/
2021-07-10 17:43:42,768 INFO WebServerImpl:com.cloudera.server.cmf.WebServerImpl: Started Jetty server. 2021-07-10 17:43:42,771 INFO ParcelUpdateService:com.cloudera.parcel.components.LocalParcelManagerImpl: Discovered parcel on CM server: CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel 2021-07-10 17:43:42,771 INFO ParcelUpdateService:com.cloudera.parcel.components.LocalParcelManagerImpl: Created torrent file: /opt/cloudera/parcel-repo/CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.torrent 2021-07-10 17:43:42,778 INFO ParcelUpdateService:com.turn.ttorrent.common.Torrent: Creating single-file torrent for CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel... 2021-07-10 17:43:42,782 INFO ParcelUpdateService:com.turn.ttorrent.common.Torrent: Hashing data from CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel with 8 threads (3972 pieces)... 2021-07-10 17:43:43,384 INFO ParcelUpdateService:com.turn.ttorrent.common.Torrent: ... 10% complete 2021-07-10 17:43:43,803 INFO ParcelUpdateService:com.turn.ttorrent.common.Torrent: ... 20% complete 2021-07-10 17:43:44,284 INFO ParcelUpdateService:com.turn.ttorrent.common.Torrent: ... 30% complete 2021-07-10 17:43:44,687 INFO ParcelUpdateService:com.turn.ttorrent.common.Torrent: ... 40% complete 2021-07-10 17:43:45,120 INFO ParcelUpdateService:com.turn.ttorrent.common.Torrent: ... 50% complete 2021-07-10 17:43:45,532 INFO ParcelUpdateService:com.turn.ttorrent.common.Torrent: ... 60% complete 2021-07-10 17:43:45,952 INFO ParcelUpdateService:com.turn.ttorrent.common.Torrent: ... 70% complete 2021-07-10 17:43:46,370 INFO ParcelUpdateService:com.turn.ttorrent.common.Torrent: ... 80% complete 2021-07-10 17:43:46,766 INFO ParcelUpdateService:com.turn.ttorrent.common.Torrent: ... 90% complete 2021-07-10 17:43:47,203 INFO ParcelUpdateService:com.turn.ttorrent.common.Torrent: Hashed 1 file(s) (2082186246 bytes) in 3972 pieces (3972 expected) in 4417.8ms. 2021-07-10 17:43:47,210 INFO ParcelUpdateService:com.turn.ttorrent.common.Torrent: Single-file torrent information: 2021-07-10 17:43:47,210 INFO ParcelUpdateService:com.turn.ttorrent.common.Torrent: Torrent name: CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel 2021-07-10 17:43:47,210 INFO ParcelUpdateService:com.turn.ttorrent.common.Torrent: Announced at: Seems to be trackerless 2021-07-10 17:43:47,210 INFO ParcelUpdateService:com.turn.ttorrent.common.Torrent: Created on..: Sat Jul 10 17:43:42 CST 2021
通过以上步骤,我们的CDH的web端应该就已经可以正常工作了。检测是否是正常的有如下方法:
[root@master ~]# netstat -antup | grep 7180 tcp 0 0 0.0.0.0:7180 0.0.0.0:* LISTEN 1027/java tcp 0 0 192.168.0.16:7180 192.168.0.111:54364 ESTABLISHED 1027/java [root@master ~]# jps 1027 Main 4117 Jps
可以看到服务端口已开。
mysql> use scm; Reading table information for completion of table and column names You can turn off this feature to get a quicker startup with -A Database changed mysql> show tables; +--------------------------------+ | Tables_in_scm | +--------------------------------+ | audits | | auth_roles | | auth_roles_scopes | | client_configs | | client_configs_to_hosts | | cluster_activated_releases | | cluster_activated_releases_aud | | cluster_managed_releases | | cluster_undistributed_releases | | clusters | | clusters_aud | | cm_peers | | cm_version | | command_schedules | | commands | | commands_detail | | config_containers | | configs | | configs_aud | | credentials | | data_context_to_base_service | | data_contexts | | diagnostics_events | | external_accounts | | external_accounts_aud | | external_mapping_auth_roles | | external_mappings | | global_settings | | host_template_to_role_conf_grp | | host_templates | | hosts | | hosts_aud | | metrics | | parcel_components | | parcels | | process_active_releases | | processes | | processes_detail | | releases | | releases_aud | | revisions | | role_config_groups | | role_config_groups_aud | | role_staleness_status | | roles | | roles_aud | | schema_version | | services | | services_aud | | snapshot_policies | | user_auth_roles | | user_roles | | user_settings | | users | +--------------------------------+ 54 rows in set (0.00 sec)
可以看到scm这个数据库有54张表,如果表的数目不对,请删除库重新初始化。
可以看到端口被激活就证明没有问题了。此时,我们的agent也就是客户端并没有做任何改动,这样服务器是认不到的,也就是192.168.88.11是认不到的。因此,需要修改文件
vim /etc/cloudera-scm-agent/config.ini
[General] # Hostname of the CM server. server_host=localhost # Port that the CM server is listening on. server_port=7182
更改为:
[General] # Hostname of the CM server. server_host=192.168.88.11 # Port that the CM server is listening on. server_port=7182
然后重启客户端:
在192.168.88.11这个服务器上执行:ansible all -m shell -a 'systemctl restart cloudera-scm-agent' 这条命令即可。
二,web端的操作
经过 第一大步的操作,如果严格安装执行,应该web端已经可以进入了。