HDP 01.Ambari安装和配置

本文涉及的产品
云数据库 RDS MySQL Serverless,0.5-2RCU 50GB
简介: HDP 01.Ambari安装和配置

1、节点说明

1.2  ambari.tianlingqun.com

主要用于 Ambari管理( ambari-server )

用于存放元数据的MySQL数据库也安装到该节点(元数据主要包括:Ambari, Hive, Ranger, Hue, Azkaban, Superset等)

1.3 client-v[01,02]. tianlingqun.com

两台客户端机器,均安装 Azkaban-exec; Azkaban-web 独立安装(Azkaban-web不支持HA)到  client-v02.tianlingqun.com

1.4 两台 Master 机器 master-v[01,02]. tianlingqun.com

基中主要包括:hdfs, yarn, hbase 的高可用相关进程; Hive metadata, Hive server进程 及 Zookeeper进程等

1.5 5台 worker 节点 datanode-v[01~05]. tianlingqun.com

主要是 Datanode, NodeManager, HRegion Server等 进程

1.6 ipa-v[01,02]. tianlingqun.com

主要用于 FreeIPA“互为主从的两个节点”,FreeIPA是一个集 kerberos, ldap, dns于一体的管理工具,

详细请参考: FreeIPA

2、对机器进行配置:

1)修改/etc/hosts,并分发到每个节点

 a8ca95d006577c7497677e8bc613a9b.png

2)免密配置:ssh-keygen -t rsa

分发到其它机器:ssh-copy-id -i  ,ipa除外

 a459f420de3acc1fd95d6c1c1bbddd6.png

3)命令脚本:ssh_to_all_node.sh

4)分发脚本:sync_to_all_node.sh

5)关闭防火墙:

systemctl stop firewalld
systemctl disable firewalld
systemctl status firewalld

 f1821612eaaff9b067fd1240a33e40a.png

3、ambari安装:

1)

yum -y install wget  
# ambari
wget  http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.7.4.0/ambari-2.7.4.0-centos7.tar.gz
wget  http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.7.4.0/ambari-2.7.4.0-centos7.tar.gz.md5
# hdp
wget  http://public-repo-1.hortonworks.com/HDP/centos7/3.x/updates/3.1.4.0/HDP-3.1.4.0-centos7-rpm.tar.gz
wget  http://public-repo-1.hortonworks.com/HDP/centos7/3.x/updates/3.1.4.0/HDP-3.1.4.0-centos7-rpm.tar.gz.md5
# hdp gpl
wget  http://public-repo-1.hortonworks.com/HDP-GPL/centos7/3.x/updates/3.1.4.0/HDP-GPL-3.1.4.0-centos7-gpl.tar.gz
wget  http://public-repo-1.hortonworks.com/HDP-GPL/centos7/3.x/updates/3.1.4.0/HDP-GPL-3.1.4.0-centos7-gpl.tar.gz.md5
# hdp utils
wget  http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.22/repos/centos7/HDP-UTILS-1.1.0.22-centos7.tar.gz
wget  http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.22/repos/centos7/HDP-UTILS-1.1.0.22-centos7.tar.gz.md5

以上文件可以在linux直接下载或者下载以后在进行上传到指定目录即可,为了避免下载过程中出现问题,要进行MD5校验

 57c7c8ec24243757a963b44e961ace4.png

       2)关闭SELINUX=disabled:

 896913f70cf5369bee5668afd792524.png

       分发配置文件或者命令修改

yum -y install perl
ls /etc/selinux/config |xargs perl -pi -e 's|SELINUX=enforcing|SELINUX=disabled|g'
cat /etc/selinux/config|grep "SELINUX"|grep -v "grep"

   3)安装MySQL:

yum -y localinstall https://dev.mysql.com/get/mysql57-community-release-el7-11.noarch.rpm
 yum -y install mysql-community-server
systemctl enable mysqld
systemctl start mysqld
systemctl status mysqld

 9a50fa983c0347c5f4f554654687bf1.png

grep 'temporary password' /var/log/mysqld.log
mysql_secure_installation

 16b971290f861f6059ee426d09c4472.png

也可以登录MySQL以后在设置密码和降低密码复杂度:mysql -u root -p 密码

set password = password('123456');
GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' IDENTIFIED BY '123456';
SHOW VARIABLES LIKE 'validate_password%';
set global validate_password_policy=0;
set global validate_password_mixed_case_count=1;
set global validate_password_number_count=1;
set global validate_password_special_char_count=1;
set global validate_password_length=4;
SHOW VARIABLES LIKE 'validate_password%';
FLUSH PRIVILEGES;

 651f7fb5493fd02d10dbed4fa07c4ca.png

   4)创建ambari数据库,用于存放ambari元数据:

      表示服务端版本号大于4.1.00时会被执行,使用utf8编码

drop database if exists ambari;
CREATE DATABASE `ambari` /*!40100 DEFAULT CHARACTER SET utf8 */;
drop user  'ambari'@' ambari.tianlingqun.com';
GRANT ALL PRIVILEGES ON ambari.* TO 'ambari'@' ambari.tianlingqun.com' IDENTIFIED BY'Ambari_123' WITH
GRANT OPTION;
flush privileges;
show create database ambari;

   5)创建hive数据库,存放元数据:

CREATE DATABASE `hive` /*!40100 DEFAULT CHARACTER SET utf8 */;
drop user  'hive'@' master-v01.tianlingqun.com';
drop user  'hive'@' master-v02.tianlingqun.com';
GRANT ALL PRIVILEGES ON hive.* TO 'hive'@' master-v01.tianlingqun.com' IDENTIFIED BY 'Wfhive_4321' WITH
GRANT OPTION;
GRANT ALL PRIVILEGES ON hive.* TO 'hive'@' master-v02.tianlingqun.com' IDENTIFIED BY 'Wfhive_4321' WITH
GRANT OPTION;
flush privileges;
show create database hive;

   6)创建ranger数据库,存放元数据:

drop database if exists `ranger`;
use mysql;
create database `ranger`;
delete from mysql.user where User='rangeradmin';
grant all privileges on ranger.* to 'rangeradmin'@' master-v01.tianlingqun.com' identified by 'Wfrangeradmin_8899'
WITH GRANT OPTION;
grant all privileges on ranger.* to 'rangeradmin'@' master-v02.tianlingqun.com' identified by 'Wfrangeradmin_8899'
WITH GRANT OPTION;
grant all privileges on ranger.* to 'rangeradmin'@' client-v01.tianlingqun.com' identified by 'Wfrangeradmin_8899'
WITH GRANT OPTION;
grant all privileges on ranger.* to 'rangeradmin'@' client-v02.tianlingqun.com' identified by 'Wfrangeradmin_8899'
WITH GRANT OPTION;
grant all privileges on ranger.* to 'rangeradmin'@' ambari.tianlingqun.com' identified by 'Wfrangeradmin_8899'
WITH GRANT OPTION;
flush privileges;

   7)创建rangerkms数据库,存放元数据:

drop database if exists `rangerkms`;
use mysql;
create database `rangerkms`;
delete from user where User='rangerkms';
grant all privileges on rangerkms.* to 'rangerkms'@' master-v01.tianlingqun.com' identified by 'Rangerkms_123';
grant all privileges on rangerkms.* to 'rangerkms'@' master-v02.tianlingqun.com' identified by 'Rangerkms_123';
grant all privileges on rangerkms.* to 'rangerkms'@' ambari.tianlingqun.com' identified by 'Rangerkms_123';
flush privileges;

   8)创建superset数据库,存放元数据:

drop database if exists `superset`;
use mysql;
CREATE DATABASE `superset` /*!40100 DEFAULT CHARACTER SET utf8 */;
grant all privileges on superset.* to 'superset'@' client-v01.tianlingqun.com' identified by 'wfsuperset456';
grant all privileges on superset.* to 'superset'@' client-v02.tianlingqun.com' identified by 'wfsuperset456';
flush privileges;

4、配置同步时间:

yum -y install chrony
systemctl enable chronyd
systemctl start chronyd

5、配置Maximum Open Files Requirements

   服务器 vi /etc/security/limits.conf添加文件

*  -  nofile  409600
*  -  nproc  63535
root -  nofile  409600
root -  nproc  63535
执行 sysctl -p 命令,使上面修改生效 (注意:需要新开登陆会话,才能看到效果)
sh ./sync_to_all_node.sh /etc/security/limits.conf /etc/security/
sh ./ssh_to_all_node.sh "sysctl -p"
sh ./ssh_to_all_node.sh "ulimit -n"

4e964c807dad7b7cc0831f0500c7336.png

6、安装相关文件并分发其它机器:

yum -y install yum-utils createrepo yum-plugin-priorities
cd ~/scripts/;
sh ./ssh_to_all_node.sh "yum -y install yum-utils createrepo yum-plugin-priorities;"

7、安装httpd

yum -y install yum-utils createrepo
yum -y install httpd
mkdir -p /var/www/html/
systemctl enable httpd
systemctl start httpd
systemctl status httpd

8、配置yum相关的 repo 文件

vi /etc/yum.repos.d/ambari.repo
#VERSION_NUMBER=2.7.4.0-118
[ambari-2.7.4.0]
json.url = http://ambari.tianlingqun.com/HDP/hdp_urlinfo.json
name=ambari Version - ambari-2.7.4.0
baseurl= http://ambari.tianlingqun.com/ambari/centos7/2.7.4.0-118
gpgcheck=1
gpgkey= http://ambari.tianlingqun.com/ambari/centos7/2.7.4.0-118/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1
vi /etc/yum.repos.d/hdp.repo
#VERSION_NUMBER=3.1.4.0-315
[HDP-3.1.4.0]
name=HDP Version - HDP-3.1.4.0
baseurl= http://ambari.tianlingqun.com/HDP/centos7/3.1.4.0-315
gpgcheck=1
gpgkey= http://ambari.tianlingqun.com/HDP/centos7/3.1.4.0-315/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1
[HDP-UTILS-1.1.0.22]
name=HDP-UTILS Version - HDP-UTILS-1.1.0.22
baseurl= http://ambari.tianlingqun.com/HDP-UTILS/centos7/1.1.0.22
gpgcheck=1
gpgkey= http://ambari.tianlingqun.com/HDP/centos7/3.1.4.0-315/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1
vi /etc/yum.repos.d/hdp.gpl.repo
#VERSION_NUMBER=3.1.4.0-315
[HDP-GPL-3.1.4.0]
name=HDP-GPL Version - HDP-GPL-3.1.4.0
baseurl= http://ambari.tianlingqun.com/HDP-GPL/centos7/3.1.4.0-315
gpgcheck=1
gpgkey= http://ambari.tianlingqun.com/HDP-GPL/centos7/3.1.4.0-315/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1

9、校验

yum clean all
yum makecache
yum repolist

10、software解压缩jar包

tar -zxvf ambari-2.7.4.0-centos7.tar.gz
tar -zxvf HDP-3.1.4.0-centos7-rpm.tar.gz
tar -zxvf HDP-GPL-3.1.4.0-centos7-gpl.tar.gz
tar -zxvf HDP-UTILS-1.1.0.22-centos7.tar.gz
tar -zxvf jdk-8u231-linux-x64.tar.gz

11、创建软连接

cd /var/www/html/
ln -s /software/ambari ./ambari
ln -s /software/HDP ./HDP
ln -s /software/HDP-GPL ./HDP-GPL
ln -s /software/HDP-UTILS ./HDP-UTILS
ls -l ./

 8bd74bfaed7889431d9e862375a4bc8.png

12、启动时激活互联网

ETWORKING=yes
HOSTNAME=
cd ~/scripts/;
sh ./ssh_to_all_node.sh 'echo "NETWORKING=yes" >> /etc/sysconfig/network;hostname=$(hostname -f);
echo "HOSTNAME=${hostname}" >> /etc/sysconfig/network;'
sh ./ssh_to_all_node.sh "cat /etc/sysconfig/network;"

 f25ab582674b381c15fa26dd1aae513.png

13、

Install MySQL JDBC Driver (只需在 ambari 服务器上执行)
yum -y install mysql-connector-java-5*

14、安装 JDK再同步到其它节点

cd /software/
tar -xvf jdk-8u201-linux-x64.tar.gz
mkdir -p /usr/java/
mv jdk1.8.0_231/ /usr/java/
cd /usr/java/
ln -s ./jdk1.8.0_231 ./current
chown -R root:root ./jdk1.8.0_231
ls -l ./
cd ~/scripts/
sh ./sync_to_all_node.sh /usr/java/jdk1.8.0_231 /usr/java/
sh ./ssh_to_all_node.sh "mkdir -p /usr/java/;"
sh ./sync_to_all_node.sh /usr/java/jdk1.8.0_231 /usr/java/;
sh ./ssh_to_all_node.sh "cd /usr/java/;ln -s ./jdk1.8.0_231 ./current;ls -l ./;"
echo "" >> /etc/profile;
echo "#set Java environment" >> /etc/profile;
echo "export JAVA_HOME=/usr/java/current" >> /etc/profile;
echo "export JRE_HOME=\$JAVA_HOME/jre" >> /etc/profile;
echo "export CLASSPATH=.:\$JAVA_HOME/lib:\$JRE_HOME/lib:\$CLASSPATH" >> /etc/profile;
echo "export PATH=\$JAVA_HOME/bin:\$JRE_HOME/bin:\$PATH" >> /etc/profile;
java -version
sh ./sync_to_all_node.sh /etc/profile /etc/;

15、安装 ambari-server RPM包

yum -y install ambari-server

 8f74c91cf13480bc15ccc8c49ca6135.png

16、配置 ambari-server

ambari-server setup

9241a99bd0ea77a6a636e57ee1ce49e.png

a797a8e0b56bcd6ea9469e4686e90f5.png

17、生成 ambari 相关元数据

source /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql

 74a1b9dc1ef3878c2b03d8f88bc8a8e.png

 57b3865a9dcaac01c25b8001e381331.png

18、添加 jdbc 驱动到 ambari-server,执行如下命令

ambari-server setup --jdbc-db=mysql --jdbc-driver=/usr/share/java/mysql-connector-java.jar
#ambari-server setup --jdbc-db=mysql --jdbc-driver=/path/to/mysql/mysql-connector-java.jar

 3758c5dcc29a1ccf72fc58f9a8a5120.png

19、启动 ambari-server,执行如下命令

ambari-server start

 9a50228674ab1a53f8b2584b931b5ae.png

20、查看 ambari-server 当前状态的命令如下:

ambari-server status

 92482140a29fc8cd0ccdd4346edb6f8.png

21、启动成功后,可以执行类似如下命令,去查看日志,有没有报错信息

tail -n 400 /var/log/ambari-server/ambari-server.log

22、启动成功后,就可以浏览 Ambari 的 web 页面了,链接类似如下(端口默认 8080,也可以通过编辑

/etc/ambari-server/conf/ambari.properties 文件去个性化各参数)
http://xxx:8080

默认的用户名密码均为 admin

 05f2d9dfe6bde6045e312adc8260e40.png

23、ambari安装配置:

   1)开始配置:

 

 ac20f97c8c825255dcd26430c373b50.png

 

 a3da07ff9633ebb2c4dd10ad52b0ec9.png

 fe0c47ddb04ea59db724ad75713a49a.png

 85474d212262a0f6555d327c3242f66.png

 4e29965579bdd7da72f5a507bd332b4.png

 b7caa8ec095dc008e1b06ab82ef2153.png

 e09a872145af4725c4a39955c659067.png

 45c970e3541a18d34669fc9450df926.png

 5e9317d685ce8d03fe7fcbcfcb7db7d.png

 01e783830853c20ae4c48643226b9ab.png

cc608ebda236206e6c22dd224258456.png

 4e80529e2abbfdce111e604c10f4965.png

 fbb0a88cbcd95415726a8bd084b9df7.png48a8f4986f872f857d04307b418af7b.pngc58044f4b783886934f21a5d6414cd4.png8c7d995b803e3a97e44d029c4addc0b.png0ecbb840a232a1ade356e7b312313b7.png

feccf55b32156ba5ea63973f79d3499.png

fd5a52d395f5b73028b7b37996c6577.png

023c64f1c7558c15cb1c70284d0d258.png

bb264de27f06ef05aca183d7d4af044.png

d610dfdd1616dea0239e7521d989f36.png

638aa7ad178d133da80206a2707e61f.png

97b59dafe357ea86b7d4b9cfb77db74.png

22、HA高可用:

1、HDFS高可用 7cf9f247b6b0f7828cbd9ca3a1703cd.pngaddc92bf25c74e4955f600d0fa7f624.png 495082e1413d9176cd523ed530e9f9a.png

 8c5a0125d5f98769ba57ad462ae0dfb.png

 

691b57b3c450a6e5031ca90cc068928.png1b3507d6b068829fe4e60892409d97a.png  6f27c5348078b3d064ea390f6765fcb.pngf0956bea2bef79451bfa76717862a91.png

 9d4e83ad1003d93766c590710108dd3.png

 2a8a9a6a254ec670f0d3eab8651b848.png

fd8659f404c4aa226666978f29faf18.pngf5dfc671506af1c1f365863ab3895f0.png4861451c21886d0880b0d191dbfd88f.png

 d186fd2019304b169025db4202d119f.png

2、YARN高可用

4650f9ebfdd23a5779efab52bece73f.png

 70024fdff9091fdbab62c1a7c7c3bf0.png

fa25c5bae6690065a4995db5065a7ba.pnga80e6212eda8e4527bcc2624cfb97d8.png

ecee31bb35df07c4a64a49d5928e41d.png

3、hive高可用

900d87db413e2c440737260a0f2cd9d.png

4ae539a7e23f9b75533bae43e126a0e.png

89b4f13a7b58bb326aaa3a37a0aa0d5.png


相关实践学习
基于CentOS快速搭建LAMP环境
本教程介绍如何搭建LAMP环境,其中LAMP分别代表Linux、Apache、MySQL和PHP。
全面了解阿里云能为你做什么
阿里云在全球各地部署高效节能的绿色数据中心,利用清洁计算为万物互联的新世界提供源源不断的能源动力,目前开服的区域包括中国(华北、华东、华南、香港)、新加坡、美国(美东、美西)、欧洲、中东、澳大利亚、日本。目前阿里云的产品涵盖弹性计算、数据库、存储与CDN、分析与搜索、云通信、网络、管理与监控、应用服务、互联网中间件、移动服务、视频服务等。通过本课程,来了解阿里云能够为你的业务带来哪些帮助     相关的阿里云产品:云服务器ECS 云服务器 ECS(Elastic Compute Service)是一种弹性可伸缩的计算服务,助您降低 IT 成本,提升运维效率,使您更专注于核心业务创新。产品详情: https://www.aliyun.com/product/ecs
相关文章
|
4月前
|
SQL 分布式计算 Hadoop
Hadoop学习笔记(HDP)-Part.08 部署Ambari集群
01 关于HDP 02 核心组件原理 03 资源规划 04 基础环境配置 05 Yum源配置 06 安装OracleJDK 07 安装MySQL 08 部署Ambari集群 09 安装OpenLDAP 10 创建集群 11 安装Kerberos 12 安装HDFS 13 安装Ranger 14 安装YARN+MR 15 安装HIVE 16 安装HBase 17 安装Spark2 18 安装Flink 19 安装Kafka 20 安装Flume
85 0
Hadoop学习笔记(HDP)-Part.08 部署Ambari集群
|
9月前
|
关系型数据库 MySQL Java
安装Ambari和HDP
安装Ambari和HDP
205 1
|
关系型数据库 Java 大数据
CentOS7下利用Ambari搭建HDP大数据平台(下)
CentOS7下利用Ambari搭建HDP大数据平台(下)
305 0
CentOS7下利用Ambari搭建HDP大数据平台(下)
|
域名解析 SQL 分布式计算
CentOS7下利用Ambari搭建HDP大数据平台(上)
CentOS7下利用Ambari搭建HDP大数据平台(上)
380 0
CentOS7下利用Ambari搭建HDP大数据平台(上)
|
SQL 分布式数据库 HIVE
ambari发布新服务(HDP自带的服务)流程简介
ambari发布新服务(HDP自带的服务)流程简介
118 0
ambari发布新服务(HDP自带的服务)流程简介
|
关系型数据库 Java API
|
6月前
|
SQL 关系型数据库 网络安全
ambari 安装及注意事项
ambari从无到有安装
86 0
|
分布式计算 安全 大数据
ambari安装HDFS组件(一)
ambari安装HDFS组件(一)
225 0
ambari安装HDFS组件(一)
|
数据安全/隐私保护
Ambari 2.7.3.0安装新组件
Ambari 2.7.3.0安装新组件和之前版本略有不同,本文将简述安装新组件的简单过程。 前提是大家已经安装好Ambari 2.7.3.0 这时候由于有一些组件没有添加,就需要安装新的组件。
262 0
Ambari 2.7.3.0安装新组件
安装 OushuDB Ambari 插件
我们假设用户已经安装了HDP Ambari,并且已经安装了HDFS和Zookeeper。安装步骤可以参考: ● https://docs.hortonworks.com/HDPDocuments/Ambari/Ambari-2.4.2.0/index.html ● https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.3/index.html
138 0
安装 OushuDB Ambari 插件

相关实验场景

更多