其实,基本思路跟如下差不多,我就不多详细说了,贴出主要图。
博主,我是直接借鉴下面这位博主,来进行安装的!(灰常感谢他们!)
Ubuntu14.04用apt在线/离线安装CDH5.1.2[Apache Hadoop 2.3.0
https://www.zybuluo.com/ncepuwanghui/note/474966
ClouderManager官网
https://www.cloudera.com/documentation/manager/5-1-x/Cloudera-Manager-Installation-Guide/cm5ig_mysql.html#id_trn_pps_xm_unique_1
说在前面的话
我的机器是总共4台,分别为ubuntucmbigdata1、ubuntucmbigdata2、ubuntucmbigdata3和ubuntucmbigdata4。
在ubuntucmbigdata1上执行了
sudo apt-get install cloudera-manager-daemons cloudera-manager-server sudo apt-get install cloudera-manager-daemons cloudera-manager-agent
即,ubuntucmbigdata1是既做server,又做agent。
在ubuntucmbigdata2、ubuntucmbigdata3和ubuntucmbigdata4上执行了
sudo apt-get install cloudera-manager-daemons cloudera-manager-agent
即,ubuntucmbigdata2、3、和4是只做agent。
在ubuntucmbigdata1、ubuntucmbigdata2、ubuntucmbigdata3、ubuntucmbigdata4上
新建文件sudo vi /etc/apt/sources.list.d/cloudera-manager.list
,复制以下内容到cloudera.list里
# Packages for Cloudera Manager, Version 5, on Ubuntu 14.04 x86_64 deb [arch=amd64] http://archive.cloudera.com/cm5/ubuntu/trusty/amd64/cm trusty-cm5 contrib deb-src http://archive.cloudera.com/cm5/ubuntu/trusty/amd64/cm trusty-cm5 contrib
并且在在ubuntucmbigdata1、ubuntucmbigdata2、ubuntucmbigdata3、ubuntucmbigdata4上
向apt添加cloudera源公钥,在所有节点执行如下命令:
curl -s http://archive.cloudera.com/cm5/ubuntu/trusty/amd64/cm/archive.key | sudo apt-key add -
然后,再
sudo apt-get update
具体在后面我会详细教大家怎么安装。
的确,我也跟这位博主一样,也需要在实验室里部署。以及也是4台机器。(至于原因,大家都懂得,中国高校研究生喜欢4、3、2、1来进行加速比做实验论文)
在实验室安装过社区版的Hadoop,当时用的是普通PC机,安装的过程遇到了好多问题。当解决了大部分的安装问题之后由于实验室的需求,需要安装一个商业发行版的hadoop ,我们就采用的Cloudera 公司发行的版本。首先要说明的是CDH对内存要求比较高,当我们采用原来的4G内存的PC机,只要启动基本内存就满了,并且会拖慢整个集群的运算速度。所以说安装CDH对硬件的要求比较高。安装CDH 有三种安装方式:在线安装、半离线安装、离线安装。在线安装和离线安装我都安装过,虽然在线安装貌似简单但是实际用起来还是没有离线安装方便,我还是建议大家采用离线安装的方式。不过下面都会介绍在线安装的方式和离线安装的方式。希望对别人有所帮助
其它的,ubuntucmbigdata2、ubuntucmbigdata3和ubuntucmbigdata4。
参考 第一步:
Cloudera Manager安装之Cloudera Manager安装前准备(CentOS6.5)(一)
Ubuntu14.04安装之后的一些配置
参考
ubuntu 下 visudo的使用
在文件末尾(按快捷键,Shift + g)添加bigdata ALL=(ALL) NOPASSWD:ALL
ctrl+o 保存
ctrl+x 退出。
最后,得到如下
我这里,仅拿ubuntucmbigdata1来说,其他的ubuntucmbigdata2、ubuntucmbigdata3和ubuntucmbigdata4是一样的。
防火墙
参考
hadoop-2.6.0.tar.gz + spark-1.5.2-bin-hadoop2.6.tgz的集群搭建(单节点)(Ubuntu系统)
由此,表明Ubuntu14.04是默认没开启防火墙的。
我这里,仅拿ubuntucmbigdata1来说,其他的ubuntucmbigdata2、ubuntucmbigdata3和ubuntucmbigdata4是一样的。
相关服务的安装
如果没有安装的话可以使用下面的命令安装一下。
参考
Ubuntu 14.04 安装 Cloudera Manager 5 And CDH 5 (在线安装)
bigdata@ubuntucmbigdata1:~$ sudo apt-get install lrzsz bigdata@ubuntucmbigdata1:~$ sudo apt-get install openssh-server
这个工具要装(你懂得),2个理由:1,可以远程管理系统。2,Cloudera Manages 集群管理需要用该服务。
我这里,仅拿ubuntucmbigdata1来说,其他的ubuntucmbigdata2、ubuntucmbigdata3和ubuntucmbigdata4是一样的。
ssh 服务配置、vim的安装
参考
Ubuntu14.04安装之后的一些配置
打开sshd_config文件, 在 /etc/ssh 目录下。
# Authentication: LoginGraceTime 120 #PermitRootLogin without-password PermitRootLogin yes StrictModes yes
注释掉 #PermitRootLogin without-password 行。
添加 PermitRootLogin yes 行
修改后,如下:
我这里,仅拿ubuntucmbigdata1来说,其他的ubuntucmbigdata2、ubuntucmbigdata3和ubuntucmbigdata4是一样的。
修改hosts文件
vi /etc/hosts
在里面添加主机名和IP之间的对应关系
#127.0.1.1 ubuntucmbigdata1 192.168.86.31 ubuntucmbigdata1 192.168.86.32 ubuntucmbigdata2 192.168.86.33 ubuntucmbigdata3 192.168.86.34 ubuntucmbigdata4
我这里,仅拿ubuntucmbigdata1来说,其他的ubuntucmbigdata2、ubuntucmbigdata3和ubuntucmbigdata4是一样的。
主机名
bigdata@ubuntucmbigdata1:~$ sudo vim /etc/hostname bigdata@ubuntucmbigdata2:~$ sudo vim /etc/hostname
bigdata@ubuntucmbigdata3:~$ sudo vim /etc/hostname
bigdata@ubuntucmbigdata4:~$ sudo vim /etc/hostname
这里肯定是对的。
配置ssh免密码登录 (对于这一步,其实可以不做,因为,在CM时会自动安装,大家可以跳过)
见
Cloudera Manager安装之Cloudera Manager安装前准备(CentOS6.5)(一)
bigdata@ubuntucmbigdata1:~$ mkdir .ssh bigdata@ubuntucmbigdata1:~$ ssh-keygen -t rsa Generating public/private rsa key pair. Enter file in which to save the key (/home/bigdata/.ssh/id_rsa):(回车) Enter passphrase (empty for no passphrase):(回车) Enter same passphrase again: (回车) Your identification has been saved in /home/bigdata/.ssh/id_rsa. Your public key has been saved in /home/bigdata/.ssh/id_rsa.pub. The key fingerprint is: 58:c2:3f:25:cb:82:5b:d5:b4:b2:ea:54:ac:e2:ea:c8 bigdata@ubuntucmbigdata1 The key's randomart image is: +--[ RSA 2048]----+ | . | | . o . | | o = + | | . O * | | . + S | | o = . | | o + | |.. . + | |.Eo.. . | +-----------------+ bigdata@ubuntucmbigdata1:~$ cd .ssh bigdata@ubuntucmbigdata1:~/.ssh$ cat id_rsa.pub >> authorized_keys bigdata@ubuntucmbigdata1:~/.ssh$ cd .. bigdata@ubuntucmbigdata1:~$ chmod 700 .ssh bigdata@ubuntucmbigdata1:~$ chmod 600 .ssh/* bigdata@ubuntucmbigdata1:~$ ssh ubuntucmbigdata1 The authenticity of host 'ubuntucmbigdata1 (127.0.1.1)' can't be established. ECDSA key fingerprint is 1c:85:65:a1:d7:f0:fe:56:4d:86:0f:6e:25:e3:80:46. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'ubuntucmbigdata1' (ECDSA) to the list of known hosts. Welcome to Ubuntu 14.04.4 LTS (GNU/Linux 4.2.0-27-generic x86_64) * Documentation: https://help.ubuntu.com/ New release '16.04.2 LTS' available. Run 'do-release-upgrade' to upgrade to it. Last login: Sun Jun 25 14:45:01 2017 from 192.168.86.1 bigdata@ubuntucmbigdata1:~$ exit; logout Connection to ubuntucmbigdata1 closed. bigdata@ubuntucmbigdata1:~$ ssh ubuntucmbigdata1 Welcome to Ubuntu 14.04.4 LTS (GNU/Linux 4.2.0-27-generic x86_64) * Documentation: https://help.ubuntu.com/ New release '16.04.2 LTS' available. Run 'do-release-upgrade' to upgrade to it. Last login: Sun Jun 25 15:52:15 2017 from localhost bigdata@ubuntucmbigdata1:~$ exit;
我这里,仅拿ubuntucmbigdata1来说,其他的ubuntucmbigdata2、ubuntucmbigdata3和ubuntucmbigdata4是一样的。
将所有节点(ubuntucmbigdata2、ubuntucmbigdata3和ubuntucmbigdata4)中的共钥id_ras.pub拷贝到ubuntucmbigdata1中的authorized_keys文件中。
bigdata@ubuntucmbigdata2:~$ cat ~/.ssh/id_rsa.pub | ssh bigdata@ubuntucmbigdata1 'cat >> ~/.ssh/authorized_keys' The authenticity of host 'ubuntucmbigdata1 (192.168.86.31)' can't be established. ECDSA key fingerprint is 1c:85:65:a1:d7:f0:fe:56:4d:86:0f:6e:25:e3:80:46. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'ubuntucmbigdata1,192.168.86.31' (ECDSA) to the list of known hosts. bigdata@ubuntucmbigdata1's password: bigdata@ubuntucmbigdata2:~$
bigdata@ubuntucmbigdata3:~$ cat ~/.ssh/id_rsa.pub | ssh bigdata@ubuntucmbigdata1 'cat >> ~/.ssh/authorized_keys' The authenticity of host 'ubuntucmbigdata1 (192.168.86.31)' can't be established. ECDSA key fingerprint is 1c:85:65:a1:d7:f0:fe:56:4d:86:0f:6e:25:e3:80:46. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'ubuntucmbigdata1,192.168.86.31' (ECDSA) to the list of known hosts. bigdata@ubuntucmbigdata1's password: bigdata@ubuntucmbigdata3:~$
bigdata@ubuntucmbigdata4:~$ cat ~/.ssh/id_rsa.pub | ssh bigdata@ubuntucmbigdata1 'cat >> ~/.ssh/authorized_keys' The authenticity of host 'ubuntucmbigdata1 (192.168.86.31)' can't be established. ECDSA key fingerprint is 1c:85:65:a1:d7:f0:fe:56:4d:86:0f:6e:25:e3:80:46. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'ubuntucmbigdata1,192.168.86.31' (ECDSA) to the list of known hosts. bigdata@ubuntucmbigdata1's password: bigdata@ubuntucmbigdata4:~$
得到
bigdata@ubuntucmbigdata1:~$ cd .ssh bigdata@ubuntucmbigdata1:~/.ssh$ ls authorized_keys id_rsa id_rsa.pub known_hosts bigdata@ubuntucmbigdata1:~/.ssh$ cat authorized_keys ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCrEPrX5HHQCnwWoavXUoM8N+fINUckzZ7EcDjJqs1EYkrzltdmMrKQv9VeGGj2BPaGoxzmgit+DPFGuPmPgftBOPWQnLVoILqtQd+H5m0wrK67W7Y98cUpZ+0thCbnmUZRuAES1mt2xS/bL2ibw8AvRrd6HUHOhH17Tcd/2gok32ximn48kPzs6li5d2ay8rPqgqD3QQw3NlAhVmK7RVmpsvlHCTnhrQIoNCz8A+adtK5r1jSgH7uKmPAJTqMUP2Zgv02XQTf/kjW3eacNAWR96Khdh0HsktaK6HuC+FFC+8bXqAj4Hih25EX8RTMFMl3nhGT5FPMLyt7C0Z65QhKV bigdata@ubuntucmbigdata1 ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDi7mZhUMzB1h7XjWPIZo3w/Tps0Oxr3mgOeKVfoD4pme/lzgTz61FpEmgggc2WB+mNLKSG/WiazTzuSXKUdBXnM5LNeCGV3ZcWBxFrwp2+wAYH6L5wOGqRq+oQzKUHIsvqEc7TBpN3JmaqZOw4zVCwRBBzQ4cg1B2SjSsjDFLBpjlfM+N8UDG+g9OaZDlEu1z5oUyz0KsKbBtvkkSott+sB+EX4zYSDz/cwUPmSf76RJLk89AI8aX4wGS+P4Eahs5/J2wfLOnAWBG2NQoYVPKyN4sXj7OWw+aErriUJXOqLayYV5xSM2yJjT9WuzbNOiR/D7HGZBlC4wXWG+11GdO5 bigdata@ubuntucmbigdata2 ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZtE7Z4cXPjv/t0bC4Ozi3HuovI6XiNe+tJ7cQ6WKp/m44a8w/FqgB4HVZLLQusuXwQM/mEzHKiL1EdPsi5om/s7A7xyDakl2OSUOqki871+/TSjFVTUOOz/H/wwHvl2QzBkb8/dMZLphU1/E4odrFhhlhGGpjK5WUdY8cYH6yyli6W3oDPZ55UG+8EnM89T1AyhC1E81xmyMzGMCj4603oZFwckOeff5n2dqeXg0CRhXO9HhsF9kSZQnE2GmR5pLZlORMkitsCKIhaf5UOBzrGENeJylHGCV1F+KlYjQ0ND4ufTUq14aAnUwBOCqCMhB1tBOBGI4O3S0+j27xumwN bigdata@ubuntucmbigdata3 ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCXPtqzCU+AmRi+l5lysCEG8gbFKPItYjczvSZIe6+arOfgE/3Ozu2cwr9MABTjgd80hk7ZG4YNRxBPxKvXKtcZxGAe6poe1G5UOrlKOMLeqv1mnUiwtRVsiWp0qnPFcmMt17mpCyN7XnCOLTsdTcl+BVl0c2S2i1xoC6vRKvqV1C/nneJUmuD3KRR1zmR3+a9Wj4yhRRahO6ro/LhgqdwrEaR+NOuT+/1d058fXANToaqyUMa72pz5M2kdxZprumzmHAJ5KlOYImlGrahdLk1wPFckA2m1BMKKycxZHFm1X1O9HeSFrD9N+pdtZY1CXdc6N9hmWKzjBDqw71rZgdUd bigdata@ubuntucmbigdata4 bigdata@ubuntucmbigdata1:~/.ssh$
说明成功!
然后将中ubuntucmbigdata1的authorized_keys文件分发到所有节点上面。
bigdata@ubuntucmbigdata1:~/.ssh$ scp -r authorized_keys bigdata@ubuntucmbigdata2:~/.ssh/ The authenticity of host 'ubuntucmbigdata2 (192.168.86.32)' can't be established. ECDSA key fingerprint is 17:a2:fb:1e:47:dc:79:d5:2d:5a:aa:79:9b:10:c2:eb. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'ubuntucmbigdata2,192.168.86.32' (ECDSA) to the list of known hosts. bigdata@ubuntucmbigdata2's password: authorized_keys 100% 1624 1.6KB/s 00:00 bigdata@ubuntucmbigdata1:~/.ssh$ scp -r authorized_keys bigdata@ubuntucmbigdata3:~/.ssh/ The authenticity of host 'ubuntucmbigdata3 (192.168.86.33)' can't be established. ECDSA key fingerprint is 6a:32:65:61:98:78:3f:ad:08:2c:f3:a1:bc:c6:c4:91. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'ubuntucmbigdata3,192.168.86.33' (ECDSA) to the list of known hosts. bigdata@ubuntucmbigdata3's password: authorized_keys 100% 1624 1.6KB/s 00:00 bigdata@ubuntucmbigdata1:~/.ssh$ scp -r authorized_keys bigdata@ubuntucmbigdata4:~/.ssh/ The authenticity of host 'ubuntucmbigdata4 (192.168.86.34)' can't be established. ECDSA key fingerprint is f4:d5:92:17:dc:b1:8f:95:61:11:95:cd:d3:fa:2e:57. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'ubuntucmbigdata4,192.168.86.34' (ECDSA) to the list of known hosts. bigdata@ubuntucmbigdata4's password: authorized_keys 100% 1624 1.6KB/s 00:00 bigdata@ubuntucmbigdata1:~/.ssh$
大家通过ssh 相互访问,如果都能无密码访问,代表ssh配置成功。这里很简单,我就不多赘述了。
bigdata@ubuntucmbigdata1:~$ ssh ubuntucmbigdata1 Welcome to Ubuntu 14.04.4 LTS (GNU/Linux 4.2.0-27-generic x86_64) * Documentation: https://help.ubuntu.com/ New release '16.04.2 LTS' available. Run 'do-release-upgrade' to upgrade to it. Last login: Sun Jun 25 15:52:23 2017 from localhost bigdata@ubuntucmbigdata1:~$ exit;Connection to ubuntucmbigdata1 closed. bigdata@ubuntucmbigdata1:~$ ssh ubuntucmbigdata2 Welcome to Ubuntu 14.04.4 LTS (GNU/Linux 4.2.0-27-generic x86_64) * Documentation: https://help.ubuntu.com/ New release '16.04.2 LTS' available. Run 'do-release-upgrade' to upgrade to it. Last login: Sun Jun 25 15:56:00 2017 from localhost bigdata@ubuntucmbigdata2:~$ exit; logout Connection to ubuntucmbigdata2 closed. bigdata@ubuntucmbigdata1:~$ ssh ubuntucmbigdata3 Welcome to Ubuntu 14.04.4 LTS (GNU/Linux 4.2.0-27-generic x86_64) * Documentation: https://help.ubuntu.com/ New release '16.04.2 LTS' available. Run 'do-release-upgrade' to upgrade to it. Last login: Sun Jun 25 15:56:14 2017 from localhost bigdata@ubuntucmbigdata3:~$ exit; logout Connection to ubuntucmbigdata3 closed. bigdata@ubuntucmbigdata1:~$ ssh ubuntucmbigdata4 Welcome to Ubuntu 14.04.4 LTS (GNU/Linux 4.2.0-27-generic x86_64) * Documentation: https://help.ubuntu.com/ New release '16.04.2 LTS' available. Run 'do-release-upgrade' to upgrade to it. Last login: Sun Jun 25 15:56:27 2017 from localhost bigdata@ubuntucmbigdata4:~$ exit; logout Connection to ubuntucmbigdata4 closed. bigdata@ubuntucmbigdata1:~$
这样,就说明是成功!
JDK的安装(建议1.8)(对于这一步,其实可以不做,因为,在CM时会自动安装,大家可以跳过)(但是要先卸载linux系统自带的openjdk,因为自动也是安装的是orcal公司的)
见
Cloudera Manager安装之Cloudera Manager安装前准备(CentOS6.5)(一)
这里,因为是ubuntu14.04版本,作为专业人员来说,
Ubuntu14.04下Neo4j图数据库官网安装部署步骤(图文详解)(博主推荐)
bigdata@ubuntucmbigdata1:~$ sudo apt-get remove openjdk*
bigdata@ubuntucmbigdata1:/usr/local$ pwd /usr/local bigdata@ubuntucmbigdata1:/usr/local$ ll total 40 drwxr-xr-x 10 root root 4096 2月 18 2016 ./ drwxr-xr-x 10 root root 4096 2月 18 2016 ../ drwxr-xr-x 2 root root 4096 2月 18 2016 bin/ drwxr-xr-x 2 root root 4096 2月 18 2016 etc/ drwxr-xr-x 2 root root 4096 2月 18 2016 games/ drwxr-xr-x 2 root root 4096 2月 18 2016 include/ drwxr-xr-x 4 root root 4096 2月 18 2016 lib/ lrwxrwxrwx 1 root root 9 6月 24 15:00 man -> share/man/ drwxr-xr-x 2 root root 4096 2月 18 2016 sbin/ drwxr-xr-x 7 root root 4096 2月 18 2016 share/ drwxr-xr-x 2 root root 4096 2月 18 2016 src/ bigdata@ubuntucmbigdata1:/usr/local$ sudo mkdir jdk bigdata@ubuntucmbigdata1:/usr/local$ ll total 44 drwxr-xr-x 11 root root 4096 6月 25 16:08 ./ drwxr-xr-x 10 root root 4096 2月 18 2016 ../ drwxr-xr-x 2 root root 4096 2月 18 2016 bin/ drwxr-xr-x 2 root root 4096 2月 18 2016 etc/ drwxr-xr-x 2 root root 4096 2月 18 2016 games/ drwxr-xr-x 2 root root 4096 2月 18 2016 include/ drwxr-xr-x 2 root root 4096 6月 25 16:08 jdk/ drwxr-xr-x 4 root root 4096 2月 18 2016 lib/ lrwxrwxrwx 1 root root 9 6月 24 15:00 man -> share/man/ drwxr-xr-x 2 root root 4096 2月 18 2016 sbin/ drwxr-xr-x 7 root root 4096 2月 18 2016 share/ drwxr-xr-x 2 root root 4096 2月 18 2016 src/ bigdata@ubuntucmbigdata1:/usr/local$ sudo chown -R bigdata:bigdata jdk/ bigdata@ubuntucmbigdata1:/usr/local$ cd jdk/ bigdata@ubuntucmbigdata1:/usr/local/jdk$ pwd /usr/local/jdk bigdata@ubuntucmbigdata1:/usr/local/jdk$ sudo rz rz waiting to receive. bigdata@ubuntucmbigdata1:/usr/local/jdk$ ll total 177000 drwxr-xr-x 2 bigdata bigdata 4096 6月 25 16:09 ./ drwxr-xr-x 11 root root 4096 6月 25 16:08 ../ -rw-r--r-- 1 root root 181238643 9月 3 2016 jdk-8u60-linux-x64.tar.gz bigdata@ubuntucmbigdata1:/usr/local/jdk$ sudo tar -zxvf jdk-8u60-linux-x64.tar.gz
bigdata@ubuntucmbigdata1:/usr/local/jdk$ ll total 177004 drwxr-xr-x 3 bigdata bigdata 4096 6月 25 16:09 ./ drwxr-xr-x 11 root root 4096 6月 25 16:08 ../ drwxr-xr-x 8 uucp 143 4096 8月 5 2015 jdk1.8.0_60/ -rw-r--r-- 1 root root 181238643 9月 3 2016 jdk-8u60-linux-x64.tar.gz bigdata@ubuntucmbigdata1:/usr/local/jdk$ sudo rm jdk-8u60-linux-x64.tar.gz bigdata@ubuntucmbigdata1:/usr/local/jdk$ sudo chown -R bigdata:bigdata jdk1.8.0_60/ bigdata@ubuntucmbigdata1:/usr/local/jdk$ ll total 12 drwxr-xr-x 3 bigdata bigdata 4096 6月 25 16:10 ./ drwxr-xr-x 11 root root 4096 6月 25 16:08 ../ drwxr-xr-x 8 bigdata bigdata 4096 8月 5 2015 jdk1.8.0_60/ bigdata@ubuntucmbigdata1:/usr/local/jdk$ sudo vim /etc/profile bigdata@ubuntucmbigdata1:/usr/local/jdk$ sudo source /etc/profile sudo: source: command not found bigdata@ubuntucmbigdata1:/usr/local/jdk$ source /etc/profile bigdata@ubuntucmbigdata1:/usr/local/jdk$ java -version java version "1.8.0_60" Java(TM) SE Runtime Environment (build 1.8.0_60-b27) Java HotSpot(TM) 64-Bit Server VM (build 25.60-b23, mixed mode) bigdata@ubuntucmbigdata1:/usr/local/jdk$
我这里,仅拿ubuntucmbigdata1来说,其他的ubuntucmbigdata2、ubuntucmbigdata3和ubuntucmbigdata4是一样的。
以及,我们还可以设置文件打开数量和用于最大进程数
文件打开数量
参考
Cloudera Manager安装之Cloudera Manager安装前准备(CentOS6.5)(一)
bigdata@ubuntucmbigdata1:~$ sudo vi /etc/security/limits.conf
* soft nofile 65535 * hard nofile 65535 * soft nproc 32000 * hard nproc 32000
我这里,仅拿ubuntucmbigdata1来说,其他的ubuntucmbigdata2、ubuntucmbigdata3和ubuntucmbigdata4是一样的。
ubuntu系统的root用户开启
参考
Ubuntu14.04安装之后的一些配置
bigdata@ubuntucmbigdata1:~$ sudo -s root@ubuntucmbigdata1:~# pwd /home/bigdata root@ubuntucmbigdata1:~#
MySQL的安装(以下步骤仅仅在ubuntucmbigdata1这台机器上做)
注意:Cloudera-Manages-Server 端 需要安装 MySQL,Cloudera-Manages-Agent 端 可以不装。
即,在我这里就是,Cloudera-Manages-Server 是cmbigdata1。同时,我也是喜欢把oozie、hive等这些,都是安装在cmbigdata1这台机器上。
Cloudera-Manages-Agent,在我这里就是,cmbigdata2、cmbigdata3和cmbigdata4。
参考
60分钟内从零起步驾驭Hive实战学习笔记(Ubuntu里安装mysql)
Ubuntu 14.04 MySQL 安装配置 For Cloudera Manages 5
root@ubuntucmbigdata1:~# apt-get install mysql-server
。。。 。。。 Setting up libdbd-mysql-perl (4.025-1ubuntu0.1) ... Setting up libterm-readkey-perl (2.31-1) ... Setting up mysql-client-5.5 (5.5.55-0ubuntu0.14.04.1) ... Setting up mysql-server-5.5 (5.5.55-0ubuntu0.14.04.1) ... 170625 16:28:33 [Warning] Using unique option prefix key_buffer instead of key_buffer_size is deprecated and will be removed in a future release. Please use the full name instead. 170625 16:28:33 [Note] Ignoring --secure-file-priv value as server is running with --bootstrap. 170625 16:28:33 [Note] /usr/sbin/mysqld (mysqld 5.5.55-0ubuntu0.14.04.1) starting as process 7930 ... mysql start/running, process 8062 Setting up libhtml-template-perl (2.95-1) ... Processing triggers for ureadahead (0.100.0-16) ... Setting up mysql-server (5.5.55-0ubuntu0.14.04.1) ... Processing triggers for libc-bin (2.19-0ubuntu6.7) ... root@ubuntucmbigdata1:~#
安装成功!
启动mysql服务
安装MySql成功后,mysql服务会自动启动
root@ubuntucmbigdata1:~# service mysql start
或者
root@ubuntucmbigdata1:~# sudo /etc/init.d/mysql start
安装mysql JDBC驱动
root@ubuntucmbigdata1:~# apt-get install libmysql-java
设置MySQL的root用户设置密码,这一步,就不需要做了,在CentOS版本里,需要做。在Ubuntu版本,在安装时,就已经设置了。
跳过,就是
创建Cloudera-Manages-Server 需要用到的数据库表 for MySql。
Creating Databases for Activity Monitor, Reports Manager, Hive Metastore Server, Sentry Server, Cloudera Navigator AuditServer, and Cloudera Navigator Metadata Server
root@ubuntucmbigdata1:~# mysql -root -p Enter password: (密码为rootroot) Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 42 Server version: 5.5.55-0ubuntu0.14.04.1 (Ubuntu) Copyright (c) 2000, 2017, Oracle and/or its affiliates. All rights reserved. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. mysql>
执行创建数据库脚本语句。
mysql> create database hive DEFAULT CHARACTER SET utf8; Ignoring query to other database mysql> grant all on hive.* TO 'hive'@'localhost' IDENTIFIED BY 'hive_password'; Ignoring query to other database mysql> grant all on *.* TO 'hive'@'ubuntucmbigdata1' IDENTIFIED BY 'hive_password'; Ignoring query to other database mysql> grant all on hive.* TO 'hive'@'%' IDENTIFIED BY 'hive_password'; Ignoring query to other database mysql> create database oozie DEFAULT CHARACTER SET utf8; Ignoring query to other database mysql> grant all on oozie.* TO 'oozie'@'localhost' IDENTIFIED BY 'oozie_password'; Ignoring query to other database mysql> grant all on *.* TO 'oozie'@'ubuntucmbigdata1' IDENTIFIED BY 'oozie_password'; Ignoring query to other database mysql> grant all on oozie.* TO 'oozie'@'%' IDENTIFIED BY 'oozie_password'; Ignoring query to other database mysql> create database amon DEFAULT CHARACTER SET utf8; Ignoring query to other database mysql> grant all on amon.* TO 'amon'@'localhost' IDENTIFIED BY 'amon_password'; Ignoring query to other database mysql> grant all on *.* TO 'amon'@'ubuntucmbigdata1' IDENTIFIED BY 'amon_password'; Ignoring query to other database mysql> grant all on amon.* TO 'amon'@'%' IDENTIFIED BY 'amon_password'; Ignoring query to other database mysql> create database smon DEFAULT CHARACTER SET utf8; Ignoring query to other database mysql> grant all on smon.* TO 'smon'@'localhost' IDENTIFIED BY 'smon_password'; Ignoring query to other database mysql> grant all on *.* TO 'smon'@'ubuntucmbigdata1' IDENTIFIED BY 'smon_password'; Ignoring query to other database mysql> grant all on smon.* TO 'smon'@'%' IDENTIFIED BY 'smon_password'; Ignoring query to other database mysql> create database rman DEFAULT CHARACTER SET utf8; Ignoring query to other database mysql> grant all on rman.* TO 'rman'@'localhost' IDENTIFIED BY 'rman_password'; Ignoring query to other database mysql> grant all on *.* TO 'rman'@'ubuntucmbigdata1' IDENTIFIED BY 'rman_password'; Ignoring query to other database mysql> grant all on rman.* TO 'rman'@'%' IDENTIFIED BY 'rman_password'; Ignoring query to other database mysql> create database hmon DEFAULT CHARACTER SET utf8; Ignoring query to other database mysql> grant all on hmon.* TO 'hmon'@'localhost' IDENTIFIED BY 'hmon_password'; Ignoring query to other database mysql> grant all on *.* TO 'hmon'@'ubuntucmbigdata1' IDENTIFIED BY 'hmon_password'; Ignoring query to other database mysql> grant all on hmon.* TO 'hmon'@'%' IDENTIFIED BY 'hmon_password'; Ignoring query to other database mysql> create database nav DEFAULT CHARACTER SET utf8; Ignoring query to other database mysql> grant all on nav.* TO 'nav'@'localhost' IDENTIFIED BY 'nav_password'; Ignoring query to other database mysql> grant all on *.* TO 'nav'@'ubuntucmbigdata1' IDENTIFIED BY 'nav_password'; Ignoring query to other database mysql> grant all on nav.* TO 'nav'@'%' IDENTIFIED BY 'nav_password'; Ignoring query to other database mysql>
mysql> flush privileges; Query OK, 0 rows affected (0.08 sec) mysql> select user,host,password from mysql.user; +------------------+------------------+-------------------------------------------+ | user | host | password | +------------------+------------------+-------------------------------------------+ | root | localhost | *6C362347EBEAA7DF44F6D34884615A35095E80EB | | root | ubuntucmbigdata1 | *6C362347EBEAA7DF44F6D34884615A35095E80EB | | root | 127.0.0.1 | *6C362347EBEAA7DF44F6D34884615A35095E80EB | | root | ::1 | *6C362347EBEAA7DF44F6D34884615A35095E80EB | | debian-sys-maint | localhost | *0C134AA3949C6DEE1804B1D1D641A7C2824C599A | | scm | % | *E2D6A4ADCCA7B38098E85EAF9BB785AB21451139 | | scm | ubuntucmbigdata1 | *E2D6A4ADCCA7B38098E85EAF9BB785AB21451139 | | scm | localhost | *E2D6A4ADCCA7B38098E85EAF9BB785AB21451139 | | hive | localhost | *8AC2E431CC7A9F2C4C0E51A65B8D8175892D9F22 | | hive | ubuntucmbigdata1 | *8AC2E431CC7A9F2C4C0E51A65B8D8175892D9F22 | | hive | % | *8AC2E431CC7A9F2C4C0E51A65B8D8175892D9F22 | | oozie | localhost | *81A1BB46F79EBD0AA76E6EFAA31D62458CFCAF62 | | oozie | ubuntucmbigdata1 | *81A1BB46F79EBD0AA76E6EFAA31D62458CFCAF62 | | oozie | % | *81A1BB46F79EBD0AA76E6EFAA31D62458CFCAF62 | | amon | localhost | *4A1243A12C2625553EC0465DD4E640DB056AA07A | | amon | ubuntucmbigdata1 | *4A1243A12C2625553EC0465DD4E640DB056AA07A | | amon | % | *4A1243A12C2625553EC0465DD4E640DB056AA07A | | smon | localhost | *F65C49868BBCA6D143F5AF81D1864B1C64DF25B4 | | smon | ubuntucmbigdata1 | *F65C49868BBCA6D143F5AF81D1864B1C64DF25B4 | | smon | % | *F65C49868BBCA6D143F5AF81D1864B1C64DF25B4 | | rman | localhost | *AEF345BFE495D8E678EA9D3D5708FD110AD2F08E | | rman | ubuntucmbigdata1 | *AEF345BFE495D8E678EA9D3D5708FD110AD2F08E | | rman | % | *AEF345BFE495D8E678EA9D3D5708FD110AD2F08E | | hmon | localhost | *DC736DB1383C5AC16BFE895C856F8D30214AF19A | | hmon | ubuntucmbigdata1 | *DC736DB1383C5AC16BFE895C856F8D30214AF19A | | hmon | % | *DC736DB1383C5AC16BFE895C856F8D30214AF19A | | nav | localhost | *993A16D875A9AB1932CC2F57E7DADA76AD35C3B2 | | nav | ubuntucmbigdata1 | *993A16D875A9AB1932CC2F57E7DADA76AD35C3B2 | | nav | % | *993A16D875A9AB1932CC2F57E7DADA76AD35C3B2 | +------------------+------------------+-------------------------------------------+ 29 rows in set (0.00 sec) mysql> exit;
本文转自大数据躺过的坑博客园博客,原文链接:http://www.cnblogs.com/zlslch/p/7078450.html,如需转载请自行联系原作者