大数据Hadoop环境搭建

下载JDK1.8安装包
wget https://download.java.net/openjdk/jdk8u41/ri/openjdk-8u41-b04-linux-x64-14_jan_2020.tar.gz

解压安装包
tar -zxvf openjdk-8u41-b04-linux-x64-14_jan_2020.tar.gz

移动java8文件夹中
mv java-se-8u41-ri/ /usr/java8

配置Java环境变量
echo 'export JAVA_HOME=/usr/java8' >> /etc/profile
echo 'export PATH=$PATH:$JAVA_HOME/bin' >> /etc/profile
source /etc/profile

查看版本
java -version

下载Hadoop安装包
wget --no-check-certificate https://mirrors.bfsu.edu.cn/apache/hadoop/common/hadoop-2.10.1/hadoop-2.10.1.tar.gz

解压Hadoop安装包至/opt/hadoop
tar -zxvf hadoop-2.10.1.tar.gz -C /opt/
mv /opt/hadoop-2.10.1 /opt/hadoop

配置Hadoop环境变量
echo 'export HADOOP_HOME=/opt/hadoop/' >> /etc/profile
echo 'export PATH=$PATH:$HADOOP_HOME/bin' >> /etc/profile
echo 'export PATH=$PATH:$HADOOP_HOME/sbin' >> /etc/profile
source /etc/profile

修改配置文件yarn-env.sh和hadoop-env.sh
echo "export JAVA_HOME=/usr/java8" >> /opt/hadoop/etc/hadoop/yarn-env.sh
echo "export JAVA_HOME=/usr/java8" >> /opt/hadoop/etc/hadoop/hadoop-env.sh

测试Hadoop是否安装成功
hadoop version

修改Hadoop配置文件 core-site.xml
vim /opt/hadoop/etc/hadoop/core-site.xml

    <property>
        <name>hadoop.tmp.dir</name>
        <value>file:/opt/hadoop/tmp</value>
        <description>location to store temporary files</description>
    </property>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://localhost:9000</value>
    </property>

修改Hadoop配置文件 hdfs-site.xml
vim /opt/hadoop/etc/hadoop/hdfs-site.xml

    <property>
        <name>dfs.replication</name>
        <value>1</value>
    </property>
    <property>
        <name>dfs.namenode.name.dir</name>
        <value>file:/opt/hadoop/tmp/dfs/name</value>
    </property>
    <property>
        <name>dfs.datanode.data.dir</name>
        <value>file:/opt/hadoop/tmp/dfs/data</value>
    </property>

配置SSH免密登录
ssh-keygen -t rsa

将公钥添加到authorized_keys文件中
cd .ssh
cat id_rsa.pub >> authorized_keys

初始化namenode
hadoop namenode -format

启动Hadoop
start-dfs.sh

start-yarn.sh

查看启动进程
jps

访问web8088端口和50070端口