0x01 Dockerfile文件的编写
1. 编写Dockerfile文件
为了方便，我复制了一份hive_sny_all的文件，取名flume_sny_all。
a. Flume安装步骤
参考文章：D009 复制粘贴玩大数据之安装与配置Flume集群
- 其实安装内容都是一样的，这里只是就根据我写的步骤整理了一下
2. 编写Dockerfile文件的关键点
与D006 复制粘贴玩大数据之Dockerfile安装HBase集群的“0x01 3. a. Dockerfile参考文件”相比较，不同点体现在：
具体步骤：
a. 添加安装包并解压（ADD指令会自动解压）
#添加Flume ADD ./apache-flume-1.8.0-bin.tar.gz /usr/local/
b. 添加环境变量（FLUME_HOME、PATH）
#Flume环境变量 ENV FLUME_HOME /usr/local/apache-flume-1.8.0-bin
#PATH里面追加内容 $FLUME_HOME/bin:
c. 添加配置文件（注意给之前的语句加“&& \”，表示未结束）
&& \ mv /tmp/flume-env.sh $FLUME_HOME/conf/flume-env.sh
3. 完整的Dockerfile文件参考
a. 安装hadoop、spark、zookeeper、hbase、hive、flume
FROM ubuntu MAINTAINER shaonaiyi shaonaiyi@163.com ENV BUILD_ON 2019-01-28 RUN apt-get update -qqy #添加JDK ADD ./jdk-8u161-linux-x64.tar.gz /usr/local/ #添加hadoop ADD ./hadoop-2.7.5.tar.gz /usr/local/ #添加scala ADD ./scala-2.11.8.tgz /usr/local/ #添加spark ADD ./spark-2.2.0-bin-hadoop2.7.tgz /usr/local/ #添加zookeeper ADD ./zookeeper-3.4.10.tar.gz /usr/local/ #添加HBase ADD ./hbase-1.2.6-bin.tar.gz /usr/local/ #添加Hive ADD ./apache-hive-2.3.3-bin.tar.gz /usr/local/ #添加Flume ADD ./apache-flume-1.8.0-bin.tar.gz /usr/local/ ENV CHECKPOINT 2019-03-11 #增加JAVA_HOME环境变量 ENV JAVA_HOME /usr/local/jdk1.8.0_161 #hadoop环境变量 ENV HADOOP_HOME /usr/local/hadoop-2.7.5 #scala环境变量 ENV SCALA_HOME /usr/local/scala-2.11.8 #spark环境变量 ENV SPARK_HOME /usr/local/spark-2.2.0-bin-hadoop2.7 #zk环境变量 ENV ZK_HOME /usr/local/zookeeper-3.4.10 #HBase环境变量 ENV HBASE_HOME /usr/local/hbase-1.2.6 #Hive环境变量 ENV HIVE_HOME /usr/local/apache-hive-2.3.3-bin #Flume环境变量 ENV FLUME_HOME /usr/local/apache-flume-1.8.0-bin #将环境变量添加到系统变量中 RUN ssh-keygen -t rsa -f ~/.ssh/id_rsa -P '' && \ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys && \ chmod 600 ~/.ssh/authorized_keys #复制配置到/tmp目录 COPY config /tmp #zk环境变量 ENV ZK_HOME /usr/local/zookeeper-3.4.10 #HBase环境变量 ENV HBASE_HOME /usr/local/hbase-1.2.6 #Hive环境变量 ENV HIVE_HOME /usr/local/apache-hive-2.3.3-bin #Flume环境变量 ENV FLUME_HOME /usr/local/apache-flume-1.8.0-bin #将环境变量添加到系统变量中 ENV PATH $FLUME_HOME/bin:$HIVE_HOME/bin:$HBASE_HOME/bin:$ZK_HOME/bin:$SCALA_HOME/bin:$SPARK_HOME/bin:$HADOOP_HOME/bin:$JAVA_HOME/bin:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar:$PATH RUN ssh-keygen -t rsa -f ~/.ssh/id_rsa -P '' && \ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys && \ chmod 600 ~/.ssh/authorized_keys #复制配置到/tmp目录 COPY config /tmp #将配置移动到正确的位置 RUN mv /tmp/ssh_config ~/.ssh/config && \ mv /tmp/profile /etc/profile && \ mv /tmp/masters $SPARK_HOME/conf/masters && \ cp /tmp/slaves $SPARK_HOME/conf/ && \ mv /tmp/spark-defaults.conf $SPARK_HOME/conf/spark-defaults.conf && \ mv /tmp/spark-env.sh $SPARK_HOME/conf/spark-env.sh && \ mv /tmp/hadoop-env.sh $HADOOP_HOME/etc/hadoop/hadoop-env.sh && \ mv /tmp/hdfs-site.xml $HADOOP_HOME/etc/hadoop/hdfs-site.xml && \ mv /tmp/core-site.xml $HADOOP_HOME/etc/hadoop/core-site.xml && \ mv /tmp/yarn-site.xml $HADOOP_HOME/etc/hadoop/yarn-site.xml && \ mv /tmp/mapred-site.xml $HADOOP_HOME/etc/hadoop/mapred-site.xml && \ mv /tmp/master $HADOOP_HOME/etc/hadoop/master && \ mv /tmp/slaves $HADOOP_HOME/etc/hadoop/slaves && \ mv /tmp/start-hadoop.sh ~/start-hadoop.sh && \ mv /tmp/init_zk.sh ~/init_zk.sh && \ mkdir -p /usr/local/hadoop2.7/dfs/data && \ mkdir -p /usr/local/hadoop2.7/dfs/name && \ mkdir -p /usr/local/zookeeper-3.4.10/datadir && \ mkdir -p /usr/local/zookeeper-3.4.10/log && \ mv /tmp/zoo.cfg $ZK_HOME/conf/zoo.cfg && \ mv /tmp/hbase-env.sh $HBASE_HOME/conf/hbase-env.sh && \ mv /tmp/hbase-site.xml $HBASE_HOME/conf/hbase-site.xml && \ mv /tmp/regionservers $HBASE_HOME/conf/regionservers && \ mv /tmp/hive-env.sh $HIVE_HOME/conf/hive-env.sh && \ mv /tmp/flume-env.sh $FLUME_HOME/conf/flume-env.sh RUN echo $JAVA_HOME #设置工作目录 WORKDIR /root #启动sshd服务 RUN /etc/init.d/ssh start #修改start-hadoop.sh权限为700 RUN chmod 700 start-hadoop.sh #修改init_zk.sh权限为700 RUN chmod 700 init_zk.sh #修改root密码 RUN echo "root:shaonaiyi" | chpasswd CMD ["/bin/bash"]
0x02 校验Flume前准备工作
1. 环境及资源准备
a. 安装Docker
请参考：D001.5 Docker入门（超级详细基础篇）的“0x01 Docker的安装”小节
b. 准备Flume的安装包，放于与Dockerfile同级目录下
c. 准备Flume的配置文件（放于config目录下）
cd /home/shaonaiyi/docker_bigdata/flume_sny_all/config
配置文件一：vi flume-env.sh
# Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file # distributed with this work for additional information # regarding copyright ownership. The ASF licenses this file # to you under the Apache License, Version 2.0 (the # "License"); you may not use this file except in compliance # with the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # If this file is placed at FLUME_CONF_DIR/flume-env.sh, it will be sourced # during Flume startup. # Enviroment variables can be set here. # export JAVA_HOME=/usr/lib/jvm/java-8-oracle export JAVA_HOME=/usr/local/jdk1.8.0_161 # Give Flume more memory and pre-allocate, enable remote monitoring via JMX # export JAVA_OPTS="-Xms100m -Xmx2000m -Dcom.sun.management.jmxremote" # Let Flume write raw event data and configuration information to its log files for debugging # purposes. Enabling these flags is not recommended in production, # as it may result in logging sensitive user information or encryption secrets. # export JAVA_OPTS="$JAVA_OPTS -Dorg.apache.flume.log.rawdata=true -Dorg.apache.flume.log.printconfig=true " # Note that the Flume conf directory is always included in the classpath. #FLUME_CLASSPATH=""
d. 修改环境变量配置文件（放于config目录下）
配置文件二：vi profile
添加内容：
export FLUME_HOME=/usr/local/apache-flume-1.8.0-bin export PATH=$PATH:$FLUME_HOME/bin
vi hbase-env.sh
注释掉此句：
export HBASE_CLASSPATH=/usr/local/hadoop-2.7.5/etc/hadoop
0x03 校验Flume是否安装成功
1. 修改生成容器脚本
a. 修改start_containers.sh文件（样本镜像名称成shaonaiyi/flume）
vi start_containers.sh
本人把里面的三个shaonaiyi/hive改为了shaonaiyi/flume
ps:当然，你可以新建一个新的网络，换ip，这里偷懒，用了旧的网络，只换了ip
2. 生成镜像
a. 删除之前的hbase集群容器（节省资源），如已删可省略此步
cd /home/shaonaiyi/docker_bigdata/hive_sny_all/config/
如果是复制的，此句可以省略：chmod 700 stop_containers.sh
./stop_containers.sh
b. 生成装好hadoop、spark、zookeeper、hbase、hive、flume的镜像（如果之前shaonaiyi/hbase未删除，则此次会快很多）
cd /home/shaonaiyi/docker_bigdata/flume_sny_all
docker build -t shaonaiyi/flume .
3. 生成容器
a. 生成容器（start_containers.sh如果没权限则给权限）：
config/start_containers.sh
b. 进入master容器
sh ~/master.sh
4. 启动Flume
a. 查看Flume版本号
flume-ng version
说明安装成功！
0xFF 总结
- 安装很简单，只需要知道步骤，不清楚请参考文章：D009 复制粘贴玩大数据之安装与配置Flume集群。
- Dockerfile常用指令，请参考文章：D004.1 Dockerfile例子详解及常用指令
- 越来越简单，信心有没有大涨了小伙计【斜眼.jpg】。
