hadoop简单介绍和简单实践视频教程 慕课网: https://www.imooc.com/video/7642
预备知识:
liunx基本操作
java开发基础知识
google的大数据技术:MapReduce,BigTables,GFS
革命性变化:
- 成本降低,能用pc机就不用大型机和高端存储。
- 软件容错,硬件故障视为常态,通过软件保证可靠性。
- 简化并行分布式计算,无需控制节点同步和数据交换。
google分布式计算的开源实现 Hadoop ,由apache开发。
解决两个问题,分布式存储和分布式数据处理。
对应HDFS(分布式文件系统)和MapReduce(映射规约)
![img_48f5b8d11be7602455ae108741b1e08b.png](https://yqfile.alicdn.com/img_48f5b8d11be7602455ae108741b1e08b.png?x-oss-process=image/resize,w_1400/format,webp)
Hdoop优势:
- 高扩展
- 低成本
-
成熟的生态圈
Hadoop 生态系统
Hive 小蜜蜂,牵引
![img_ba4fa75374492bdc8fd8845ec1aef404.png](https://yqfile.alicdn.com/img_ba4fa75374492bdc8fd8845ec1aef404.png?x-oss-process=image/resize,w_1400/format,webp)
Hbase
![img_5c6f9722e926ae6da1f8e7ae1a9619da.png](https://yqfile.alicdn.com/img_5c6f9722e926ae6da1f8e7ae1a9619da.png?x-oss-process=image/resize,w_1400/format,webp)
![img_8bab5d0476d475af8c28f1f34099c15f.png](https://yqfile.alicdn.com/img_8bab5d0476d475af8c28f1f34099c15f.png?x-oss-process=image/resize,w_1400/format,webp)
zookeeper
![img_7fd88dbe69488f7e7789ad6a8f9cbbf1.png](https://yqfile.alicdn.com/img_7fd88dbe69488f7e7789ad6a8f9cbbf1.png?x-oss-process=image/resize,w_1400/format,webp)
Hadoop安装
- 准备linux
租用云主机,比如阿里云。 - jdk环境
centos下有所不同
1、下载 : yum install java-1.7.0-openjdk
2、配置环境变量 vim /etc/profile
export JAVA_HOME=/jdk路径
export JRE_HOME=$JAVA_HOME/jre
export CLASSPATH=$JAVA_HOME/lib:$JRE_HOME/lib:$CLASSPATH
export PATH=$JAVA_HOME/lib:$JRE_HOME/lib:$PATH
3、让配置生效source /etc/profile
4、输入 java -version 进行测试
vim /etc/profile
配置环境变量
export JAVA_HOME=/jdk路径
export JRE_HOME=$JAVA_HOME/jre
export CLASSPATH=$JAVA_HOME/lib:$JRE_HOME/lib:$CLASSPATH
export PATH=$JAVA_HOME/lib:$JRE_HOME/lib:$PATH
让配置生效
source /etc/profile
- 配置Hadoop
wegt http://mirror.bit.edu.cn/apache/hadoop/common/hadoop-1.2.1/hadoop-1.2.1.tar.gz
![img_e317f3cdae2550ab757fc8ded9ce4156.png](https://yqfile.alicdn.com/img_e317f3cdae2550ab757fc8ded9ce4156.png?x-oss-process=image/resize,w_1400/format,webp)
修改四个配置文件
![img_940793c2b0bba6aa157b243ea1f5c7c0.png](https://yqfile.alicdn.com/img_940793c2b0bba6aa157b243ea1f5c7c0.png?x-oss-process=image/resize,w_1400/format,webp)
<!-- core-site.xml -->
<property>
<name>hadoop.tmp.dir</name>
<value>/hadoop</value>
</property>
<property>
<name>dfs.name.dir</name>
<value>/hadoop/name</value>
</property>
<property>
<name>fs.default.name</name>
<value>hdfs://localhost:9000</value>
</property>
<configuration>
<property>
<name>mapred.job.tracker</name>
<value>localhost:9001</value>
</property>
</configuration>
hdfs-site.xml
<property>
<name>dfs.data.dir</name>
<value>/hadoop/data</value>
</property>
HDFS
![img_7e0a7c3331befcf1e884e5d4274e78b4.png](https://yqfile.alicdn.com/img_7e0a7c3331befcf1e884e5d4274e78b4.png?x-oss-process=image/resize,w_1400/format,webp)
![img_0924a7943320c35a16846addf1d2b9c3.png](https://yqfile.alicdn.com/img_0924a7943320c35a16846addf1d2b9c3.png?x-oss-process=image/resize,w_1400/format,webp)
![img_5b1bdb1994b4c0e804834a689070bd3f.png](https://yqfile.alicdn.com/img_5b1bdb1994b4c0e804834a689070bd3f.png?x-oss-process=image/resize,w_1400/format,webp)
namenode
![img_efdbdd4104503a98852ffd2677714e2d.png](https://yqfile.alicdn.com/img_efdbdd4104503a98852ffd2677714e2d.png?x-oss-process=image/resize,w_1400/format,webp)
datenode
文件读取流程:
客户端发送请求给NameNode,NameNode告诉客户端数据在哪些DataNode,然后客户端找DataNode读取数据。
![img_5fdb507be9d94a720b795d64f9965149.png](https://yqfile.alicdn.com/img_5fdb507be9d94a720b795d64f9965149.png?x-oss-process=image/resize,w_1400/format,webp)
文件写入流程:
客户端先分拆分成块64M大小,然后客户端通知NameNode存储,NameNode找几个空闲的DataNode返回客户端,客户端写数据个某个DataNode,然后DataNode进行复制,复制完毕更新NameNode记录。
![img_572d130dd865a3de6961860112b46dde.png](https://yqfile.alicdn.com/img_572d130dd865a3de6961860112b46dde.png?x-oss-process=image/resize,w_1400/format,webp)
HDFS 特点
- 数据冗余,硬件容错
- 流式的数据访问(一次写入,多次读取,顺序写入)
- 适合存储大文件
HDFS使用
shell命令操作
MapReduce 原理
分而治之思想
map 影视
reduce 归约
![img_e1dfdbb758bc0a73541c89c36e7c3eda.png](https://yqfile.alicdn.com/img_e1dfdbb758bc0a73541c89c36e7c3eda.png?x-oss-process=image/resize,w_1400/format,webp)
Job&Task
JobTracker
TaskTracker
![img_2378d82dccea9ea6535034f054ead9c6.png](https://yqfile.alicdn.com/img_2378d82dccea9ea6535034f054ead9c6.png?x-oss-process=image/resize,w_1400/format,webp)
image.png
![img_59245caf95d0daef877f6f5c165646b8.png](https://yqfile.alicdn.com/img_59245caf95d0daef877f6f5c165646b8.png?x-oss-process=image/resize,w_1400/format,webp)
![img_c6a66587d6d51bcf4fe9e8cb85675957.png](https://yqfile.alicdn.com/img_c6a66587d6d51bcf4fe9e8cb85675957.png?x-oss-process=image/resize,w_1400/format,webp)
![img_792715da7b97b6652c63700070d17858.png](https://yqfile.alicdn.com/img_792715da7b97b6652c63700070d17858.png?x-oss-process=image/resize,w_1400/format,webp)
![img_87f24963fa50d1b1b4b3f3f985dbfa2b.png](https://yqfile.alicdn.com/img_87f24963fa50d1b1b4b3f3f985dbfa2b.png?x-oss-process=image/resize,w_1400/format,webp)
MapReduce容错机制:
- 重复执行
- 推测执行
helloword 入门 单词计数程序
![img_6c395952da754460b6180c174257ff96.png](https://yqfile.alicdn.com/img_6c395952da754460b6180c174257ff96.png?x-oss-process=image/resize,w_1400/format,webp)
![img_e4b759b7aabf33e0b3bafe075369aa55.png](https://yqfile.alicdn.com/img_e4b759b7aabf33e0b3bafe075369aa55.png?x-oss-process=image/resize,w_1400/format,webp)
![img_0c0ca97c5775d2b994b2ab23b0606117.png](https://yqfile.alicdn.com/img_0c0ca97c5775d2b994b2ab23b0606117.png?x-oss-process=image/resize,w_1400/format,webp)
![img_a39158936222439da80af9e23a1dc174.png](https://yqfile.alicdn.com/img_a39158936222439da80af9e23a1dc174.png?x-oss-process=image/resize,w_1400/format,webp)
![img_143bab56961deec51a12ad6bac137452.png](https://yqfile.alicdn.com/img_143bab56961deec51a12ad6bac137452.png?x-oss-process=image/resize,w_1400/format,webp)
![img_843869135146c1b1528af43c8be3c53a.png](https://yqfile.alicdn.com/img_843869135146c1b1528af43c8be3c53a.png?x-oss-process=image/resize,w_1400/format,webp)
![img_e49e22b83ccab1277b8dbb61b94cdc16.png](https://yqfile.alicdn.com/img_e49e22b83ccab1277b8dbb61b94cdc16.png?x-oss-process=image/resize,w_1400/format,webp)
![img_855c250d4351610f7556b2b6d0b75636.png](https://yqfile.alicdn.com/img_855c250d4351610f7556b2b6d0b75636.png?x-oss-process=image/resize,w_1400/format,webp)