构建基于hadoop的spark安装包
上一篇说了spark与其它组件兼容的版本，这里具体说下如何构建基于hadoop的spark安装包。首先我们需要有spark源码，上一篇已经交给大家如何使用git下载。当然我这里提供了百度网盘链接
链接：http://pan.baidu.com/s/1gfMpTqb 密码：c6dc
默认情况下，spark的执行不需要hadoop，也就是说没有hadoop集群下，spark集群可以设置，启动，使用。。尽管如此，如果你使用spark如Yarn的执行模式，或则访问hdfs创建rdd，它将会依赖hadoop。如果是这种情况，你的spark安装包必须兼容你所使用的hadoop集群的安装包
如果你使用的是spark2.3.0对应的hadoop默认为2.6.在假如使用的是spark1.2.0对应的是hadoop2.4.
这里需要注意的是：有些hadoop版本，是有小版本的，比如hadoop2.6.5，hadoop2.7有hadoop2.7.1，hadoop2.7.3.对于hadoop版本的+或则-的小版本之间，它们与spark大多都是兼容可以正常运行的。
spark构建运行开发者指定hadoop版本，你可以直接使用maven命令或则 make-distribution.sh脚本生成安装包。例如构建spark2.3.0，hadoop2.6
maven profile 使用-P选项, 如:
$SPARK_SRC/make-distribution.sh --tgz -Pyarn -Phadoop-2.6 -Dhadoop.version=2.6 -Phive
spark2.3.0支持2.6，2.7
如何自定义hadoop版本
基本的你需要在pom文件中添加profile指定hadoop版本
假如你想构建 Hadoop 2.6.5，按照下面步骤
第一步：
在 $SPARK_SRC/pom.xml中添加maven profile hadoop-2.6.5 在<profiles> 部分
<profile> <id>hadoop-2.6.5</id> <properties> <hadoop.version>2.6.5</hadoop.version> <protobuf.version>2.5.0</protobuf.version> <jets3t.version>0.9.0</jets3t.version> <commons.math3.version>3.1.1</commons.math3.version> <avro.mapred.classifier>hadoop2</avro.mapred.classifier> </properties> </profile>
protobuf, jets3t, commons.math3 和avro.mapred.classifier 版本都是来自Apache Hadoop 2.5.2 二进制分布式libs. 你需要修改要构建的hadoop的对应版本。
对于这个avro.mapred.classifier，大家可以找找，不过在spark2.3.0 pom文件中也是有的
https://github.com/apache/spark/blob/master/pom.xml
上面是比较复杂的，我们也可以使用最简单的方式：
<profile> <id>hadoop-2.6.5</id> </profile>
第二步：构建执行
使用make-distribution.sh构建
$SPARK_SRC/make-distribution.sh --tgz -Pyarn -Phadoop-2.6.5 -Dhadoop.version=2.6.5 -Phive
使用maven命令构建
mvn -Pyarn -Dhadoop.version=2.6.5 -Dyarn.version=2.6.5 -DskipTests clean package
构建成功后，可以找到spark-2.*.*-bin-2.6.5.tgz
注意：
这种方式建议使用在hadoop小版本，对于hadoop主版本即使构建成功，也可能在生产中遇到一些问题。
参考：
https://etushar89.wordpress.com/2014/12/28/building-apache-spark-against-specific-hadoop-version/
构建基于hadoop的spark安装包实践及遇到问题总结
spark编译时间还是比较长的，可能要一两个小时，而且有时候卡住那不动。
在编译的过程中，有编译失败，也有编译成功的。总之两个条件：
1.跟版本有关系
版本不对，可能编译失败
2.跟网速有关系
网速不好，有些下载失败，导致编译失败。
使用hadoop2.6.5
spark源码下载：
链接：http://pan.baidu.com/s/1gfMpTqb 密码：c6dc
$SPARK_SRC/make-distribution.sh --tgz -Pyarn -Phadoop-2.6.5 -Dhadoop.version=2.6.5 -Phive
这里使用的是spark2.3.0，hadoop版本为2.6.5，最后编译失败。报错如下
[INFO] [INFO] ------------------------------------------------------------------------ [INFO] ------------------------------------------------------------------------ [INFO] Reactor Summary: [INFO] [INFO] Skipping Spark Integration for Kafka 0.10 Assembly [INFO] This project has been banned from the build due to previous failures. [INFO] ------------------------------------------------------------------------ [INFO] Spark Project Parent POM ........................... SUCCESS [08:36 min] [INFO] Spark Project Tags ................................. SUCCESS [04:56 min] [INFO] Spark Project Sketch ............................... SUCCESS [ 10.413 s] [INFO] Spark Project Local DB ............................. SUCCESS [01:15 min] [INFO] Spark Project Networking ........................... SUCCESS [ 36.812 s] [INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [ 11.964 s] [INFO] Spark Project Unsafe ............................... SUCCESS [ 34.261 s] [INFO] Spark Project Launcher ............................. SUCCESS [02:40 min] [INFO] Spark Project Core ................................. SUCCESS [12:34 min] [INFO] Spark Project ML Local Library ..................... SUCCESS [02:19 min] [INFO] Spark Project GraphX ............................... SUCCESS [01:29 min] [INFO] Spark Project Streaming ............................ SUCCESS [01:52 min] [INFO] Spark Project Catalyst ............................. SUCCESS [05:25 min] [INFO] Spark Project SQL .................................. FAILURE [10:13 min] [INFO] Spark Project ML Library ........................... SKIPPED [INFO] Spark Project Tools ................................ SUCCESS [ 17.955 s] [INFO] Spark Project Hive ................................. SKIPPED [INFO] Spark Project REPL ................................. SKIPPED [INFO] Spark Project YARN Shuffle Service ................. SUCCESS [ 19.981 s] [INFO] Spark Project YARN ................................. SUCCESS [01:14 min] [INFO] Spark Project Assembly ............................. SKIPPED [INFO] Spark Integration for Kafka 0.10 ................... SUCCESS [01:33 min] [INFO] Kafka 0.10 Source for Structured Streaming ......... SKIPPED [INFO] Spark Project Examples ............................. SKIPPED [INFO] Spark Integration for Kafka 0.10 Assembly .......... SKIPPED [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 56:26 min [INFO] Finished at: 2017-11-08T10:44:58+08:00 [INFO] Final Memory: 65M/296M [INFO] ------------------------------------------------------------------------ [WARNING] The requested profile "hadoop-2.6.5" could not be activated because it does not exist. [ERROR] Failed to execute goal net.alchim31.maven:scala-maven-plugin:3.2.2:testCompile (scala-test-compile-first) on project spark-sql_2.11: Execution scala-test-compile-first of goal net.alchim31.maven:scala-maven-plugin:3.2.2:testCompile failed. CompileFailed -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginExecutionException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn <goals> -rf :spark-sql_2.11
有一个错误和警告是比较关键的
[WARNING] The requested profile "hadoop-2.6.5" could not be activated because it does not exist.
[ERROR] Failed to execute goal net.alchim31.maven:scala-maven-plugin:3.2.2:testCompile (scala-test-compile-first) on project spark-sql_2.11:
上面是profile中是没有hadoop-2.6.5，因此我们需要增加profile
<profile> <id>hadoop-2.6.5</id> </profile>
第二个问题不能执行，这个可能就跟网速有关系。多次执行仍然失败，也可能跟版本有关系
更换hadoop2.7.1
上面失败，接着我们尝试hadoop2.7.1
<profile> <id>hadoop-2.6.5</id> </profile>
编译成功
但是报错如下
+ TARDIR_NAME='spark-[WARNING] The requested profile "hadoop-2.7.1" could not be activated because it does not exist.-bin-[WARNING] The requested profile "hadoop-2.7.1" could not be activated because it does not exist.' + TARDIR='/home/aboutyun/spark/spark-[WARNING] The requested profile "hadoop-2.7.1" could not be activated because it does not exist.-bin-[WARNING] The requested profile "hadoop-2.7.1" could not be activated because it does not exist.' + rm -rf '/home/aboutyun/spark/spark-[WARNING] The requested profile "hadoop-2.7.1" could not be activated because it does not exist.-bin-[WARNING] The requested profile "hadoop-2.7.1" could not be activated because it does not exist.' + cp -r /home/aboutyun/spark/dist '/home/aboutyun/spark/spark-[WARNING] The requested profile "hadoop-2.7.1" could not be activated because it does not exist.-bin-[WARNING] The requested profile "hadoop-2.7.1" could not be activated because it does not exist.' + tar czf 'spark-[WARNING] The requested profile "hadoop-2.7.1" could not be activated because it does not exist.-bin-[WARNING] The requested profile "hadoop-2.7.1" could not be activated because it does not exist..tgz' -C /home/aboutyun/spark 'spark-[WARNING] The requested profile "hadoop-2.7.1" could not be activated because it does not exist.-bin-[WARNING] The requested profile "hadoop-2.7.1" could not be activated because it does not exist.' + rm -rf '/home/aboutyun/spark/spark-[WARNING] The requested profile "hadoop-2.7.1" could not be activated because it does not exist.-bin-[WARNING] The requested profile "hadoop-2.7.1" could not be activated because it does not exist.'
[WARNING] The requested profile "hadoop-2.7.1" could not be activated because it does not exist.这句很关键，意思是虽然编译成功，但是不能使用。在后面将截图给大家看。
所以在pom.xml文件中添加如下属性
[mw_shl_code=bash,true]<profile> <id>hadoop-2.7.1</id> </profile>
编译成功
下面我们通过winscp查看第一个为未添加profile，第二个添加后，编译成功。
编译包下载：链接：http://pan.baidu.com/s/1nv7QAwT 密码：oo2r
