引言
在安装编译Hadoop的时候需要提前安装protobuf-2.5.0的，而且版本不要高了或者低了，这个是因为hadoop的rpc通信里面是直接使用protobuf-2.5.0作为数据格式交换的，因为遇到很多次，每次都从头安装，这次记录下来。
编译时候的提示
我的编译版本源码是hadoop-2.7.3,如果环境中没有安装protobuf-2.5.0,在做源码编译的时候有如下提示：
[INFO] [INFO] --- hadoop-maven-plugins:2.7.3:protoc (compile-protoc) @ hadoop-common --- [WARNING] [protoc, --version] failed: java.io.IOException: Cannot run program "protoc": error=2, 没有那个文件或目录 [ERROR] stdout: []
然后在我们的编译进程中是会中断的，具体表现如下：
[INFO] Executed tasks [INFO] [INFO] --- hadoop-maven-plugins:2.7.3:protoc (compile-protoc) @ hadoop-common --- [WARNING] [protoc, --version] failed: java.io.IOException: Cannot run program "protoc": error=2, 没有那个文件或目录 [ERROR] stdout: [] [INFO] ------------------------------------------------------------------------ [INFO] Reactor Summary: [INFO] [INFO] Apache Hadoop Main 2.7.3 ........................... SUCCESS [ 42.510 s] [INFO] Apache Hadoop Build Tools .......................... SUCCESS [ 47.049 s] [INFO] Apache Hadoop Project POM .......................... SUCCESS [ 24.040 s] [INFO] Apache Hadoop Annotations .......................... SUCCESS [01:22 min] [INFO] Apache Hadoop Assemblies ........................... SUCCESS [ 12.138 s] [INFO] Apache Hadoop Project Dist POM ..................... SUCCESS [ 19.555 s] [INFO] Apache Hadoop Maven Plugins ........................ SUCCESS [01:28 min] [INFO] Apache Hadoop MiniKDC .............................. SUCCESS [11:00 min] [INFO] Apache Hadoop Auth ................................. SUCCESS [03:57 min] [INFO] Apache Hadoop Auth Examples ........................ SUCCESS [ 14.794 s] [INFO] Apache Hadoop Common ............................... FAILURE [03:47 min] [INFO] Apache Hadoop NFS .................................. SKIPPED [INFO] Apache Hadoop KMS .................................. SKIPPED [INFO] Apache Hadoop Common Project ....................... SUCCESS [ 6.255 s] ......
当然，作为一名大数据开发工程师，除了知道有这回事，还要知道怎么来的。具体其实是在hadoop-common模块里面的pom.xml有定义,具体是在hadoop-2.7.3-src/hadoop-common-project/hadoop-common/pom.xml中，314-191行有对protoc的配置：
<plugins> <plugin> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-maven-plugins</artifactId> <executions> <execution> <id>version-info</id> <phase>generate-resources</phase> <goals> <goal>version-info</goal> </goals> <configuration> <source> <directory>${basedir}/src/main</directory> <includes> <include>java/**/*.java</include> <include>proto/**/*.proto</include> </includes> </source> </configuration> </execution> <execution> <id>compile-protoc</id> <phase>generate-sources</phase> <goals> <goal>protoc</goal> </goals> <configuration> <protocVersion>${protobuf.version}</protocVersion> <protocCommand>${protoc.path}</protocCommand> <imports> <param>${basedir}/src/main/proto</param> </imports> <source> <directory>${basedir}/src/main/proto</directory> <includes> <include>HAServiceProtocol.proto</include> <include>IpcConnectionContext.proto</include> <include>ProtocolInfo.proto</include> <include>RpcHeader.proto</include> <include>ZKFCProtocol.proto</include> <include>ProtobufRpcEngine.proto</include> <include>Security.proto</include> <include>GetUserMappingsProtocol.proto</include> <include>TraceAdmin.proto</include> <include>RefreshAuthorizationPolicyProtocol.proto</include> <include>RefreshUserMappingsProtocol.proto</include> <include>RefreshCallQueueProtocol.proto</include> <include>GenericRefreshProtocol.proto</include> </includes> </source> <output>${project.build.directory}/generated-sources/java</output> </configuration> </execution> <execution> <id>compile-test-protoc</id> <phase>generate-test-sources</phase> <goals> <goal>protoc</goal> </goals> <configuration> <protocVersion>${protobuf.version}</protocVersion> <protocCommand>${protoc.path}</protocCommand> <imports> <param>${basedir}/src/test/proto</param> </imports> <source> <directory>${basedir}/src/test/proto</directory> <includes> <include>test.proto</include> <include>test_rpc_service.proto</include> </includes> </source> <output>${project.build.directory}/generated-test-sources/java</output> </configuration> </execution> </executions> </plugin>
这一步的操作就是在maven编译的时候需要使用环境中的proto编译源码里面的proto文件，然后会生成rpc通信的源码。
proto编译
源码包是protobuf-2.5.0.tar.gz，需要翻墙下载，我用公司的网络是可以直接上谷歌的，不过百度一下也很多可以下载，我自己本机也有，如果实在不知道怎么找的话我单发也行
wget https://protobuf.googlecode.com/files/protobuf-2.5.0.tar.gz
源码上传到linux解压，编译
tar xvf protobuf-2.5.0.tar.gz cd protobuf-2.5.0 ./configure --prefix=/apps/svr/protobuf/ make make install
验证
我把命令配置到我的环境变量中
export PATH=/apps/svr/protobuf/:$PATH
验证
[root@zhu-91-134 yum.repos.d]# protoc --version libprotoc 2.5.0
单独验证的话不需要所有都重头再来，可以单独验证common模块
cd hadoop-common-project/hadoop-common && mvn package -Pdist -DskipTests -Dtar
信息比较多，但是看到关键的成功就行
...... [INFO] Building jar: /root/hadoop-2.7.3-src/hadoop-common-project/hadoop-common/target/hadoop-common-2.7.3-javadoc.jar [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 01:33 min [INFO] Finished at: 2022-05-03T23:18:15+08:0
错误记录
error: cannot install `libprotoc.la' to a directory not ending in /usr/local/lib
这个错误其实是在我前面一次有配置过，需要执行一下make clean清理掉之前的配置
