- 首先安装java，可以使用brew install openjdk8，建议不选择高版本，很容易导致版本错误
- brew install hadoop 安装hadoop，因为hive的数据存储和计算是基于hadoop框架，需要先安装hadoop
- 下面警告信息是针对brew升级macos后的问题
We do not provide support for this pre-release version. You will encounter build failures with some formulae. Please create pull requests instead of asking for help on Homebrew's GitHub,Twitter or any other official channels. You are responsible for resolvingany issues you experience while you are running thispre-release version.
- 在目录下的sbin中启动dfs，报错
Starting namenodes on [bogon] bogon: ssh: Could not resolve hostname bogon: nodename nor servname provided, or not known Starting datanodes localhost: ssh: connect to host localhost port 22: Connection refused Starting secondary namenodes [bogon] bogon: ssh: Could not resolve hostname bogon: nodename nor servname provided, or not known 2022-12-2918:06:39,462 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
- 使用ssh localhost，如果出现22端口拒绝访问，可以在mac打开系统设置，将远程登录打开，用户选择所有用户
- 然后进入到/Users/mac/.ssh目录下，执行ssh-keygen，生成公钥和密钥，再使用cat id_rsa.pub >> authorized_keys将公钥写入，重新执行后，报错信息会发生改变
Starting namenodes on [bogon] bogon: ssh: Could not resolve hostname bogon: nodename nor servname provided, or not known Starting datanodes localhost: WARNING: /usr/local/Cellar/hadoop/3.3.4/libexec/logs does not exist. Creating. Starting secondary namenodes [bogon] bogon: ssh: Could not resolve hostname bogon: nodename nor servname provided, or not known
- 从网上按照教程配置了hadoop的配置，不过应该和这个没有太大关系，
cd /usr/local/Cellar/hadoop/3.3.1/libexec/etc/hadoop - 配置文件如下：
<!--core-site--><configuration><property><name>fs.defaultFS</name><value>hdfs://localhost:8020</value></property><!--用来指定hadoop运行时产生文件的存放目录 自己创建--><property><name>hadoop.tmp.dir</name><value>file:/usr/local/Cellar/hadoop/tmp</value></property></configuration>
<!--hdfs-site--><configuration><property><name>dfs.replication</name><value>1</value></property><!--不是root用户也可以写文件到hdfs--><property><name>dfs.permissions</name><value>false</value><!--关闭防火墙--></property><!--把路径换成本地的name坐在位置--><property><name>dfs.namenode.name.dir</name><value>file:/usr/local/Cellar/hadoop/tmp/dfs/name</value></property><!--在本地新建一个存放hadoop数据的文件夹，然后将路径在这里配置一下--><property><name>dfs.datanode.data.dir</name><value>file:/usr/local/Cellar/hadoop/tmp/dfs/data</value></property></configuration>
<!--mapred-site--><configuration><property><!--指定mapreduce运行在yarn上--><name>mapreduce.framework.name</name><value>yarn</value></property><property><name>mapred.job.tracker</name><value>localhost:9010</value></property><!-- 新添加 --><!-- 下面的路径就是你hadoop distribution directory --><property><name>yarn.app.mapreduce.am.env</name><value>HADOOP_MAPRED_HOME=/usr/local/Cellar/hadoop/3.3.1/libexec</value></property><property><name>mapreduce.map.env</name><value>HADOOP_MAPRED_HOME=/usr/local/Cellar/hadoop/3.3.1/libexec</value></property><property><name>mapreduce.reduce.env</name><value>HADOOP_MAPRED_HOME=/usr/local/Cellar/hadoop/3.3.1/libexec</value></property></configuration>
<!--yarn-site--><configuration><property><name>yarn.nodemanager.aux-services</name><value>mapreduce_shuffle</value></property><property><name>yarn.resourcemanager.address</name><value>localhost:9000</value></property><property><name>yarn.scheduler.capacity.maximum-am-resource-percent</name><value>100</value></property></configuration>
- 重新启动后，报错就变成了
bogon: ssh: Could not resolve hostname bogon: nodename nor servname provided, or not known - 此刻需要到sbin同目录的bin目录下格式化namenode
./hdfs namenode -format，可以成功启动hadoop的dfshttp://localhost:9870/dfshealth.html#tab-overview - 而且需要注意./start-dfs 和 start-dfs.sh是不同的结果，如果不加./，会报错JAVA_HOME @@HOMEBREW_JAVA@@ does not exist；怀疑是没有配置hadoop全局变量的问题
- 再使用./start-yarn就可以启动yranhttp://localhost:8088/cluster/cluster
- brew install hive 使用brew安装
- 直接到hive的bin目录下启动hive，报错如下
SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/usr/local/Cellar/hive/3.1.3/libexec/lib/log4j-slf4j-impl-2.17.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/usr/local/Cellar/hadoop/3.3.4/libexec/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory] Hive Session ID = dac69e20-6f04-489b-adbb-f0458e86fa5d Exception in thread "main" java.lang.ClassCastException: class jdk.internal.loader.ClassLoaders$AppClassLoader cannot be cast to class java.net.URLClassLoader (jdk.internal.loader.ClassLoaders$AppClassLoader and java.net.URLClassLoader are in module java.base of loader 'bootstrap') at org.apache.hadoop.hive.ql.session.SessionState.<init>(SessionState.java:413) at org.apache.hadoop.hive.ql.session.SessionState.<init>(SessionState.java:389) at org.apache.hadoop.hive.cli.CliSessionState.<init>(CliSessionState.java:60) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:705) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:683) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.apache.hadoop.util.RunJar.run(RunJar.java:323) at org.apache.hadoop.util.RunJar.main(RunJar.java:236)
- 这里是因为最开始我hadoop配置的是openjdk11，将其换成openjdk8即可，然后就可以重新启动，此刻报错如下
FAILED: HiveException java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
- 网上说是hive的元数据未开启
hive --service metastore &；未解决问题，我又删除metastore_db，执行schematool -initSchema -dbType derby却没有发现问题解决 - 隔了几条，我重新启动机器，发现问题已经解决，😓，全局变量配置好像没有生效，只能在bin目录下执行hive
