Spark安装教程
1. 检查jdk版本
检查jdk是否安装并且版本是否为1.8
javac -version
# javac 1.8.0_171
2. 获取Spark版本安装资源
本文以Spark3.1.2为例，资源详见文章上方。
3.环境变量
vim /etc/profile
export SPARK_HOME=/opt/software/spark-3.1.2
export PATH=$SPARK_HOME/bin:$PATH
4.配置文件
cd $SPARK_HOME/conf
mv spark-env.sh.template spark-env.sh
vim spark-env.sh
------------------------------------------------
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop/
export YARN_CONF_DIR=$HADOOP_HOME/etc/hadoop/
------------------------------------------------
cd $HADOOP_HOME/etc/hadoop
vim yarn-site.xml
------------------------------------------------
# 添加两个property
<property>
<name>yarn.nodemanager.pmem-check-enabled</name>
<value>false</value>
</property>
<property>
<name>yarn.nodemanager.vmem-check-enabled</name>
<value>false</value>
</property>
------------------------------------------------
5. 重启Hadoop集群(使配置生效)
stop-all.sh
start-all.sh
6. 启动Spark集群
/opt/software/spark-3.1.2/sbin/start-all.sh
6.1 查看Spark服务
jps -ml
----------------------------------------------------------------
1649 org.apache.spark.deploy.master.Master --host single --port 7077 --webui-port 8080
1707 org.apache.spark.deploy.worker.Worker --webui-port 8081 spark://single:7077
----------------------------------------------------------------
6.2 访问Spark WEB UI
http://single01:8080/
7. 启动 Spark-Shell 测试 Scala 交互式环境
spark-shell --master spark://single:7077
----------------------------------------------------------------
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
Spark context Web UI available at http://single:4040
Spark context available as 'sc' (master = spark://single:7077, app id = app-20240315091621-0000).
Spark session available as 'spark'.
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 3.1.2
/_/
Using Scala version 2.12.10 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_171)
Type in expressions to have them evaluated.
Type :help for more information.
scala> _
----------------------------------------------------------------
8. 测试Spark On Yarn
spark-shell --master yarn
----------------------------------------------------------------
Spark context Web UI available at http://single:4040
Spark context available as 'sc' (master = yarn, app id = application_1710465965758_0001).
Spark session available as 'spark'.
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 3.1.2
/_/
Using Scala version 2.12.10 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_171)
Type in expressions to have them evaluated.
Type :help for more information.
scala>
----------------------------------------------------------------
9.关闭Spark集群
/opt/software/spark-3.1.2/sbin/stop-all.sh
