slaves配置（三台一样）
vi /opt/middleware/spark-2.1.0-bin-hadoop2.7/conf/slaves
内容如下
slave1 slave2 slave3
6、启动
#在master上执行 /opt/middleware/spark-2.4.0-bin-hadoop2.7/sbin/start-all.sh
7、页面访问
spark首页监控
截图是在两个slave情况下的，不影响阅读
常见问题汇总
问题1
NoClassDefFoundError: org/apache/spark/internal/Logging
解决方案:
sprintbootjar包排除log相关
<dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-web</artifactId> <version>${springboot.version}</version> <exclusions> <exclusion> <groupId>ch.qos.logback</groupId> <artifactId>logback-classic</artifactId> </exclusion> </exclusions> </dependency>
问题2
NoSuchMethodError: javax.validation.BootstrapConfiguration.getClockProviderClassName()Ljava/lang/String;
解决方案:
删除spark jars下的validation-api-1.x.Final.jar,和springboot2.0冲突
问题3
cannot assign instance of java.lang.invoke.SerializedLambda to field
解决方案:
org.apache.spark.api.java.JavaRDD
原因未知，设置setJars()不生效，只好将Lambda表达式换成function
问题4
NoSuchMethodError: com.google.gson.GsonBuilder.setLenient()Lcom/google/gson/GsonBuilder
解决方案:
gson冲突，删除spark自带的gson2.2.4
问题5
ClassNotFoundException: com.mongodb.spark.rdd.partitioner.MongoPartition
解决方案:
将mongo-spark-connector_2.11-2.1.5.jar拷贝到jars下
问题6
ClassNotFoundException: com.mongodb.client.model.Collation
解决方案:
将mongo-java-driver-3.10.2.jar拷贝到jars下（加载不到自己app所属的jar包？？？？）
问题7
ClassCastException: cannot assign instance of scala.collection.immutable.List$SerializationProxy to field org.apache.spark.rdd.RDD.org$apache$spark$rdd$RDD
问题8
19/06/04 15:46:40 WARN Utils: Service 'Driver' could not bind on port 0. Attempting port 1. 19/06/04 15:46:40 WARN Utils: Service 'Driver' could not bind on port 0. Attempting port 1. 19/06/04 15:46:40 WARN Utils: Service 'Driver' could not bind on port 0. Attempting port 1. 19/06/04 15:46:40 WARN Utils: Service 'Driver' could not bind on port 0. Attempting port 1. Exception in thread "main" java.net.BindException: 无法指定被请求的地址: Service 'Driver' failed after 16 retries (starting from 0)! Consider explicitly setting the appropriate port for the service 'Driver' (for example spark.ui.port for SparkUI) to an available port or increasing spark.port.maxRetries. at sun.nio.ch.Net.bind0(Native Method) at sun.nio.ch.Net.bind(Net.java:433) at sun.nio.ch.Net.bind(Net.java:425) at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223) at io.netty.channel.socket.nio.NioServerSocketChannel.doBind(NioServerSocketChannel.java:127) at io.netty.channel.AbstractChannel$AbstractUnsafe.bind(AbstractChannel.java:501) at io.netty.channel.DefaultChannelPipeline$HeadContext.bind(DefaultChannelPipeline.java:1218) at io.netty.channel.AbstractChannelHandlerContext.invokeBind(AbstractChannelHandlerContext.java:506) at io.netty.channel.AbstractChannelHandlerContext.bind(AbstractChannelHandlerContext.java:491) at io.netty.channel.DefaultChannelPipeline.bind(DefaultChannelPipeline.java:965) at io.netty.channel.AbstractChannel.bind(AbstractChannel.java:210) at io.netty.bootstrap.AbstractBootstrap$2.run(AbstractBootstrap.java:353) at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:408) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:455) at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:140) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) at java.lang.Thread.run(Thread.java:748)
解决方案 spark-env.sh中不需要配置SPARK_LOCAL_IP
问题9
check your cluster UI to ensure that workers are registered and have sufficient resources
原因分析： SPARK_WORKER_MEMORY配置的是2g 而spark.executor.cores为3，spark.executor.memory -> 1g，超出worker配置的2g内存，所以获取不到资源 解决方案 spark-evn.sh中配置 export SPARK_WORKER_MEMORY=3g SparkLauncher提交任务时配置 spark.executor.cores=2 spark.executor.memory=1g
