阿里云提交spark集群运行-问答-阿里云开发者社区-阿里云

开发者社区> 问答> 正文

阿里云提交spark集群运行

游客izzln4qddaezk 2020-06-16 19:47:20 121

在阿里云上提交spark运行之后 image.png 出现如下报错,说是sparkdriver找不到后面跟的是该主机的私网ip请问这是怎么回事

分布式计算 Spark
分享到
取消 提交回答
全部回答(1)
  • python小菜菜
    2020-06-19 15:00:03

    特别坑的地方:

    1. 不能使用公网ip, 就算你把1001/65533给0.0.0.0/0 全部开放端口都没有用, 只能使用内网ip和主机名关联, 用内网ip启动spark

        在/etc/hosts文件:

    127.0.0.1 localhost

    你的内网ip 你的主机名

        对应的spark-env.sh文件

    export SPARK_LOCAL_IP=你的主机名

    export SPARK_MASTER_HOST=你的主机名

        对应的slaves文件

    你的主机名

    用公网ip就报错

    17/07/24 21:16:03 WARN Utils: Service 'sparkMaster' could not bind on port 7077. Attempting port 7078.

    17/07/24 21:16:03 WARN Utils: Service 'sparkMaster' could not bind on port 7078. Attempting port 7079.

    17/07/24 21:16:03 WARN Utils: Service 'sparkMaster' could not bind on port 7079. Attempting port 7080.

    17/07/24 21:16:03 WARN Utils: Service 'sparkMaster' could not bind on port 7080. Attempting port 7081.

    17/07/24 21:16:03 WARN Utils: Service 'sparkMaster' could not bind on port 7081. Attempting port 7082.

    17/07/24 21:16:03 WARN Utils: Service 'sparkMaster' could not bind on port 7082. Attempting port 7083.

    17/07/24 21:16:03 WARN Utils: Service 'sparkMaster' could not bind on port 7083. Attempting port 7084.

    17/07/24 21:16:03 WARN Utils: Service 'sparkMaster' could not bind on port 7084. Attempting port 7085.

    17/07/24 21:16:03 WARN Utils: Service 'sparkMaster' could not bind on port 7085. Attempting port 7086.

    17/07/24 21:16:03 WARN Utils: Service 'sparkMaster' could not bind on port 7086. Attempting port 7087.

    17/07/24 21:16:03 WARN Utils: Service 'sparkMaster' could not bind on port 7087. Attempting port 7088.

    17/07/24 21:16:03 WARN Utils: Service 'sparkMaster' could not bind on port 7088. Attempting port 7089.

    17/07/24 21:16:03 WARN Utils: Service 'sparkMaster' could not bind on port 7089. Attempting port 7090.

    17/07/24 21:16:03 WARN Utils: Service 'sparkMaster' could not bind on port 7090. Attempting port 7091.

    17/07/24 21:16:03 WARN Utils: Service 'sparkMaster' could not bind on port 7091. Attempting port 7092.

    17/07/24 21:16:03 WARN Utils: Service 'sparkMaster' could not bind on port 7092. Attempting port 7093.

    Exception in thread "main" java.net.BindException: Cannot assign requested address: Service 'sparkMaster' failed after 16 retries (starting from 7077)! Consider explicitly setting the appropriate port for the service 'sparkMaster' (for example spark.ui.port for SparkUI) to an available port or increasing spark.port.maxRetries.

        2. spark2.1.1版本的spark居然不能spark-shell到spark-2.2.0版本, 版本居然步兼容, 一言不合就报错:

    org.apache.spark.SparkException: Exception thrown in awaitResult

    我的服务器的spark的版本是2.2.0的

    我的电脑上的spark是2.1.1的

    坑了我一晚上

    0 0
大数据
使用钉钉扫一扫加入圈子
+ 订阅

大数据计算实践乐园,近距离学习前沿技术

推荐文章
相似问题