开发者社区> 问答> 正文

安装后无法执行pyspark

社区小助手 2018-12-12 13:27:58 791

我已手动复制spark-2.4.0-bin-hadoop2.7.tgz并提取。然后我进入.bash_profile如下:

export SPARK_HOME = / Users / suman / Pyspark / spark-2.4.0-bin-hadoop2.7 export PATH = $ SPARK_HOME / bin:$ PATH

我确定我已安装jdk.Response在下面:

ABCDEFGH:bin suman $ java -version java version“11”2018-09-25 Java(TM)SE运行时环境18.9(版本11 + 28)Java HotSpot(TM)64位服务器VM 18.9(版本11 + 28,混合模式)

错误如下:

ABCDEFGH:bin suman $ pyspark

Python 3.7.0 (default, Jun 28 2018, 07:39:16)
[Clang 4.0.1 (tags/RELEASE_401/final)] :: Anaconda, Inc. on darwin
Type "help", "copyright", "credits" or "license" for more information.
Exception in thread "main" java.lang.ExceptionInInitializerError

at org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:80)
at org.apache.hadoop.security.SecurityUtil.getAuthenticationMethod(SecurityUtil.java:611)
at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:273)
at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:261)
at org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:791)
at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:761)
at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:634)
at org.apache.spark.util.Utils

$$ anonfun$getCurrentUserName$1.apply(Utils.scala:2422) at org.apache.spark.util.Utils $$

anonfun$getCurrentUserName$1.apply(Utils.scala:2422)

at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.util.Utils$.getCurrentUserName(Utils.scala:2422)
at org.apache.spark.SecurityManager.<init>(SecurityManager.scala:79)
at org.apache.spark.deploy.SparkSubmit.secMgr$lzycompute$1(SparkSubmit.scala:359)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit

$$ secMgr$1(SparkSubmit.scala:359) at org.apache.spark.deploy.SparkSubmit $$

anonfun$prepareSubmitEnvironment$7.apply(SparkSubmit.scala:367)

at org.apache.spark.deploy.SparkSubmit

$$ anonfun$prepareSubmitEnvironment$7.apply(SparkSubmit.scala:367) at scala.Option.map(Option.scala:146) at org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment(SparkSubmit.scala:366) at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:143) at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86) at org.apache.spark.deploy.SparkSubmit $$

anon$2.doSubmit(SparkSubmit.scala:924)

at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:933)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

Caused by: java.lang.StringIndexOutOfBoundsException: begin 0, end 3, length 2

at java.base/java.lang.String.checkBoundsBeginEnd(String.java:3319)
at java.base/java.lang.String.substring(String.java:1874)
at org.apache.hadoop.util.Shell.<clinit>(Shell.java:52)
... 23 more

Traceback (most recent call last):
File "/Users/suman/Pyspark/spark-2.4.0-bin-hadoop2.7/python/pyspark/shell.py", line 38, in

SparkContext._ensure_initialized()

File "/Users/suman/Pyspark/spark-2.4.0-bin-hadoop2.7/python/pyspark/context.py", line 298, in _ensure_initialized

SparkContext._gateway = gateway or launch_gateway(conf)

File "/Users/suman/Pyspark/spark-2.4.0-bin-hadoop2.7/python/pyspark/java_gateway.py", line 94, in launch_gateway

raise Exception("Java gateway process exited before sending its port number")

Exception: Java gateway process exited before sending its port number

分布式计算 Java Apache Spark Python
分享到
取消 提交回答
全部回答(1)
  • 社区小助手
    2019-07-17 23:20:07

    我将此路径添加到.bash_profile中,它对我有用:

    export JAVA_HOME="/Library/Internet Plug-Ins/JavaAppletPlugin.plugin/Contents/Home -v 1.8"

    0 0
+ 订阅

大数据计算实践乐园,近距离学习前沿技术

推荐文章
相似问题