AI 助理
备案控制台
开发者社区 大数据 文章 正文

pyspark提交作业到yarn集群一些问题总结

2022-07-26 1668
版权
版权声明：
本文内容由阿里云实名注册用户自发贡献，版权归原作者所有，阿里云开发者社区不拥有其著作权，亦不承担相应法律责任。具体规则请查看《 阿里云开发者社区用户服务协议》和 《阿里云开发者社区知识产权保护指引》。如果您发现本社区中有涉嫌抄袭的内容，填写 侵权投诉表单进行举报，一经查实，本社区将立刻删除涉嫌侵权内容。
简介： pyspark提交job到yarn集群一些问题总结


环境






centos7

spark3.3

hive3.3

hadoop3.2




创建pyspark作业


作业用于访问hive元数据的案例

frompyspark.sqlimportSparkSessionfrompyspark.sql.functionsimportudf, colfrompyspark.sql.typesimportStringTypedefconvertCase(str):
resStr=""arr=str.split(" ")
forxinarr:
resStr=resStr+x[0:1].upper() +x[1:len(x)] +" "returnresStrdefmain():
# hive.metastore.uris: 访问hive metastore 服务的地址spark=SparkSession.builder \
        .appName('SparkByTry') \
        .config("hive.metastore.uris", "thrift://192.168.10.100:9083") \
        .enableHiveSupport() \
        .getOrCreate()
spark.conf.set("spark.sql.execution.arrow.pyspark.enabled", 'true')
df=spark.createDataFrame([("Scala", 25000), ("Spark", 35000), ("PHP", 21000)])
df.show()
# Spark SQLdf.createOrReplaceTempView("sample_table")
df2=spark.sql("SELECT _1,_2 FROM sample_table")
df2.show()
# Create Hive table & query it.spark.table("sample_table").write.saveAsTable("sample_hive_table")
df3=spark.sql("SELECT _1,_2 FROM sample_hive_table")
df3.show()
dbs=spark.catalog.listDatabases()
print(dbs)
if__name__=='__main__':
main()
# 1. 把宿主机上的.py文件, 提交到虚拟机hadoop集群中# scp try_pyspark.py root@192.168.10.100:/opt/spark-3.3.0-bin-hadoop3/app/# 2. 虚拟机hadoop集群中，提交作业到yarn上# spark-submit --master yarn --deploy-mode client --executor-cores 1 try_pyspark.py


提交作业时，产生问题集合


问题1


[hadoop@devapp]$spark-submit--masteryarn--deploy-modecluster--executor-cores1try_pyspark.py22/07/2502:08:59WARNUtils: Yourhostname, devresolvestoaloopbackaddress: 127.0.0.1; using192.168.10.100instead (oninterfaceens33)
22/07/2502:08:59WARNUtils: SetSPARK_LOCAL_IPifyouneedtobindtoanotheraddress22/07/2502:08:59WARNNativeCodeLoader: Unabletoloadnative-hadooplibraryforyourplatform... usingbuiltin-javaclasseswhereapplicable22/07/2502:08:59INFODefaultNoHARMFailoverProxyProvider: ConnectingtoResourceManagerat/0.0.0.0:803222/07/2502:09:00INFOConfiguration: resource-types.xmlnotfound22/07/2502:09:00INFOResourceUtils: Unabletofind'resource-types.xml'.
22/07/2502:09:00INFOClient: Verifyingourapplicationhasnotrequestedmorethanthemaximummemorycapabilityofthecluster (8192MBpercontainer)
22/07/2502:09:00INFOClient: WillallocateAMcontainer, with1408MBmemoryincluding384MBoverhead22/07/2502:09:00INFOClient: SettingupcontainerlaunchcontextforourAM22/07/2502:09:00INFOClient: SettingupthelaunchenvironmentforourAMcontainer22/07/2502:09:00INFOClient: PreparingresourcesforourAMcontainer22/07/2502:09:00WARNClient: Neitherspark.yarn.jarsnorspark.yarn.archiveisset, fallingbacktouploadinglibrariesunderSPARK_HOME.
22/07/2502:09:02INFOClient: Uploadingresourcefile:/tmp/spark-dcc409bb-6353-4726-b1dd-c9b4950f0c26/__spark_libs__7142235299947056098.zip->hdfs://localhost:9000/user/hadoop/.sparkStaging/application_1658714171526_0005/__spark_libs__7142235299947056098.zip22/07/2502:09:04INFOClient: Uploadingresourcefile:/opt/spark-3.3.0-bin-hadoop3/app/try_pyspark.py->hdfs://localhost:9000/user/hadoop/.sparkStaging/application_1658714171526_0005/try_pyspark.py22/07/2502:09:04INFOClient: Uploadingresourcefile:/opt/spark-3.3.0-bin-hadoop3/python/lib/pyspark.zip->hdfs://localhost:9000/user/hadoop/.sparkStaging/application_1658714171526_0005/pyspark.zip22/07/2502:09:04INFOClient: Uploadingresourcefile:/opt/spark-3.3.0-bin-hadoop3/python/lib/py4j-0.10.9.5-src.zip->hdfs://localhost:9000/user/hadoop/.sparkStaging/application_1658714171526_0005/py4j-0.10.9.5-src.zip22/07/2502:09:05INFOClient: Uploadingresourcefile:/tmp/spark-dcc409bb-6353-4726-b1dd-c9b4950f0c26/__spark_conf__6560447230202963905.zip->hdfs://localhost:9000/user/hadoop/.sparkStaging/application_1658714171526_0005/__spark_conf__.zip22/07/2502:09:05INFOSecurityManager: Changingviewaclsto: hadoop22/07/2502:09:05INFOSecurityManager: Changingmodifyaclsto: hadoop22/07/2502:09:05INFOSecurityManager: Changingviewaclsgroupsto: 
22/07/2502:09:05INFOSecurityManager: Changingmodifyaclsgroupsto:
22/07/2502:09:05INFOSecurityManager: SecurityManager: authenticationdisabled; uiaclsdisabled; userswithviewpermissions: Set(hadoop); groupswithviewpermissions: Set(); userswithmodifypermissions: Set(hadoop); groupswithmodifypermissions: Set()
22/07/2502:09:05INFOClient: Submittingapplicationapplication_1658714171526_0005toResourceManager22/07/2502:09:05INFOYarnClientImpl: Submittedapplicationapplication_1658714171526_000522/07/2502:09:06INFOClient: Applicationreportforapplication_1658714171526_0005 (state: ACCEPTED)
22/07/2502:09:06INFOClient:
clienttoken: N/Adiagnostics: AMcontainerislaunched, waitingforAMcontainertoRegisterwithRMApplicationMasterhost: N/AApplicationMasterRPCport: -1queue: defaultstarttime: 1658729345928finalstatus: UNDEFINEDtrackingURL: http://localhost:8088/proxy/application_1658714171526_0005/user: hadoop22/07/2502:09:08INFOClient: Applicationreportforapplication_1658714171526_0005 (state: ACCEPTED)
22/07/2502:09:09INFOClient: Applicationreportforapplication_1658714171526_0005 (state: ACCEPTED)
22/07/2502:09:10INFOClient: Applicationreportforapplication_1658714171526_0005 (state: ACCEPTED)
22/07/2502:09:11INFOClient: Applicationreportforapplication_1658714171526_0005 (state: FAILED)
22/07/2502:09:11INFOClient:
clienttoken: N/Adiagnostics: Applicationapplication_1658714171526_0005failed2timesduetoAMContainerforappattempt_1658714171526_0005_000002exitedwithexitCode: 127Failingthisattempt.Diagnostics: [2022-07-2502:09:10.245]Exceptionfromcontainer-launch.
Containerid: container_1658714171526_0005_02_000001Exitcode: 127[2022-07-2502:09:10.247]Containerexitedwithanon-zeroexitcode127.Errorfile: prelaunch.err.
Last4096bytesofprelaunch.err :
Last4096bytesofstderr :
/bin/bash: /bin/java: Nosuchfileordirectory[2022-07-2502:09:10.247]Containerexitedwithanon-zeroexitcode127.Errorfile: prelaunch.err.
Last4096bytesofprelaunch.err :
Last4096bytesofstderr :
/bin/bash: /bin/java: NosuchfileordirectoryFormoredetailedoutput, checktheapplicationtrackingpage: http://localhost:8088/cluster/app/application_1658714171526_0005Thenclickonlinkstologsofeachattempt.
. Failingtheapplication.
ApplicationMasterhost: N/AApplicationMasterRPCport: -1queue: defaultstarttime: 1658729345928finalstatus: FAILEDtrackingURL: http://localhost:8088/cluster/app/application_1658714171526_0005user: hadoop22/07/2502:09:11INFOClient: Deletedstagingdirectoryhdfs://localhost:9000/user/hadoop/.sparkStaging/application_1658714171526_000522/07/2502:09:11ERRORClient: Applicationdiagnosticsmessage: Applicationapplication_1658714171526_0005failed2timesduetoAMContainerforappattempt_1658714171526_0005_000002exitedwithexitCode: 127Failingthisattempt.Diagnostics: [2022-07-2502:09:10.245]Exceptionfromcontainer-launch.
Containerid: container_1658714171526_0005_02_000001Exitcode: 127[2022-07-2502:09:10.247]Containerexitedwithanon-zeroexitcode127.Errorfile: prelaunch.err.
Last4096bytesofprelaunch.err :
Last4096bytesofstderr :
/bin/bash: /bin/java: Nosuchfileordirectory[2022-07-2502:09:10.247]Containerexitedwithanon-zeroexitcode127.Errorfile: prelaunch.err.
Last4096bytesofprelaunch.err :
Last4096bytesofstderr :
/bin/bash: /bin/java: NosuchfileordirectoryFormoredetailedoutput, checktheapplicationtrackingpage: http://localhost:8088/cluster/app/application_1658714171526_0005Thenclickonlinkstologsofeachattempt.
. Failingtheapplication.
Exceptioninthread"main"org.apache.spark.SparkException: Applicationapplication_1658714171526_0005finishedwithfailedstatusatorg.apache.spark.deploy.yarn.Client.run(Client.scala:1342)
atorg.apache.spark.deploy.yarn.YarnClusterApplication.start(Client.scala:1764)
atorg.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:958)
atorg.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
atorg.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
atorg.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
atorg.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1046)
atorg.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1055)
atorg.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
22/07/2502:09:11INFOShutdownHookManager: Shutdownhookcalled22/07/2502:09:11INFOShutdownHookManager: Deletingdirectory/tmp/spark-dcc409bb-6353-4726-b1dd-c9b4950f0c2622/07/2502:09:11INFOShutdownHookManager: Deletingdirectory/tmp/spark-575f8a55-72f6-471d-8bb6-9fdb6af8dce5


问题原因：

没有在etc/hadoop/yarn-env.sh中配置java的环境变量的路径


解决：


编辑yarn-env.sh文件，在文件内添加如下JAVA_HOME信息即可。




问题2



[hadoop@devapp]$spark-submit--masteryarn--deploy-modecluster--executor-cores1try_pyspark.py22/07/2502:34:13WARNUtils: Yourhostname, devresolvestoaloopbackaddress: 127.0.0.1; using192.168.10.100instead (oninterfaceens33)
22/07/2502:34:13WARNUtils: SetSPARK_LOCAL_IPifyouneedtobindtoanotheraddress22/07/2502:34:13WARNNativeCodeLoader: Unabletoloadnative-hadooplibraryforyourplatform... usingbuiltin-javaclasseswhereapplicable22/07/2502:34:13INFODefaultNoHARMFailoverProxyProvider: ConnectingtoResourceManagerat/0.0.0.0:803222/07/2502:34:14INFOConfiguration: resource-types.xmlnotfound22/07/2502:34:14INFOResourceUtils: Unabletofind'resource-types.xml'.
22/07/2502:34:14INFOClient: Verifyingourapplicationhasnotrequestedmorethanthemaximummemorycapabilityofthecluster (8192MBpercontainer)
22/07/2502:34:14INFOClient: WillallocateAMcontainer, with1408MBmemoryincluding384MBoverhead22/07/2502:34:14INFOClient: SettingupcontainerlaunchcontextforourAM22/07/2502:34:14INFOClient: SettingupthelaunchenvironmentforourAMcontainer22/07/2502:34:14INFOClient: PreparingresourcesforourAMcontainer22/07/2502:34:14WARNClient: Neitherspark.yarn.jarsnorspark.yarn.archiveisset, fallingbacktouploadinglibrariesunderSPARK_HOME.
22/07/2502:34:16INFOClient: Uploadingresourcefile:/tmp/spark-1418dbb5-6bef-4ee2-a299-2540be42102b/__spark_libs__4897785966014047449.zip->hdfs://localhost:9000/user/hadoop/.sparkStaging/application_1658730416274_0003/__spark_libs__4897785966014047449.zip22/07/2502:34:17INFOClient: Uploadingresourcefile:/opt/spark-3.3.0-bin-hadoop3/app/try_pyspark.py->hdfs://localhost:9000/user/hadoop/.sparkStaging/application_1658730416274_0003/try_pyspark.py22/07/2502:34:17INFOClient: Uploadingresourcefile:/opt/spark-3.3.0-bin-hadoop3/python/lib/pyspark.zip->hdfs://localhost:9000/user/hadoop/.sparkStaging/application_1658730416274_0003/pyspark.zip22/07/2502:34:17INFOClient: Uploadingresourcefile:/opt/spark-3.3.0-bin-hadoop3/python/lib/py4j-0.10.9.5-src.zip->hdfs://localhost:9000/user/hadoop/.sparkStaging/application_1658730416274_0003/py4j-0.10.9.5-src.zip22/07/2502:34:18INFOClient: Uploadingresourcefile:/tmp/spark-1418dbb5-6bef-4ee2-a299-2540be42102b/__spark_conf__4669748042767956956.zip->hdfs://localhost:9000/user/hadoop/.sparkStaging/application_1658730416274_0003/__spark_conf__.zip22/07/2502:34:18INFOSecurityManager: Changingviewaclsto: hadoop22/07/2502:34:18INFOSecurityManager: Changingmodifyaclsto: hadoop22/07/2502:34:18INFOSecurityManager: Changingviewaclsgroupsto:
22/07/2502:34:18INFOSecurityManager: Changingmodifyaclsgroupsto:
22/07/2502:34:18INFOSecurityManager: SecurityManager: authenticationdisabled; uiaclsdisabled; userswithviewpermissions: Set(hadoop); groupswithviewpermissions: Set(); userswithmodifypermissions: Set(hadoop); groupswithmodifypermissions: Set()
22/07/2502:34:18INFOClient: Submittingapplicationapplication_1658730416274_0003toResourceManager22/07/2502:34:18INFOYarnClientImpl: Submittedapplicationapplication_1658730416274_000322/07/2502:34:19INFOClient: Applicationreportforapplication_1658730416274_0003 (state: ACCEPTED)
22/07/2502:34:19INFOClient: 
clienttoken: N/Adiagnostics: AMcontainerislaunched, waitingforAMcontainertoRegisterwithRMApplicationMasterhost: N/AApplicationMasterRPCport: -1queue: defaultstarttime: 1658730858710finalstatus: UNDEFINEDtrackingURL: http://localhost:8088/proxy/application_1658730416274_0003/user: hadoop22/07/2502:34:20INFOClient: Applicationreportforapplication_1658730416274_0003 (state: ACCEPTED)
22/07/2502:34:21INFOClient: Applicationreportforapplication_1658730416274_0003 (state: ACCEPTED)
22/07/2502:34:22INFOClient: Applicationreportforapplication_1658730416274_0003 (state: ACCEPTED)
22/07/2502:34:23INFOClient: Applicationreportforapplication_1658730416274_0003 (state: ACCEPTED)
22/07/2502:34:24INFOClient: Applicationreportforapplication_1658730416274_0003 (state: ACCEPTED)
22/07/2502:34:25INFOClient: Applicationreportforapplication_1658730416274_0003 (state: ACCEPTED)
22/07/2502:34:26INFOClient: Applicationreportforapplication_1658730416274_0003 (state: ACCEPTED)
22/07/2502:34:27INFOClient: Applicationreportforapplication_1658730416274_0003 (state: ACCEPTED)
22/07/2502:34:28INFOClient: Applicationreportforapplication_1658730416274_0003 (state: ACCEPTED)
22/07/2502:34:29INFOClient: Applicationreportforapplication_1658730416274_0003 (state: ACCEPTED)
22/07/2502:34:30INFOClient: Applicationreportforapplication_1658730416274_0003 (state: ACCEPTED)
22/07/2502:34:31INFOClient: Applicationreportforapplication_1658730416274_0003 (state: ACCEPTED)
22/07/2502:34:32INFOClient: Applicationreportforapplication_1658730416274_0003 (state: ACCEPTED)
22/07/2502:34:33INFOClient: Applicationreportforapplication_1658730416274_0003 (state: ACCEPTED)
22/07/2502:34:34INFOClient: Applicationreportforapplication_1658730416274_0003 (state: ACCEPTED)
22/07/2502:34:35INFOClient: Applicationreportforapplication_1658730416274_0003 (state: FAILED)
22/07/2502:34:35INFOClient:
clienttoken: N/Adiagnostics: Applicationapplication_1658730416274_0003failed2timesduetoAMContainerforappattempt_1658730416274_0003_000002exitedwithexitCode: 13Failingthisattempt.Diagnostics: [2022-07-2502:34:35.030]Exceptionfromcontainer-launch.
Containerid: container_1658730416274_0003_02_000001Exitcode: 13[2022-07-2502:34:35.032]Containerexitedwithanon-zeroexitcode13.Errorfile: prelaunch.err.
Last4096bytesofprelaunch.err :
Last4096bytesofstderr :
UsingSpark's default log4j profile: org/apache/spark/log4j2-defaults.properties[2022-07-2502:34:35.034]Containerexitedwithanon-zeroexitcode13.Errorfile: prelaunch.err.
Last4096bytesofprelaunch.err :
Last4096bytesofstderr :
UsingSpark's default log4j profile: org/apache/spark/log4j2-defaults.propertiesFormoredetailedoutput, checktheapplicationtrackingpage: http://localhost:8088/cluster/app/application_1658730416274_0003Thenclickonlinkstologsofeachattempt.
. Failingtheapplication.
ApplicationMasterhost: N/AApplicationMasterRPCport: -1queue: defaultstarttime: 1658730858710finalstatus: FAILEDtrackingURL: http://localhost:8088/cluster/app/application_1658730416274_0003user: hadoop22/07/2502:34:35ERRORClient: Applicationdiagnosticsmessage: Applicationapplication_1658730416274_0003failed2timesduetoAMContainerforappattempt_1658730416274_0003_000002exitedwithexitCode: 13Failingthisattempt.Diagnostics: [2022-07-2502:34:35.030]Exceptionfromcontainer-launch.
Containerid: container_1658730416274_0003_02_000001Exitcode: 13[2022-07-2502:34:35.032]Containerexitedwithanon-zeroexitcode13.Errorfile: prelaunch.err.
Last4096bytesofprelaunch.err :
Last4096bytesofstderr :
UsingSpark's default log4j profile: org/apache/spark/log4j2-defaults.properties[2022-07-2502:34:35.034]Containerexitedwithanon-zeroexitcode13.Errorfile: prelaunch.err.
Last4096bytesofprelaunch.err :
Last4096bytesofstderr :
UsingSpark's default log4j profile: org/apache/spark/log4j2-defaults.propertiesFormoredetailedoutput, checktheapplicationtrackingpage: http://localhost:8088/cluster/app/application_1658730416274_0003Thenclickonlinkstologsofeachattempt.
. Failingtheapplication.
Exceptioninthread"main"org.apache.spark.SparkException: Applicationapplication_1658730416274_0003finishedwithfailedstatusatorg.apache.spark.deploy.yarn.Client.run(Client.scala:1342)
atorg.apache.spark.deploy.yarn.YarnClusterApplication.start(Client.scala:1764)
atorg.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:958)
atorg.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
atorg.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
atorg.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
atorg.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1046)
atorg.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1055)
atorg.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
22/07/2502:34:35INFOShutdownHookManager: Shutdownhookcalled22/07/2502:34:35INFOShutdownHookManager: Deletingdirectory/tmp/spark-cd1e6fe8-443d-4660-8a60-eaaf5b2f828422/07/2502:34:35INFOShutdownHookManager: Deletingdirectory/tmp/spark-1418dbb5-6bef-4ee2-a299-2540be42102b


主要问题


Application application_1658730416274_0003 failed 2 times due to AM Container for appattempt_1658730416274_0003_000002 exited with  exitCode: 13


解决方法


把  

spark-submit --master yarn --deploy-mode cluster --executor-cores 1 try_pyspark.py

修改为：

spark-submit --master yarn --deploy-mode client --executor-cores 1 try_pyspark.py



问题3


[hadoop@devapp]$spark-submit--masteryarn--deploy-modeclient--executor-cores1try_pyspark.py22/07/2502:34:53WARNUtils: Yourhostname, devresolvestoaloopbackaddress: 127.0.0.1; using192.168.10.100instead (oninterfaceens33)
22/07/2502:34:53WARNUtils: SetSPARK_LOCAL_IPifyouneedtobindtoanotheraddress22/07/2502:34:54INFOSparkContext: RunningSparkversion3.3.022/07/2502:34:54WARNNativeCodeLoader: Unabletoloadnative-hadooplibraryforyourplatform... usingbuiltin-javaclasseswhereapplicable22/07/2502:34:54INFOResourceUtils: ==============================================================22/07/2502:34:54INFOResourceUtils: Nocustomresourcesconfiguredforspark.driver.
22/07/2502:34:54INFOResourceUtils: ==============================================================22/07/2502:34:54INFOSparkContext: Submittedapplication: SparkByTry22/07/2502:34:54INFOResourceProfile: DefaultResourceProfilecreated, executorresources: Map(cores->name: cores, amount: 1, script: , vendor: , memory->name: memory, amount: 1024, script: , vendor: , offHeap->name: offHeap, amount: 0, script: , vendor: ), taskresources: Map(cpus->name: cpus, amount: 1.0)
22/07/2502:34:54INFOResourceProfile: Limitingresourceiscpusat1tasksperexecutor22/07/2502:34:54INFOResourceProfileManager: AddedResourceProfileid: 022/07/2502:34:54INFOSecurityManager: Changingviewaclsto: hadoop22/07/2502:34:54INFOSecurityManager: Changingmodifyaclsto: hadoop22/07/2502:34:54INFOSecurityManager: Changingviewaclsgroupsto:
22/07/2502:34:54INFOSecurityManager: Changingmodifyaclsgroupsto:
22/07/2502:34:54INFOSecurityManager: SecurityManager: authenticationdisabled; uiaclsdisabled; userswithviewpermissions: Set(hadoop); groupswithviewpermissions: Set(); userswithmodifypermissions: Set(hadoop); groupswithmodifypermissions: Set()
22/07/2502:34:54INFOUtils: Successfullystartedservice'sparkDriver'onport38198.22/07/2502:34:54INFOSparkEnv: RegisteringMapOutputTracker22/07/2502:34:54INFOSparkEnv: RegisteringBlockManagerMaster22/07/2502:34:54INFOBlockManagerMasterEndpoint: Usingorg.apache.spark.storage.DefaultTopologyMapperforgettingtopologyinformation22/07/2502:34:54INFOBlockManagerMasterEndpoint: BlockManagerMasterEndpointup22/07/2502:34:54INFOSparkEnv: RegisteringBlockManagerMasterHeartbeat22/07/2502:34:54INFODiskBlockManager: Createdlocaldirectoryat/tmp/blockmgr-4f892dc2-c4f8-4a1c-8a37-61193b7e411322/07/2502:34:54INFOMemoryStore: MemoryStorestartedwithcapacity366.3MiB22/07/2502:34:55INFOSparkEnv: RegisteringOutputCommitCoordinator22/07/2502:34:55INFOUtils: Successfullystartedservice'SparkUI'onport4040.22/07/2502:34:55INFODefaultNoHARMFailoverProxyProvider: ConnectingtoResourceManagerat/0.0.0.0:803222/07/2502:34:56INFOConfiguration: resource-types.xmlnotfound22/07/2502:34:56INFOResourceUtils: Unabletofind'resource-types.xml'.
22/07/2502:34:56INFOClient: Verifyingourapplicationhasnotrequestedmorethanthemaximummemorycapabilityofthecluster (8192MBpercontainer)
22/07/2502:34:56INFOClient: WillallocateAMcontainer, with896MBmemoryincluding384MBoverhead22/07/2502:34:56INFOClient: SettingupcontainerlaunchcontextforourAM22/07/2502:34:56INFOClient: SettingupthelaunchenvironmentforourAMcontainer22/07/2502:34:56INFOClient: PreparingresourcesforourAMcontainer22/07/2502:34:56WARNClient: Neitherspark.yarn.jarsnorspark.yarn.archiveisset, fallingbacktouploadinglibrariesunderSPARK_HOME.
22/07/2502:34:57INFOClient: Uploadingresourcefile:/tmp/spark-74750464-cbd4-48e9-97b0-8905a14121cf/__spark_libs__4495574967982100833.zip->hdfs://localhost:9000/user/hadoop/.sparkStaging/application_1658730416274_0004/__spark_libs__4495574967982100833.zip22/07/2502:34:59INFOClient: Uploadingresourcefile:/opt/spark-3.3.0-bin-hadoop3/python/lib/pyspark.zip->hdfs://localhost:9000/user/hadoop/.sparkStaging/application_1658730416274_0004/pyspark.zip22/07/2502:34:59INFOClient: Uploadingresourcefile:/opt/spark-3.3.0-bin-hadoop3/python/lib/py4j-0.10.9.5-src.zip->hdfs://localhost:9000/user/hadoop/.sparkStaging/application_1658730416274_0004/py4j-0.10.9.5-src.zip22/07/2502:34:59INFOClient: Uploadingresourcefile:/tmp/spark-74750464-cbd4-48e9-97b0-8905a14121cf/__spark_conf__5291090342782308591.zip->hdfs://localhost:9000/user/hadoop/.sparkStaging/application_1658730416274_0004/__spark_conf__.zip22/07/2502:35:00INFOSecurityManager: Changingviewaclsto: hadoop22/07/2502:35:00INFOSecurityManager: Changingmodifyaclsto: hadoop22/07/2502:35:00INFOSecurityManager: Changingviewaclsgroupsto:
22/07/2502:35:00INFOSecurityManager: Changingmodifyaclsgroupsto:
22/07/2502:35:00INFOSecurityManager: SecurityManager: authenticationdisabled; uiaclsdisabled; userswithviewpermissions: Set(hadoop); groupswithviewpermissions: Set(); userswithmodifypermissions: Set(hadoop); groupswithmodifypermissions: Set()
22/07/2502:35:00INFOClient: Submittingapplicationapplication_1658730416274_0004toResourceManager22/07/2502:35:00INFOYarnClientImpl: Submittedapplicationapplication_1658730416274_000422/07/2502:35:01INFOClient: Applicationreportforapplication_1658730416274_0004 (state: ACCEPTED)
22/07/2502:35:01INFOClient: 
clienttoken: N/Adiagnostics: AMcontainerislaunched, waitingforAMcontainertoRegisterwithRMApplicationMasterhost: N/AApplicationMasterRPCport: -1queue: defaultstarttime: 1658730900453finalstatus: UNDEFINEDtrackingURL: http://localhost:8088/proxy/application_1658730416274_0004/user: hadoop22/07/2502:35:02INFOClient: Applicationreportforapplication_1658730416274_0004 (state: ACCEPTED)
22/07/2502:35:03INFOClient: Applicationreportforapplication_1658730416274_0004 (state: ACCEPTED)
22/07/2502:35:04INFOClient: Applicationreportforapplication_1658730416274_0004 (state: ACCEPTED)
22/07/2502:35:05INFOClient: Applicationreportforapplication_1658730416274_0004 (state: ACCEPTED)
22/07/2502:35:06INFOClient: Applicationreportforapplication_1658730416274_0004 (state: ACCEPTED)
22/07/2502:35:07INFOClient: Applicationreportforapplication_1658730416274_0004 (state: ACCEPTED)
22/07/2502:35:07INFOYarnClientSchedulerBackend: AddWebUIFilter. org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter, Map(PROXY_HOSTS->localhost, PROXY_URI_BASES->http://localhost:8088/proxy/application_1658730416274_0004), /proxy/application_1658730416274_000422/07/2502:35:08INFOClient: Applicationreportforapplication_1658730416274_0004 (state: RUNNING)
22/07/2502:35:08INFOClient:
clienttoken: N/Adiagnostics: N/AApplicationMasterhost: 192.168.10.100ApplicationMasterRPCport: -1queue: defaultstarttime: 1658730900453finalstatus: UNDEFINEDtrackingURL: http://localhost:8088/proxy/application_1658730416274_0004/user: hadoop22/07/2502:35:08INFOYarnClientSchedulerBackend: Applicationapplication_1658730416274_0004hasstartedrunning.
22/07/2502:35:08INFOUtils: Successfullystartedservice'org.apache.spark.network.netty.NettyBlockTransferService'onport34986.22/07/2502:35:08INFONettyBlockTransferService: Servercreatedondev:3498622/07/2502:35:08INFOBlockManager: Usingorg.apache.spark.storage.RandomBlockReplicationPolicyforblockreplicationpolicy22/07/2502:35:08INFOBlockManagerMaster: RegisteringBlockManagerBlockManagerId(driver, dev, 34986, None)
22/07/2502:35:08INFOBlockManagerMasterEndpoint: Registeringblockmanagerdev:34986with366.3MiBRAM, BlockManagerId(driver, dev, 34986, None)
22/07/2502:35:08INFOBlockManagerMaster: RegisteredBlockManagerBlockManagerId(driver, dev, 34986, None)
22/07/2502:35:08INFOBlockManager: InitializedBlockManager: BlockManagerId(driver, dev, 34986, None)
22/07/2502:35:08ERRORSparkContext: ErrorinitializingSparkContext.
java.io.FileNotFoundException: /tmp/spark-events/application_1658730416274_0004.inprogress (Permissiondenied)
atjava.io.FileOutputStream.open0(NativeMethod)
atjava.io.FileOutputStream.open(FileOutputStream.java:270)
atjava.io.FileOutputStream.<init>(FileOutputStream.java:213)
atjava.io.FileOutputStream.<init>(FileOutputStream.java:101)
atorg.apache.spark.deploy.history.EventLogFileWriter.initLogFile(EventLogFileWriters.scala:95)
atorg.apache.spark.deploy.history.SingleEventLogFileWriter.start(EventLogFileWriters.scala:223)
atorg.apache.spark.scheduler.EventLoggingListener.start(EventLoggingListener.scala:83)
atorg.apache.spark.SparkContext.<init>(SparkContext.scala:612)
atorg.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:58)
atsun.reflect.NativeConstructorAccessorImpl.newInstance0(NativeMethod)
atsun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
atsun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
atjava.lang.reflect.Constructor.newInstance(Constructor.java:423)
atpy4j.reflection.MethodInvoker.invoke(MethodInvoker.java:247)
atpy4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
atpy4j.Gateway.invoke(Gateway.java:238)
atpy4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80)
atpy4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69)
atpy4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)
atpy4j.ClientServerConnection.run(ClientServerConnection.java:106)
atjava.lang.Thread.run(Thread.java:748)
22/07/2502:35:08INFOSparkUI: StoppedSparkwebUIathttp://dev:404022/07/2502:35:08WARNYarnSchedulerBackend$YarnSchedulerEndpoint: AttemptedtosendshutdownmessagebeforetheAMhasregistered!22/07/2502:35:08INFOYarnClientSchedulerBackend: Interruptingmonitorthread22/07/2502:35:08WARNYarnSchedulerBackend$YarnSchedulerEndpoint: AttemptedtorequestexecutorsbeforetheAMhasregistered!22/07/2502:35:08INFOYarnClientSchedulerBackend: Shuttingdownallexecutors22/07/2502:35:08INFOYarnSchedulerBackend$YarnDriverEndpoint: Askingeachexecutortoshutdown22/07/2502:35:08INFOYarnClientSchedulerBackend: YARNclientschedulerbackendStopped22/07/2502:35:08INFOMapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpointstopped!22/07/2502:35:08INFOMemoryStore: MemoryStorecleared22/07/2502:35:08INFOBlockManager: BlockManagerstopped22/07/2502:35:08INFOBlockManagerMaster: BlockManagerMasterstopped22/07/2502:35:08INFOOutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinatorstopped!22/07/2502:35:08INFOSparkContext: SuccessfullystoppedSparkContextTraceback (mostrecentcalllast):
File"/opt/spark-3.3.0-bin-hadoop3/app/try_pyspark.py", line57, in<module>main()
File"/opt/spark-3.3.0-bin-hadoop3/app/try_pyspark.py", line19, inmain    .config("hive.metastore.uris", "thrift:192.168.10.100:9083") \
File"/opt/spark-3.3.0-bin-hadoop3/python/lib/pyspark.zip/pyspark/sql/session.py", line269, ingetOrCreateFile"/opt/spark-3.3.0-bin-hadoop3/python/lib/pyspark.zip/pyspark/context.py", line483, ingetOrCreateFile"/opt/spark-3.3.0-bin-hadoop3/python/lib/pyspark.zip/pyspark/context.py", line208, in__init__File"/opt/spark-3.3.0-bin-hadoop3/python/lib/pyspark.zip/pyspark/context.py", line282, in_do_initFile"/opt/spark-3.3.0-bin-hadoop3/python/lib/pyspark.zip/pyspark/context.py", line402, in_initialize_contextFile"/opt/spark-3.3.0-bin-hadoop3/python/lib/py4j-0.10.9.5-src.zip/py4j/java_gateway.py", line1586, in__call__File"/opt/spark-3.3.0-bin-hadoop3/python/lib/py4j-0.10.9.5-src.zip/py4j/protocol.py", line328, inget_return_valuepy4j.protocol.Py4JJavaError: AnerroroccurredwhilecallingNone.org.apache.spark.api.java.JavaSparkContext.
: java.io.FileNotFoundException: /tmp/spark-events/application_1658730416274_0004.inprogress (Permissiondenied)
atjava.io.FileOutputStream.open0(NativeMethod)
atjava.io.FileOutputStream.open(FileOutputStream.java:270)
atjava.io.FileOutputStream.<init>(FileOutputStream.java:213)
atjava.io.FileOutputStream.<init>(FileOutputStream.java:101)
atorg.apache.spark.deploy.history.EventLogFileWriter.initLogFile(EventLogFileWriters.scala:95)
atorg.apache.spark.deploy.history.SingleEventLogFileWriter.start(EventLogFileWriters.scala:223)
atorg.apache.spark.scheduler.EventLoggingListener.start(EventLoggingListener.scala:83)
atorg.apache.spark.SparkContext.<init>(SparkContext.scala:612)
atorg.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:58)
atsun.reflect.NativeConstructorAccessorImpl.newInstance0(NativeMethod)
atsun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
atsun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
atjava.lang.reflect.Constructor.newInstance(Constructor.java:423)
atpy4j.reflection.MethodInvoker.invoke(MethodInvoker.java:247)
atpy4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
atpy4j.Gateway.invoke(Gateway.java:238)
atpy4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80)
atpy4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69)
atpy4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)
atpy4j.ClientServerConnection.run(ClientServerConnection.java:106)
atjava.lang.Thread.run(Thread.java:748)
22/07/2502:35:09INFOShutdownHookManager: Shutdownhookcalled22/07/2502:35:09INFOShutdownHookManager: Deletingdirectory/tmp/spark-de23b347-72ec-4d01-bf65-64e173fba79622/07/2502:35:09INFOShutdownHookManager: Deletingdirectory/tmp/spark-74750464-cbd4-48e9-97b0-8905a14121cf


核心问题


ERROR SparkContext: Error initializing SparkContext.

java.io.FileNotFoundException: /tmp/spark-events/application_1658730416274_0004.inprogress (Permission denied)


解决方法


在root用户下，将/tmp/spark-events/文件夹，修改为hadoop用户和用户组chown-Rhadoop:hadoop/tmp/spark-events


问题4


22/07/2503:04:22INFOHiveConf: Foundconfigurationfilefile:/opt/hive-3.3.3/conf/hive-site.xml22/07/2503:04:22INFOHiveUtils: InitializingHiveMetastoreConnectionversion2.3.9usingSparkclasses.
22/07/2503:04:23INFOHiveClientImpl: WarehouselocationforHiveclient (version2.3.9) isfile:/opt/spark-3.3.0-bin-hadoop3/app/spark-warehouse22/07/2503:04:26WARNHiveClientImpl: HiveClientgotthriftexception, destroyingclientandretrying (0triesremaining)
org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unabletoinstantiateorg.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClientatorg.apache.hadoop.hive.ql.metadata.Hive.getDatabase(Hive.java:1567)
atorg.apache.hadoop.hive.ql.metadata.Hive.databaseExists(Hive.java:1552)
atorg.apache.spark.sql.hive.client.Shim_v0_12.databaseExists(HiveShim.scala:609)
atorg.apache.spark.sql.hive.client.HiveClientImpl.$anonfun$databaseExists$1(HiveClientImpl.scala:394)
atscala.runtime.java8.JFunction0$mcZ$sp.apply(JFunction0$mcZ$sp.java:23)
atorg.apache.spark.sql.hive.client.HiveClientImpl.$anonfun$withHiveState$1(HiveClientImpl.scala:294)
atorg.apache.spark.sql.hive.client.HiveClientImpl.liftedTree1$1(HiveClientImpl.scala:225)
atorg.apache.spark.sql.hive.client.HiveClientImpl.retryLocked(HiveClientImpl.scala:224)
atorg.apache.spark.sql.hive.client.HiveClientImpl.withHiveState(HiveClientImpl.scala:274)
atorg.apache.spark.sql.hive.client.HiveClientImpl.databaseExists(HiveClientImpl.scala:394)
atorg.apache.spark.sql.hive.HiveExternalCatalog.$anonfun$databaseExists$1(HiveExternalCatalog.scala:223)
atscala.runtime.java8.JFunction0$mcZ$sp.apply(JFunction0$mcZ$sp.java:23)
atorg.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:101)
atorg.apache.spark.sql.hive.HiveExternalCatalog.databaseExists(HiveExternalCatalog.scala:223)
atorg.apache.spark.sql.internal.SharedState.externalCatalog$lzycompute(SharedState.scala:150)
atorg.apache.spark.sql.internal.SharedState.externalCatalog(SharedState.scala:140)
atorg.apache.spark.sql.hive.HiveSessionStateBuilder.externalCatalog(HiveSessionStateBuilder.scala:54)
atorg.apache.spark.sql.hive.HiveSessionStateBuilder.$anonfun$catalog$1(HiveSessionStateBuilder.scala:69)
atorg.apache.spark.sql.catalyst.catalog.SessionCatalog.externalCatalog$lzycompute(SessionCatalog.scala:121)
atorg.apache.spark.sql.catalyst.catalog.SessionCatalog.externalCatalog(SessionCatalog.scala:121)
atorg.apache.spark.sql.catalyst.catalog.SessionCatalog.tableExists(SessionCatalog.scala:488)
atorg.apache.spark.sql.DataFrameWriter.saveAsTable(DataFrameWriter.scala:642)
atorg.apache.spark.sql.DataFrameWriter.saveAsTable(DataFrameWriter.scala:573)
atsun.reflect.NativeMethodAccessorImpl.invoke0(NativeMethod)
atsun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
atsun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
atjava.lang.reflect.Method.invoke(Method.java:498)
atpy4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
atpy4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
atpy4j.Gateway.invoke(Gateway.java:282)
atpy4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
atpy4j.commands.CallCommand.execute(CallCommand.java:79)
atpy4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)
atpy4j.ClientServerConnection.run(ClientServerConnection.java:106)
atjava.lang.Thread.run(Thread.java:748)
Causedby: java.lang.RuntimeException: Unabletoinstantiateorg.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClientatorg.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1742)
atorg.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:83)
atorg.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:133)
atorg.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:104)
atorg.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:3607)
atorg.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3659)
atorg.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3639)
atorg.apache.hadoop.hive.ql.metadata.Hive.getDatabase(Hive.java:1563)
        ... 34moreCausedby: java.lang.reflect.InvocationTargetExceptionatsun.reflect.NativeConstructorAccessorImpl.newInstance0(NativeMethod)
atsun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
atsun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
atjava.lang.reflect.Constructor.newInstance(Constructor.java:423)
atorg.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1740)
        ... 41moreCausedby: MetaException(message:CouldnotconnecttometastoreusinganyoftheURIsprovided. Mostrecentfailure: org.apache.thrift.transport.TTransportException: Cannotopennullhost.
atorg.apache.thrift.transport.TSocket.open(TSocket.java:210)
atorg.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:478)
atorg.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:245)
atorg.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.<init>(SessionHiveMetaStoreClient.java:70)
atsun.reflect.NativeConstructorAccessorImpl.newInstance0(NativeMethod)
atsun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
atsun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
atjava.lang.reflect.Constructor.newInstance(Constructor.java:423)
atorg.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1740)
atorg.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:83)
atorg.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:133)
atorg.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:104)
atorg.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:3607)
atorg.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3659)
atorg.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3639)
atorg.apache.hadoop.hive.ql.metadata.Hive.getDatabase(Hive.java:1563)
atorg.apache.hadoop.hive.ql.metadata.Hive.databaseExists(Hive.java:1552)
atorg.apache.spark.sql.hive.client.Shim_v0_12.databaseExists(HiveShim.scala:609)
atorg.apache.spark.sql.hive.client.HiveClientImpl.$anonfun$databaseExists$1(HiveClientImpl.scala:394)
atscala.runtime.java8.JFunction0$mcZ$sp.apply(JFunction0$mcZ$sp.java:23)
atorg.apache.spark.sql.hive.client.HiveClientImpl.$anonfun$withHiveState$1(HiveClientImpl.scala:294)
atorg.apache.spark.sql.hive.client.HiveClientImpl.liftedTree1$1(HiveClientImpl.scala:225)
atorg.apache.spark.sql.hive.client.HiveClientImpl.retryLocked(HiveClientImpl.scala:224)
atorg.apache.spark.sql.hive.client.HiveClientImpl.withHiveState(HiveClientImpl.scala:274)
atorg.apache.spark.sql.hive.client.HiveClientImpl.databaseExists(HiveClientImpl.scala:394)
atorg.apache.spark.sql.hive.HiveExternalCatalog.$anonfun$databaseExists$1(HiveExternalCatalog.scala:223)
atscala.runtime.java8.JFunction0$mcZ$sp.apply(JFunction0$mcZ$sp.java:23)
atorg.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:101)
atorg.apache.spark.sql.hive.HiveExternalCatalog.databaseExists(HiveExternalCatalog.scala:223)
atorg.apache.spark.sql.internal.SharedState.externalCatalog$lzycompute(SharedState.scala:150)
atorg.apache.spark.sql.internal.SharedState.externalCatalog(SharedState.scala:140)
atorg.apache.spark.sql.hive.HiveSessionStateBuilder.externalCatalog(HiveSessionStateBuilder.scala:54)
atorg.apache.spark.sql.hive.HiveSessionStateBuilder.$anonfun$catalog$1(HiveSessionStateBuilder.scala:69)
atorg.apache.spark.sql.catalyst.catalog.SessionCatalog.externalCatalog$lzycompute(SessionCatalog.scala:121)
atorg.apache.spark.sql.catalyst.catalog.SessionCatalog.externalCatalog(SessionCatalog.scala:121)
atorg.apache.spark.sql.catalyst.catalog.SessionCatalog.tableExists(SessionCatalog.scala:488)
atorg.apache.spark.sql.DataFrameWriter.saveAsTable(DataFrameWriter.scala:642)
atorg.apache.spark.sql.DataFrameWriter.saveAsTable(DataFrameWriter.scala:573)
atsun.reflect.NativeMethodAccessorImpl.invoke0(NativeMethod)
atsun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
atsun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
atjava.lang.reflect.Method.invoke(Method.java:498)
atpy4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
atpy4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
atpy4j.Gateway.invoke(Gateway.java:282)
atpy4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
atpy4j.commands.CallCommand.execute(CallCommand.java:79)
atpy4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)
atpy4j.ClientServerConnection.run(ClientServerConnection.java:106)
atjava.lang.Thread.run(Thread.java:748)
)
atorg.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:527)
atorg.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:245)
atorg.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.<init>(SessionHiveMetaStoreClient.java:70)
        ... 46more22/07/2503:04:27WARNHiveClientImpl: DeadlineexceededTraceback (mostrecentcalllast):
File"/opt/spark-3.3.0-bin-hadoop3/app/try_pyspark.py", line59, in<module>main()
File"/opt/spark-3.3.0-bin-hadoop3/app/try_pyspark.py", line34, inmainspark.table("sample_table").write.saveAsTable("sample_hive_table")
File"/opt/spark-3.3.0-bin-hadoop3/python/lib/pyspark.zip/pyspark/sql/readwriter.py", line1041, insaveAsTableFile"/opt/spark-3.3.0-bin-hadoop3/python/lib/py4j-0.10.9.5-src.zip/py4j/java_gateway.py", line1322, in__call__File"/opt/spark-3.3.0-bin-hadoop3/python/lib/pyspark.zip/pyspark/sql/utils.py", line196, indecopyspark.sql.utils.AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unabletoinstantiateorg.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient22/07/2503:04:27INFOSparkContext: Invokingstop() fromshutdownhook22/07/2503:04:27INFOSparkUI: StoppedSparkwebUIathttp://dev:404022/07/2503:04:27INFOYarnClientSchedulerBackend: Interruptingmonitorthread22/07/2503:04:27INFOYarnClientSchedulerBackend: Shuttingdownallexecutors22/07/2503:04:27INFOYarnSchedulerBackend$YarnDriverEndpoint: Askingeachexecutortoshutdown22/07/2503:04:27INFOYarnClientSchedulerBackend: YARNclientschedulerbackendStopped22/07/2503:04:27INFOMapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpointstopped!22/07/2503:04:27INFOMemoryStore: MemoryStorecleared22/07/2503:04:27INFOBlockManager: BlockManagerstopped22/07/2503:04:27INFOBlockManagerMaster: BlockManagerMasterstopped22/07/2503:04:27INFOOutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinatorstopped!22/07/2503:04:27INFOSparkContext: SuccessfullystoppedSparkContext22/07/2503:04:27INFOShutdownHookManager: Shutdownhookcalled22/07/2503:04:27INFOShutdownHookManager: Deletingdirectory/tmp/spark-f401ca87-530a-4af0-bac9-f02e67045abb22/07/2503:04:27INFOShutdownHookManager: Deletingdirectory/tmp/spark-4f77b4fc-f324-4bc7-809c-9b7988a9429722/07/2503:04:27INFOShutdownHookManager: Deletingdirectory/tmp/spark-4f77b4fc-f324-4bc7-809c-9b7988a94297/pyspark-b6c6a3ca-d1c5-441d-b0b7-a40085a158d0


主要问题：


org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient



解决方法


1. 在pyspark 程序中，添加访问hive metastore 服务的支持。

注意： 192.168.10.100 这是hive所在服务器的ip地址

# hive.metastore.uris: 访问hive metastore 服务的地址

   spark = SparkSession.builder \

       .appName('SparkByTry') \

       .config("hive.metastore.uris", "thrift://192.168.10.100:9083") \

       .enableHiveSupport() \

       .getOrCreate()

     

     

2. 在 $HIVE_HOME/conf下的 hive-site.xml文件中，配置一下hive.metastore.uris

<property>

               <name>hive.metastore.uris</name>

               <value>thrift://localhost:9083</value>

               <description>Thrift URI for the remote metastore. Used by metastore client to connect to remote metastore.</description>

       </property>


3. 重启hiveserver2 和 metastore 服务

# 启动hive metastore

nohup /opt/hive-3.3.3/bin/hive --service metastore 1>/dev/null 2>&1 &

# 启动后台的hiveserver2

nohup /opt/hive-3.3.3/bin/hiveserver2 1>/dev/null 2>&1 &

























文章标签：
Java
容器
分布式计算
SQL
Linux
资源调度
Spark
Hadoop
HIVE
关键词：
Yarn集群
Yarn作业
作业Yarn
Yarn提交
游客tehculasawvxq
目录
相关文章
听风de歌
|
11月前
|
弹性计算 资源调度 Kubernetes
Flink三种集群模式，Standalone模式，Flink On YARN，Flink On K8S，这三种模式有啥优缺点，生产环境如何选择呢？
Flink三种集群模式，Standalone模式，Flink On YARN，Flink On K8S，这三种模式有啥优缺点，生产环境如何选择呢？
听风de歌
939 3
武子康
|
1月前
|
存储 分布式计算 资源调度
大数据-04-Hadoop集群 集群群起 NameNode/DataNode启动 3台公网云 ResourceManager Yarn HDFS 集群启动 UI可视化查看 YarnUI（一）
大数据-04-Hadoop集群 集群群起 NameNode/DataNode启动 3台公网云 ResourceManager Yarn HDFS 集群启动 UI可视化查看 YarnUI（一）
武子康
72 5
武子康
|
1月前
|
资源调度 数据可视化 大数据
大数据-04-Hadoop集群 集群群起 NameNode/DataNode启动 3台公网云 ResourceManager Yarn HDFS 集群启动 UI可视化查看 YarnUI（二）
大数据-04-Hadoop集群 集群群起 NameNode/DataNode启动 3台公网云 ResourceManager Yarn HDFS 集群启动 UI可视化查看 YarnUI（二）
武子康
35 4
武子康
|
1月前
|
XML 分布式计算 资源调度
大数据-02-Hadoop集群 XML配置 超详细 core-site.xml hdfs-site.xml 3节点云服务器 2C4G HDFS Yarn MapRedece（一）
大数据-02-Hadoop集群 XML配置 超详细 core-site.xml hdfs-site.xml 3节点云服务器 2C4G HDFS Yarn MapRedece（一）
武子康
141 5
武子康
|
1月前
|
XML 资源调度 网络协议
大数据-02-Hadoop集群 XML配置 超详细 core-site.xml hdfs-site.xml 3节点云服务器 2C4G HDFS Yarn MapRedece（二）
大数据-02-Hadoop集群 XML配置 超详细 core-site.xml hdfs-site.xml 3节点云服务器 2C4G HDFS Yarn MapRedece（二）
武子康
79 4
武子康
|
1月前
|
分布式计算 资源调度 Hadoop
大数据-01-基础环境搭建 超详细 Hadoop Java 环境变量 3节点云服务器 2C4G XML 集群配置 HDFS Yarn MapRedece
大数据-01-基础环境搭建 超详细 Hadoop Java 环境变量 3节点云服务器 2C4G XML 集群配置 HDFS Yarn MapRedece
武子康
68 4
尹正杰
|
2月前
|
资源调度 分布式计算 Hadoop
使用YARN命令管理Hadoop作业
本文介绍了如何使用YARN命令来管理Hadoop作业，包括查看作业列表、检查作业状态、杀死作业、获取作业日志以及检查节点和队列状态等操作。
尹正杰
52 1
使用YARN命令管理Hadoop作业
尹正杰
|
2月前
|
资源调度 分布式计算 Hadoop
搭建YARN集群
文章介绍了如何搭建YARN集群，包括启动HDFS集群、修改YARN配置文件、启动ResourceManager和NodeManager节点，并提供了用于管理Hadoop集群的自定义脚本。
尹正杰
48 2
尹正杰
|
2月前
|
分布式计算 资源调度 Hadoop
在YARN集群上运行部署MapReduce分布式计算框架
主要介绍了如何在YARN集群上配置和运行MapReduce分布式计算框架，包括准备数据、运行MapReduce任务、查看任务日志，并启动HistoryServer服务以便于日志查看。
尹正杰
58 0
嘟嘟嘟嘟嘟嘟
|
4月前
|
SQL 弹性计算 资源调度
云服务器 ECS产品使用问题之bin/spark-sql --master yarn如何进行集群模式运行
云服务器ECS（Elastic Compute Service）是各大云服务商阿里云提供的一种基础云计算服务，它允许用户租用云端计算资源来部署和运行各种应用程序。以下是一个关于如何使用ECS产品的综合指南。
嘟嘟嘟嘟嘟嘟
67 7

热门文章

最新文章

  • 1
    YARN中的CPU资源隔离-CGroups
  • 2
    Flink on YARN（上）：一张图轻松掌握基础架构与启动流程
  • 3
    Hadoop大数据平台实战(05)：深入Spark Cluster集群模式YARN vs Mesos vs Standalone vs K8s
  • 4
    [npm]使用(c)npm安装yarn
  • 5
    换npm yarn的源让install超时去死吧
  • 6
    CDP中使用YARN Web UI和CLI
  • 7
    十二、Spark的安装与部署详情（Local模式，Standalone模式，Spank on YARN模式）
  • 8
    Hadoop 2.x HDFS和YARN的启动方式
  • 9
    YARN的Memory和CPU调优配置详解
  • 10
    CCAH-CCA-500-6题:You want YARN to launch no more than 16 containers per node.
  • 1
    实时计算 Flink版产品使用合集之 Flink on YARN 中使用滚动日志时配置不生效如何解决
    73
  • 2
    够啦！前端Yarn配置和命令
    164
  • 3
    pnpm、npm、yarn是什么？怎么选择？
    219
  • 4
    Yarn安装与使用
    82
  • 5
    nvm, node.js, npm, yarn 安装配置
    283
  • 6
    nvm,npm,yarn相关指令，前端配置准备
    52
  • 7
    Yarn的安装和使用
    310
  • 8
    Yarn介绍及快速安装Debian/Ubuntu Linux
    651
  • 9
    yarn 安装 卸载降级升级 以及常用指令
    753
  • 10
    Hadoop【基础知识 03+04】【Hadoop集群资源管理器yarn】（图片来源于网络）（hadoop fs + hadoop dfs + hdfs dfs 使用举例）
    140

    • 相关课程

    更多
  • Hadoop 分布式资源管理框架 YARN
  • Apache Flink 入门
  • 快速掌握阿里云 E-MapReduce

    • 相关电子书

    更多
  • 深度学习+大数据 TensorFlow on Yarn
  • Docker on Yarn 微服务实践
  • 深度学习+大数据-TensorFlow on Yarn
    • 下一篇
    无影云桌面