使用spark streaming连接loghub报错,是什么问题-问答-阿里云开发者社区-阿里云

开发者社区> 问答> 正文

使用spark streaming连接loghub报错,是什么问题

游客lplm6xso3kx3e 2019-05-28 13:41:12 1184

"main" java.lang.ClassNotFoundException: Failed to find data source: loghub. Please find packages at http://spark.apache.org/third-party-projects.html

at org.apache.spark.sql.execution.datasources.DataSource$.lookupDataSource(DataSource.scala:652)
at org.apache.spark.sql.streaming.DataStreamReader.load(DataStreamReader.scala:159)
at com.aliyun.emr.examples.sql.streaming.RealtimeComputation.main(RealtimeComputation.java:51)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:896)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:198)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:228)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:137)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

Caused by: java.lang.ClassNotFoundException: loghub.DefaultSource

at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$27$$anonfun$apply$15.apply(DataSource.scala:635)
at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$27$$anonfun$apply$15.apply(DataSource.scala:635)
at scala.util.Try$.apply(Try.scala:192)
at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$27.apply(DataSource.scala:635)
at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$27.apply(DataSource.scala:635)
at scala.util.Try.orElse(Try.scala:84)
at org.apache.spark.sql.execution.datasources.DataSource$.lookupDataSource(DataSource.scala:635)
分布式计算 Apache Spark 流计算
分享到
取消 提交回答
全部回答(1)
  • 沐远
    2020-03-21 15:04:30

    1、需要依赖对应的loghub的datasource com.aliyun.emr emr-logservice_2.11 ${emr.version} 2、也可以参考下面的demo https://github.com/aliyun/aliyun-emapreduce-demo/blob/master-2/pom.xml

    0 0
大数据
使用钉钉扫一扫加入圈子
+ 订阅

大数据计算实践乐园,近距离学习前沿技术

推荐文章
相似问题
推荐课程