代码用途:简单测试spark-sql的读hdfs文件,然后用sql 简便提取临时数据
报错猜测:个人刚开始怀疑是pom文件依赖的版本有问题,现在也不确定啥问题,具体描述如下:
1:集群环境spark-shell下执行,很正常,可以很欢快的使用,可以查询people结果。。
val sqlContext = new org.apache.spark.sql.SQLContext(sc)
case class Person(name: String)
var people = sc.textFile("/workspace/xx/tmp/a").map(_.split(" ")).map(p => Person(p(0)))
val peopleSchema = sqlContext.createSchemaRDD(people)
peopleSchema.registerTempTable("people")
var df=sqlContext.sql("select * from people") 2:在本地打包时发现报错,不能编译。
import org.apache.spark.sql.SQLContext
import org.apache.spark.{SparkContext, SparkConf}
import org.apache.spark.SparkContext._
import org.apache.spark.sql
import org.apache.spark.sql._
import org.apache.hadoop.fs.Path
import org.apache.spark.sql.SchemaRDD
object sparkSql {
def main(args: Array[String]) {
val conf = new SparkConf().setAppName("sql_data")
var sc = new SparkContext(conf)
val sqlContext = new org.apache.spark.sql.SQLContext(sc)
case class Person(name: String)
var people = sc.textFile("/workspace/xx/tmp/a").map(_.split(" ")).map(p => Person(p(0)))
val peopleSchema = sqlContext.createSchemaRDD(people)
peopleSchema.registerTempTable("people")
var df=sqlContext.sql("select * from people")
df.map(t => t(0)).collect().foreach(println)
}
} 报错信息:
val peopleSchema = sqlContext.createSchemaRDD(people)
3:pom
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.10</artifactId>
<version>1.1.0</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_2.10</artifactId>
<version>1.1.0</version>
</dependency>
版权声明:本文内容由阿里云实名注册用户自发贡献,版权归原作者所有,阿里云开发者社区不拥有其著作权,亦不承担相应法律责任。具体规则请查看《阿里云开发者社区用户服务协议》和《阿里云开发者社区知识产权保护指引》。如果您发现本社区中有涉嫌抄袭的内容,填写侵权投诉表单进行举报,一经查实,本社区将立刻删除涉嫌侵权内容。
在本地代码中有修改了,依旧报同样的错误!
importsqlContext.createSchemaRDDcaseclassPerson(name:String)把case放外边就好了。。。啦啦啦啦啦!
Justmoveyourcaseclassoutofthemethoddefinition
http://stackoverflow.com/questions/29143756/scala-spark-app-with-no-typetag-available-error-in-def-main-style-app
thanks