欢迎访问我的GitHub
这里分类和汇总了欣宸的全部原创(含配套源码):
https://github.com/zq2599/blog_demos
- Scala语言在函数式编程方面的优势适合Spark应用开发,IDEA是我们常用的IDE工具,今天就来实战IDEA开发Scala版的Spark应用;
版本信息
- 操作系统:win10;
- JDK:1.8.0_191;
- IntelliJ IDEA:2018.2.4 (Ultimate Edition)
- Spark:2.3.3;
- Scala:2.11.12;
- Hadoop:2.7.7;
下载Spark安装包
![在这里插入图片描述 在这里插入图片描述](https://ucc.alicdn.com/images/user-upload-01/20190217083443762.jpg?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2JvbGluZ19jYXZhbHJ5,size_16,color_FFFFFF,t_70&x-oss-process=image/resize,w_1400/format,webp)
- 将下载好的文件解压,例如我这里解压后所在目录是:C:\software\spark-2.3.3-bin-hadoop2.7
IDEA安装scala插件
- 打开IDEA,选择"Configure"->"Plugins",如下图:
![在这里插入图片描述 在这里插入图片描述](https://ucc.alicdn.com/images/user-upload-01/2019021708364014.jpg?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2JvbGluZ19jYXZhbHJ5,size_16,color_FFFFFF,t_70&x-oss-process=image/resize,w_1400/format,webp)
- 如下图,在红框1输入"scala",点击红框2,开始在中央仓库说搜索:
![在这里插入图片描述 在这里插入图片描述](https://ucc.alicdn.com/images/user-upload-01/20190217083824497.jpg?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2JvbGluZ19jYXZhbHJ5,size_16,color_FFFFFF,t_70&x-oss-process=image/resize,w_1400/format,webp)
- 在搜索结果中选中"scala",再点击右侧的"Install",如下:
![在这里插入图片描述 在这里插入图片描述](https://ucc.alicdn.com/images/user-upload-01/2019021708395760.jpg?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2JvbGluZ19jYXZhbHJ5,size_16,color_FFFFFF,t_70&x-oss-process=image/resize,w_1400/format,webp)
- 等待在线安装成功后,点击"Restart IntelliJ IDEA",如下:
![在这里插入图片描述 在这里插入图片描述](https://ucc.alicdn.com/images/user-upload-01/20190217084059113.jpg?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2JvbGluZ19jYXZhbHJ5,size_16,color_FFFFFF,t_70&x-oss-process=image/resize,w_1400/format,webp)
新建scala工程
![在这里插入图片描述 在这里插入图片描述](https://ucc.alicdn.com/images/user-upload-01/20190217084212287.jpg?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2JvbGluZ19jYXZhbHJ5,size_16,color_FFFFFF,t_70&x-oss-process=image/resize,w_1400/format,webp)
- 在弹出窗口中选择"Scala"->"IDEA",如下图:
![在这里插入图片描述 在这里插入图片描述](https://ucc.alicdn.com/images/user-upload-01/20190217084335615.jpg?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2JvbGluZ19jYXZhbHJ5,size_16,color_FFFFFF,t_70&x-oss-process=image/resize,w_1400/format,webp)
- 如下图,在红框1中输入项目名称,点击红框2,选择Scala SDK:
![在这里插入图片描述 在这里插入图片描述](https://ucc.alicdn.com/images/user-upload-01/2019021708445567.jpg?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2JvbGluZ19jYXZhbHJ5,size_16,color_FFFFFF,t_70&x-oss-process=image/resize,w_1400/format,webp)
![在这里插入图片描述 在这里插入图片描述](https://ucc.alicdn.com/images/user-upload-01/20190217084518607.jpg?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2JvbGluZ19jYXZhbHJ5,size_16,color_FFFFFF,t_70&x-oss-process=image/resize,w_1400/format,webp)
![在这里插入图片描述 在这里插入图片描述](https://ucc.alicdn.com/images/user-upload-01/20190217084610329.jpg?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2JvbGluZ19jYXZhbHJ5,size_16,color_FFFFFF,t_70&x-oss-process=image/resize,w_1400/format,webp)
- 工程创建成功了,接下来是添加spark库,点击"File"->"Project Structure",如下图:
![在这里插入图片描述 在这里插入图片描述](https://ucc.alicdn.com/images/user-upload-01/20190217084902285.jpg?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2JvbGluZ19jYXZhbHJ5,size_16,color_FFFFFF,t_70&x-oss-process=image/resize,w_1400/format,webp)
![在这里插入图片描述 在这里插入图片描述](https://ucc.alicdn.com/images/user-upload-01/20190217085150270.jpg?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2JvbGluZ19jYXZhbHJ5,size_16,color_FFFFFF,t_70&x-oss-process=image/resize,w_1400/format,webp)
- 在弹出窗口选择前面安装的spark-2.3.3-bin-hadoop2.7文件夹下面的jar文件夹,如下:
![在这里插入图片描述 在这里插入图片描述](https://ucc.alicdn.com/images/user-upload-01/20190217085326359.jpg?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2JvbGluZ19jYXZhbHJ5,size_16,color_FFFFFF,t_70&x-oss-process=image/resize,w_1400/format,webp)
- 如下图,弹出的窗口用来选择模块,就选工程目录即可:
![在这里插入图片描述 在这里插入图片描述](https://ucc.alicdn.com/images/user-upload-01/20190217085433520.jpg?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2JvbGluZ19jYXZhbHJ5,size_16,color_FFFFFF,t_70&x-oss-process=image/resize,w_1400/format,webp)
- 至此,整个spark开发环境已经设置好了,现在写一个demo试试,创建一个object,源码如下:
package com.bolingcavalry.sparkscalademo.app
import org.apache.spark.{SparkConf, SparkContext}
/**
* @Description: 第一个scala语言的spark应用
* @author: willzhao E-mail: zq2599@gmail.com
* @date: 2019/2/16 20:23
*/
object FirstDemo {
def main(args: Array[String]): Unit={
val conf = new SparkConf()
.setAppName("first spark app(scala)")
.setMaster("local[1]");
new SparkContext(conf)
.parallelize(List(1,2,3,4,5,6))
.map(x=>x*x)
.filter(_>10)
.collect()
.foreach(println);
}
}
- 以上代码的功能很简单:创建用一个数组,将每个元素做平方运算,再丢弃小于10的元素,然后逐个打印出来;
- 代码完成后,点击右键选择"Run FirstDemo",即可立即在本机运行,如下图:
![在这里插入图片描述 在这里插入图片描述](https://ucc.alicdn.com/images/user-upload-01/20190217090315760.jpg?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2JvbGluZ19jYXZhbHJ5,size_16,color_FFFFFF,t_70&x-oss-process=image/resize,w_1400/format,webp)
- 由于windows环境并没有做hadoop相关配置,因此控制台上会有错误堆栈输出,但这些信息并不影响程序运行(本例没有用到hadoop),输出如下,可见结果已经被打印出来(16、25、36):
2019-02-17 09:04:21 INFO TaskSchedulerImpl:54 - Removed TaskSet 0.0, whose tasks have all completed, from pool
2019-02-17 09:04:21 INFO DAGScheduler:54 - ResultStage 0 (collect at FirstDemo.scala:20) finished in 0.276 s
2019-02-17 09:04:21 INFO DAGScheduler:54 - Job 0 finished: collect at FirstDemo.scala:20, took 0.328611 s
16
25
36
2019-02-17 09:04:21 INFO SparkContext:54 - Invoking stop() from shutdown hook
2019-02-17 09:04:21 INFO AbstractConnector:318 - Stopped Spark@452ba1db{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}
2019-02-17 09:04:21 INFO SparkUI:54 - Stopped Spark web UI at http://DESKTOP-82CCEBN:4040
2019-02-17 09:04:21 INFO MapOutputTrackerMasterEndpoint:54 - MapOutputTrackerMasterEndpoint stopped!
2019-02-17 09:04:21 INFO MemoryStore:54 - MemoryStore cleared
2019-02-17 09:04:21 INFO BlockManager:54 - BlockManager stopped
2019-02-17 09:04:21 INFO BlockManagerMaster:54 - BlockManagerMaster stopped
2019-02-17 09:04:21 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint:54 - OutputCommitCoordinator stopped!
2019-02-17 09:04:21 INFO SparkContext:54 - Successfully stopped SparkContext
2019-02-17 09:04:21 INFO ShutdownHookManager:54 - Shutdown hook called
2019-02-17 09:04:21 INFO ShutdownHookManager:54 - Deleting directory C:\Users\12167\AppData\Local\Temp\spark-4bbb584a-c7c2-4dc8-9c7e-473de7f8c326
Process finished with exit code 0
构建打包,提交到spark环境运行
- 生产环境下一般是将应用构建打包成jar,放到spark集群中运行,所以我们来操作构建打包;
- 在菜单上选择"File"->"Project Structure",弹出窗口点击"Artifacts",选择新增jar,如下图:
![在这里插入图片描述 在这里插入图片描述](https://ucc.alicdn.com/images/user-upload-01/20190217090806336.jpg?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2JvbGluZ19jYXZhbHJ5,size_16,color_FFFFFF,t_70&x-oss-process=image/resize,w_1400/format,webp)
- 如下图,在弹出的窗口中,红框1位置输入要运行的class,红框2选择的是单选框的第二个"copy to the output ...":
![在这里插入图片描述 在这里插入图片描述](https://ucc.alicdn.com/images/user-upload-01/20190217090933471.jpg?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2JvbGluZ19jYXZhbHJ5,size_16,color_FFFFFF,t_70&x-oss-process=image/resize,w_1400/format,webp)
- 在菜单上选择"Build"->"Build Artifacts...",如下图:
![在这里插入图片描述 在这里插入图片描述](https://ucc.alicdn.com/images/user-upload-01/20190217091213416.jpg?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2JvbGluZ19jYXZhbHJ5,size_16,color_FFFFFF,t_70&x-oss-process=image/resize,w_1400/format,webp)
- 在弹出的菜单中选择"sparkscalademo:jar"->"Rebuild",如下:
![在这里插入图片描述 在这里插入图片描述](https://ucc.alicdn.com/images/user-upload-01/20190217091532243.jpg?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2JvbGluZ19jYXZhbHJ5,size_16,color_FFFFFF,t_70&x-oss-process=image/resize,w_1400/format,webp)
- 如果编译成功,在项目的out\artifacts目录下就会生成文件sparkscalademo.jar,如下:
![在这里插入图片描述 在这里插入图片描述](https://ucc.alicdn.com/images/user-upload-01/20190217091856730.jpg?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2JvbGluZ19jYXZhbHJ5,size_16,color_FFFFFF,t_70&x-oss-process=image/resize,w_1400/format,webp)
spark-submit --class com.bolingcavalry.sparkscalademo.app.FirstDemo /root/jars/sparkscalademo.jar
![在这里插入图片描述 在这里插入图片描述](https://ucc.alicdn.com/images/user-upload-01/20190217092418241.jpg?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2JvbGluZ19jYXZhbHJ5,size_16,color_FFFFFF,t_70&x-oss-process=image/resize,w_1400/format,webp)
- 至此,idea开发spark应用实战就完成了,希望在您配置开发环境的时候本文能够提供一些参考;
欢迎关注华为云博客:程序员欣宸
学习路上,你不孤单,欣宸原创一路相伴...