AI 助理
备案控制台
开发者社区 大数据 文章 正文

本地windows跑Python程序调用Spark

2017-11-22 1913
版权
版权声明：
本文内容由阿里云实名注册用户自发贡献，版权归原作者所有，阿里云开发者社区不拥有其著作权，亦不承担相应法律责任。具体规则请查看《 阿里云开发者社区用户服务协议》和 《阿里云开发者社区知识产权保护指引》。如果您发现本社区中有涉嫌抄袭的内容，填写 侵权投诉表单进行举报，一经查实，本社区将立刻删除涉嫌侵权内容。
简介： 应用场景 spark是用scala写的一种极其强悍的计算工具，spark内存计算，提供了图计算，流式计算，机器学习，即时查询等十分方便的工具，当然我们也可以通过python代码，来调用实现spark计算，用spark来辅助我们计算，使代码效率更快，用户体验更强。

应用场景

spark是用scala写的一种极其强悍的计算工具，spark内存计算，提供了图计算，流式计算，机器学习，即时查询等十分方便的工具，当然我们也可以通过python代码，来调用实现spark计算，用spark来辅助我们计算，使代码效率更快，用户体验更强。

操作流程

按照windows搭建Python开发环境博文，搭建python开发环境，实际已经将Spark环境部署完成了，所以直接可以用python语言写一些spark相关的程序！

代码示例：

from pyspark import SparkContext

sc = SparkContext("local","Simple App")
doc = sc.parallelize([['a','b','c'],['b','d','d']])
words = doc.flatMap(lambda d:d).distinct().collect()
word_dict = {w:i for w,i in zip(words,range(len(words)))}
word_dict_b = sc.broadcast(word_dict)

def wordCountPerDoc(d):
    dict={}
    wd = word_dict_b.value
    for w in d:
        if dict.get(wd[w],0):
            dict[wd[w]] +=1
        else:
            dict[wd[w]] = 1
    return dict
print(doc.map(wordCountPerDoc).collect())
print("successful!")

结果展示：

D:\Anaconda\anaconda\python.exe E:/pythonworkspace/pythontest001/Test001/test002.py
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
17/11/21 15:00:18 INFO SparkContext: Running Spark version 1.6.1
17/11/21 15:00:21 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
17/11/21 15:00:21 INFO SecurityManager: Changing view acls to: lenovo
17/11/21 15:00:21 INFO SecurityManager: Changing modify acls to: lenovo
17/11/21 15:00:21 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(lenovo); users with modify permissions: Set(lenovo)
17/11/21 15:00:25 INFO Utils: Successfully started service 'sparkDriver' on port 60670.
17/11/21 15:00:25 INFO Slf4jLogger: Slf4jLogger started
17/11/21 15:00:25 INFO Remoting: Starting remoting
17/11/21 15:00:26 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriverActorSystem@192.168.114.67:60684]
17/11/21 15:00:26 INFO Utils: Successfully started service 'sparkDriverActorSystem' on port 60684.
17/11/21 15:00:26 INFO SparkEnv: Registering MapOutputTracker
17/11/21 15:00:26 INFO SparkEnv: Registering BlockManagerMaster
17/11/21 15:00:26 INFO DiskBlockManager: Created local directory at C:\Users\lenovo\AppData\Local\Temp\blockmgr-a0245427-988c-4b5a-8653-ee9e228de6ba
17/11/21 15:00:26 INFO MemoryStore: MemoryStore started with capacity 511.1 MB
17/11/21 15:00:26 INFO SparkEnv: Registering OutputCommitCoordinator
17/11/21 15:00:26 INFO Utils: Successfully started service 'SparkUI' on port 4040.
17/11/21 15:00:26 INFO SparkUI: Started SparkUI at http://192.168.114.67:4040
17/11/21 15:00:27 INFO Executor: Starting executor ID driver on host localhost
17/11/21 15:00:27 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 60691.
17/11/21 15:00:27 INFO NettyBlockTransferService: Server created on 60691
17/11/21 15:00:27 INFO BlockManagerMaster: Trying to register BlockManager
17/11/21 15:00:27 INFO BlockManagerMasterEndpoint: Registering block manager localhost:60691 with 511.1 MB RAM, BlockManagerId(driver, localhost, 60691)
17/11/21 15:00:27 INFO BlockManagerMaster: Registered BlockManager
17/11/21 15:00:28 INFO SparkContext: Starting job: collect at E:/pythonworkspace/pythontest001/Test001/test002.py:5
17/11/21 15:00:28 INFO DAGScheduler: Registering RDD 2 (distinct at E:/pythonworkspace/pythontest001/Test001/test002.py:5)
17/11/21 15:00:28 INFO DAGScheduler: Got job 0 (collect at E:/pythonworkspace/pythontest001/Test001/test002.py:5) with 1 output partitions
17/11/21 15:00:28 INFO DAGScheduler: Final stage: ResultStage 1 (collect at E:/pythonworkspace/pythontest001/Test001/test002.py:5)
17/11/21 15:00:28 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 0)
17/11/21 15:00:28 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 0)
17/11/21 15:00:28 INFO DAGScheduler: Submitting ShuffleMapStage 0 (PairwiseRDD[2] at distinct at E:/pythonworkspace/pythontest001/Test001/test002.py:5), which has no missing parents
17/11/21 15:00:28 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 6.6 KB, free 6.6 KB)
17/11/21 15:00:28 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 4.3 KB, free 11.0 KB)
17/11/21 15:00:28 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on localhost:60691 (size: 4.3 KB, free: 511.1 MB)
17/11/21 15:00:28 INFO SparkContext: Created broadcast 0 from broadcast at DAGScheduler.scala:1006
17/11/21 15:00:28 INFO DAGScheduler: Submitting 1 missing tasks from ShuffleMapStage 0 (PairwiseRDD[2] at distinct at E:/pythonworkspace/pythontest001/Test001/test002.py:5)
17/11/21 15:00:28 INFO TaskSchedulerImpl: Adding task set 0.0 with 1 tasks
17/11/21 15:00:28 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, localhost, partition 0,PROCESS_LOCAL, 2099 bytes)
17/11/21 15:00:28 INFO Executor: Running task 0.0 in stage 0.0 (TID 0)
17/11/21 15:00:30 INFO PythonRunner: Times: total = 1240, boot = 1221, init = 19, finish = 0
17/11/21 15:00:30 INFO Executor: Finished task 0.0 in stage 0.0 (TID 0). 1222 bytes result sent to driver
17/11/21 15:00:30 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 1433 ms on localhost (1/1)
17/11/21 15:00:30 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool 
17/11/21 15:00:30 INFO DAGScheduler: ShuffleMapStage 0 (distinct at E:/pythonworkspace/pythontest001/Test001/test002.py:5) finished in 1.465 s
17/11/21 15:00:30 INFO DAGScheduler: looking for newly runnable stages
17/11/21 15:00:30 INFO DAGScheduler: running: Set()
17/11/21 15:00:30 INFO DAGScheduler: waiting: Set(ResultStage 1)
17/11/21 15:00:30 INFO DAGScheduler: failed: Set()
17/11/21 15:00:30 INFO DAGScheduler: Submitting ResultStage 1 (PythonRDD[5] at collect at E:/pythonworkspace/pythontest001/Test001/test002.py:5), which has no missing parents
17/11/21 15:00:30 INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated size 5.5 KB, free 16.5 KB)
17/11/21 15:00:30 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 3.4 KB, free 19.8 KB)
17/11/21 15:00:30 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on localhost:60691 (size: 3.4 KB, free: 511.1 MB)
17/11/21 15:00:30 INFO SparkContext: Created broadcast 1 from broadcast at DAGScheduler.scala:1006
17/11/21 15:00:30 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 1 (PythonRDD[5] at collect at E:/pythonworkspace/pythontest001/Test001/test002.py:5)
17/11/21 15:00:30 INFO TaskSchedulerImpl: Adding task set 1.0 with 1 tasks
17/11/21 15:00:30 INFO TaskSetManager: Starting task 0.0 in stage 1.0 (TID 1, localhost, partition 0,NODE_LOCAL, 1894 bytes)
17/11/21 15:00:30 INFO Executor: Running task 0.0 in stage 1.0 (TID 1)
17/11/21 15:00:30 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
17/11/21 15:00:30 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 9 ms
17/11/21 15:00:31 INFO PythonRunner: Times: total = 1289, boot = 1280, init = 9, finish = 0
17/11/21 15:00:31 INFO Executor: Finished task 0.0 in stage 1.0 (TID 1). 1290 bytes result sent to driver
17/11/21 15:00:31 INFO DAGScheduler: ResultStage 1 (collect at E:/pythonworkspace/pythontest001/Test001/test002.py:5) finished in 1.377 s
17/11/21 15:00:31 INFO TaskSetManager: Finished task 0.0 in stage 1.0 (TID 1) in 1375 ms on localhost (1/1)
17/11/21 15:00:31 INFO TaskSchedulerImpl: Removed TaskSet 1.0, whose tasks have all completed, from pool 
17/11/21 15:00:31 INFO DAGScheduler: Job 0 finished: collect at E:/pythonworkspace/pythontest001/Test001/test002.py:5, took 3.307445 s
17/11/21 15:00:31 INFO MemoryStore: Block broadcast_2 stored as values in memory (estimated size 352.0 B, free 20.2 KB)
17/11/21 15:00:31 INFO MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 115.0 B, free 20.3 KB)
17/11/21 15:00:31 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on localhost:60691 (size: 115.0 B, free: 511.1 MB)
17/11/21 15:00:31 INFO SparkContext: Created broadcast 2 from broadcast at PythonRDD.scala:430
17/11/21 15:00:31 INFO SparkContext: Starting job: collect at E:/pythonworkspace/pythontest001/Test001/test002.py:18
17/11/21 15:00:31 INFO DAGScheduler: Got job 1 (collect at E:/pythonworkspace/pythontest001/Test001/test002.py:18) with 1 output partitions
17/11/21 15:00:31 INFO DAGScheduler: Final stage: ResultStage 2 (collect at E:/pythonworkspace/pythontest001/Test001/test002.py:18)
17/11/21 15:00:31 INFO DAGScheduler: Parents of final stage: List()
17/11/21 15:00:31 INFO DAGScheduler: Missing parents: List()
17/11/21 15:00:31 INFO DAGScheduler: Submitting ResultStage 2 (PythonRDD[6] at collect at E:/pythonworkspace/pythontest001/Test001/test002.py:18), which has no missing parents
17/11/21 15:00:31 INFO MemoryStore: Block broadcast_3 stored as values in memory (estimated size 4.3 KB, free 24.5 KB)
17/11/21 15:00:31 INFO MemoryStore: Block broadcast_3_piece0 stored as bytes in memory (estimated size 2.8 KB, free 27.3 KB)
17/11/21 15:00:31 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on localhost:60691 (size: 2.8 KB, free: 511.1 MB)
17/11/21 15:00:31 INFO SparkContext: Created broadcast 3 from broadcast at DAGScheduler.scala:1006
17/11/21 15:00:31 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 2 (PythonRDD[6] at collect at E:/pythonworkspace/pythontest001/Test001/test002.py:18)
17/11/21 15:00:31 INFO TaskSchedulerImpl: Adding task set 2.0 with 1 tasks
17/11/21 15:00:31 INFO TaskSetManager: Starting task 0.0 in stage 2.0 (TID 2, localhost, partition 0,PROCESS_LOCAL, 2110 bytes)
17/11/21 15:00:31 INFO Executor: Running task 0.0 in stage 2.0 (TID 2)
17/11/21 15:00:33 INFO PythonRunner: Times: total = 1199, boot = 1195, init = 3, finish = 1
17/11/21 15:00:33 INFO Executor: Finished task 0.0 in stage 2.0 (TID 2). 1040 bytes result sent to driver
17/11/21 15:00:33 INFO TaskSetManager: Finished task 0.0 in stage 2.0 (TID 2) in 1235 ms on localhost (1/1)
17/11/21 15:00:33 INFO TaskSchedulerImpl: Removed TaskSet 2.0, whose tasks have all completed, from pool 
17/11/21 15:00:33 INFO DAGScheduler: ResultStage 2 (collect at E:/pythonworkspace/pythontest001/Test001/test002.py:18) finished in 1.237 s
17/11/21 15:00:33 INFO DAGScheduler: Job 1 finished: collect at E:/pythonworkspace/pythontest001/Test001/test002.py:18, took 1.267822 s
[{0: 1, 1: 1, 2: 1}, {2: 1, 3: 2}]
successful!
17/11/21 15:00:33 INFO SparkContext: Invoking stop() from shutdown hook

Process finished with exit code 0
文章标签：
Python
Windows
分布式计算
Spark
关键词：
Python程序
apache spark程序
Windows程序
Python windows
程序apache spark
wsc449
目录
相关文章
天下无贼001
|
10天前
|
机器学习/深度学习 数据挖掘 Python
Python编程入门——从零开始构建你的第一个程序
【10月更文挑战第39天】本文将带你走进Python的世界，通过简单易懂的语言和实际的代码示例，让你快速掌握Python的基础语法。无论你是编程新手还是想学习新语言的老手，这篇文章都能为你提供有价值的信息。我们将从变量、数据类型、控制结构等基本概念入手，逐步过渡到函数、模块等高级特性，最后通过一个综合示例来巩固所学知识。让我们一起开启Python编程之旅吧！
天下无贼001
25 1
aliyun8599273441-30642
|
10天前
|
存储 Python
Python编程入门：打造你的第一个程序
【10月更文挑战第39天】在数字时代的浪潮中，掌握编程技能如同掌握了一门新时代的语言。本文将引导你步入Python编程的奇妙世界，从零基础出发，一步步构建你的第一个程序。我们将探索编程的基本概念，通过简单示例理解变量、数据类型和控制结构，最终实现一个简单的猜数字游戏。这不仅是一段代码的旅程，更是逻辑思维和问题解决能力的锻炼之旅。准备好了吗？让我们开始吧！
aliyun8599273441-30642
107 50
游客qf4jmczx4xu2y
|
12天前
|
机器学习/深度学习 数据挖掘 开发者
Python编程入门：理解基础语法与编写第一个程序
【10月更文挑战第37天】本文旨在为初学者提供Python编程的初步了解，通过简明的语言和直观的例子，引导读者掌握Python的基础语法，并完成一个简单的程序。我们将从变量、数据类型到控制结构，逐步展开讲解，确保即使是编程新手也能轻松跟上。文章末尾附有完整代码示例，供读者参考和实践。
游客qf4jmczx4xu2y
29 3
楠竹11
|
22天前
|
机器学习/深度学习 算法 编译器
Python程序到计算图一键转化，详解清华开源深度学习编译器MagPy
【10月更文挑战第26天】MagPy是一款由清华大学研发的开源深度学习编译器，可将Python程序一键转化为计算图，简化模型构建和优化过程。它支持多种深度学习框架，具备自动化、灵活性、优化性能好和易于扩展等特点，适用于模型构建、迁移、部署及教学研究。尽管MagPy具有诸多优势，但在算子支持、优化策略等方面仍面临挑战。
楠竹11
54 3
东方睿赢
|
24天前
|
算法 测试技术 开发者
在Python开发中，性能优化和代码审查至关重要。性能优化通过改进代码结构和算法提高程序运行速度，减少资源消耗
在Python开发中，性能优化和代码审查至关重要。性能优化通过改进代码结构和算法提高程序运行速度，减少资源消耗；代码审查通过检查源代码发现潜在问题，提高代码质量和团队协作效率。本文介绍了一些实用的技巧和工具，帮助开发者提升开发效率。
东方睿赢
26 3
shuj
|
27天前
|
存储 人工智能 数据挖掘
Python编程入门：构建你的第一个程序
【10月更文挑战第22天】编程，这个听起来高深莫测的词汇，实际上就像搭积木一样简单有趣。本文将带你走进Python的世界，用最浅显的语言和实例，让你轻松掌握编写第一个Python程序的方法。无论你是编程新手还是希望了解Python的爱好者，这篇文章都将是你的理想起点。让我们一起开始这段奇妙的编程之旅吧！
shuj
20 3
游客moiomvrp3vyac2
|
1月前
|
IDE 开发工具 Python
Python 编程入门：打造你的第一个程序
【10月更文挑战第6天】编程，这个听起来高大上又充满神秘感的领域，其实就像学习骑自行车一样。一开始你可能会觉得难以掌握平衡，但一旦你学会了，就能自由地穿梭在广阔的道路上。本文将带你走进 Python 的世界，用最简单的方式让你体验编写代码的乐趣。不需要复杂的理论，我们将通过一个简单的例子——制作一个猜数字游戏，来实践学习。准备好了吗？让我们开始吧！
游客moiomvrp3vyac2
48 5
游客moiomvrp3vyac2
|
17天前
|
存储 机器学习/深度学习 搜索推荐
Python编程入门：从零开始构建你的第一个程序
【10月更文挑战第32天】本文旨在通过浅显易懂的方式引导编程新手进入Python的世界。我们将一起探索Python的基础语法，并通过实例学习如何构建一个简单的程序。文章将不直接展示代码，而是鼓励读者在阅读过程中自行尝试编写，以加深理解和记忆。无论你是编程初学者还是希望巩固基础知识的开发者，这篇文章都将是你的良师益友。让我们开始吧！
游客moiomvrp3vyac2
25 0
何雨晨
|
1月前
|
存储 JSON 安全
面向企业应用程序的 Python 配置管理
面向企业应用程序的 Python 配置管理
何雨晨
29 9
历年考试不作弊
|
1月前
|
人工智能 数据挖掘 程序员
Python 编程入门：打造你的第一个程序
【10月更文挑战第3天】编程，这个看似高深莫测的技能，实际上就像学骑自行车一样，一旦掌握，便能开启全新的世界。本文将带领初学者步入Python编程的殿堂，从基础语法到编写实用程序，一步步解锁编程的乐趣。
历年考试不作弊
27 2

热门文章

最新文章

  • 1
    美国国家安全局网络武器被公开，微软宣布已修复可攻破的 Windows 漏洞
  • 2
    在Windows上以服务方式运行 MSOPenTech/Redis
  • 3
    手机上的搜索引擎-Windows Live Search Mobile 发布！
  • 4
    1、Windows API函数归总，网络函数、消息函数、文件处理
  • 5
    浅谈Windows 7 32位与64位之区别
  • 6
    Windows SDK编程(Delphi版) 之 消息框和菜单资源
  • 7
    windows 2008 server 域管理命令
  • 8
    Windows 8实用窍门系列：13.windows 8的文件.文件夹管理---2.文件以及文件夹操作
  • 9
    Windows Phone 及 Metro UI 设计讨论
  • 10
    Windows Server 2008 安装
  • 1
    spark用于分析数据并将数据保存到数据库中是
    71
  • 2
    数据分享|Python、Spark SQL、MapReduce决策树、回归对车祸发生率影响因素可视化分析
    93
  • 3
    如何解决Spark在深度学习中的资源消耗问题？
    89
  • 4
    Spark在深度学习中的优缺点是什么？
    241
  • 5
    使用Spark进行机器学习
    71
  • 6
    Spark是一个基于内存的通用数据处理引擎，可以进行大规模数据处理和分析
    136
  • 7
    【Spark】Spark基础教程知识点
    82
  • 8
    DataWorks产品使用合集之在DataWorks中，通过spark访问外网的步骤如何解决
    168
  • 9
    MaxCompute操作报错合集之大数据计算的MaxCompute Spark引擎无法读取到表，是什么原因
    144
  • 10
    MaxCompute操作报错合集之spark3.1.1通过resource目录下的conf文件配置，报错如何解决
    78

    • 相关课程

    更多
  • 大数据实战项目：反爬虫系统（Lua+Spark+Redis+Hadoop框架搭建）第一阶段
  • 大数据实战项目：反爬虫系统（Lua+Spark+Redis+Hadoop框架搭建）第二阶段
  • 大数据实战项目：反爬虫系统（Lua+Spark+Redis+Hadoop框架搭建）第三阶段
  • 大数据实战项目：反爬虫系统（Lua+Spark+Redis+Hadoop框架搭建）第五阶段
  • 大数据实战项目 - 反爬虫系统（Lua+Spark+Redis+Hadoop框架搭建）第六阶段
  • 大数据实时计算框架Spark快速入门

    • 相关电子书

    更多
  • Hybrid Cloud and Apache Spark
  • Scalable Deep Learning on Spark
  • Comparison of Spark SQL with Hive

    • 相关实验场景

    更多
  • 用Python画圣诞树
  • 阿里云平台上进行Java程序的编译与运行
  • Python选择及循环结构
  • 云端Python及基本操作
  • 快速配置Windows云服务器
  • Python网络通信程序典型应用

    • 推荐镜像

    更多
  • python-release
  • nodejs-release
  • golang
    • 下一篇
    无影云桌面