开发者社区> 祝威廉> 正文

StreamingPro

简介: StreamingPro is not a complete application, but rather a extensible and programmable framework for spark streaming (also include spark,storm)that can
+关注继续查看

Declarative workflows for building Spark Streaming

1de7721f4209f17f306f024d216317c55367bc2f
Spark Streaming
Spark Streaming is an extension of the core Spark API that enables stream processing from a variety of sources.Spark is a extensible and programmable framework for massive distributed processing of datasets,called Resilient Distributed Datasets (RDD). Spark Streaming receives input data streams and divides the data into batches, which are then processed by the Spark engine to generate the results.Spark Streaming data is organized into a sequence of DStreams,represented internally as a sequence of RDDs.

StreamingPro

StreamingPro is not a complete application, but rather  a extensible and programmable framework for spark streaming (also include spark,storm)that can easily be used to build your streaming application.
StreamingPro also make it possible that all you should do to build streaming program is assembling components(eg. SQL Component) in configuration file. 

Features

  • Pure Spark Streaming(Or normal Spark) program (Storm in future)
  • No need of coding, only declarative workflows
  • Rest API for interactive
  • SQL-Oriented workflows support  
  • Data continuously streamed in & processed in near real-time
  • dynamically CURD of workflows  at runtime via Rest API 
  • Flexible workflows (input, output, parsers, etc...) 
  • High performance
  • Scalable   

Documents

Architecture

cfc7ad03f8758fe950f25976c1e140fbc7af0690
Snip20160510_3.png

Declarative workflows

1de7721f4209f17f306f024d216317c55367bc2f
Snip20160510_4.png

Implementation

e7ea91ecaf0f3c5a6a3f0c6288608a460ec1b282
Snip20160510_1.png

版权声明:本文内容由阿里云实名注册用户自发贡献,版权归原作者所有,阿里云开发者社区不拥有其著作权,亦不承担相应法律责任。具体规则请查看《阿里云开发者社区用户服务协议》和《阿里云开发者社区知识产权保护指引》。如果您发现本社区中有涉嫌抄袭的内容,填写侵权投诉表单进行举报,一经查实,本社区将立刻删除涉嫌侵权内容。

相关文章
Structured_Source_HDFS_Spark 代码 | 学习笔记
快速学习 Structured_Source_HDFS_Spark 代码
36 0
Structured_Source_HDFS_案例介绍 | 学习笔记
快速学习 Structured_Source_HDFS_案例介绍
48 0
carbondata1.5.1编译
What is CarbonData? Apache CarbonData is an indexed columnar data format for fast analytics on big data platform, e.g. Apache Hadoop, Apache Spark, etc. 因为我的spark是2.3.1的版本,而最新版的carbondata1.5.1才支持,但是官网没有编译好的,需要我们自己编译,在编译的时候遇到一些问题,记录一下.
63 0
九十四、Spark-SparkSQL(整合Hive)
九十四、Spark-SparkSQL(整合Hive)
107 0
Dremio与Drill的对比
1.简述 Dremio与Drill简述 2.区别 a).数据源支持 使用最新版本Dremio 3.3.1和Drill 1.16.0Dremio3.1.3版本开始不支持HBase,将来会开源社区版HBase连接器 b).
2646 0
Apache Carbondata on Preto
1.download apache carbondata - 1.5.3apache carbondata - 1.5.4apache spark - 2.3.2apache hadoop - 2.7.
1164 0
让CarbonData使用更简单
Apache CarbonData是一种新的高性能数据存储格式,针对当前大数据领域分析场景需求各异而导致的存储冗余问题,CarbonData提供了一种新的融合数据存储方案,以一份数据同时支持“任意维度组合的过滤查询、快速扫描、详单查询等”多种应用场景,并通过多级索引、字典编码、列存等特性提升了IO扫描和计算性能,实现百亿数据级秒级响应。
5520 0
+关注
祝威廉
文章
问答
视频
文章排行榜
最热
最新
相关电子书
更多
SparkSQL在ETL中的应用
立即下载
低代码开发师(初级)实战教程
立即下载
阿里巴巴DevOps 最佳实践手册
立即下载