MaxCompute2.0 Performance Metrics: Faster, Stronger Computing

本文涉及的产品
云原生大数据计算服务 MaxCompute,5000CU*H 100GB 3个月
云原生大数据计算服务MaxCompute,500CU*H 100GB 3个月
简介: This evaluation focuses on performance comparison between MaxCompute2.0 and other offline computing products, as well as between MaxCompute2.

MaxCompute (originally ODPS) is a Big Data processing platform used for batch structural data storage and processing, to provide massive data warehouse solutions and data modeling. MaxCompute2.0 is an upgrade to the original MaxCompute; the system's entire operating process has been optimized for performance. MaxCompute2.0 includes a next generation execution engine and compilation engine, and a cost-based optimization engine.

From testing, we can see that MaxCompute2.0 is better, faster, and stronger with greater functionality, more efficient use and better updating. MaxCompute2.0 doesn't only beat its predecessor, it is also as competitive as services provided by other Big Data vendors. According to performance metrics, MaxCompute2.0 offline computing performs better than its counterpart, Apache Hive2.0 on Tez, by more than 90%. Furthermore, the open ecosystem improves performance when dealing with SQL and SQL offline batch computing which respectively take up 80% of online computing and 50% of offline computing.

Next Generation Execution Engine

MaxCompute2.0 has developed a next generation execution engine. The new execution engine is based on Codegen and utilizes vectorized execution and cache-friendly algorithms. We can see that the new generation of execution engines boast remarkable advantages in performance over open source's next generation open source offline computing execution engine—Hive on Tez.

The chart below demonstrates operation times (in seconds) for both the next generation execution engine employed by MaxCompute2.0 and the community developed next generation offline execution engine Hive on Tez operating on the same amount of data using the same number of instances.

1

We can see that when operating with terabytes of data using hundreds of instances, Maxcompute2.0 consistently out-performs Hive2.0 on Tez (Hive2.0 is already using optimized settings). This includes:

1.Two times faster sum with group performance

2.Three times faster sort-merge join speeds

3.Two times faster hash join

4.50% faster streamline

Next Generation Compilation Engine and Cost-Based Optimizer

MaxCompute2.0 has developed an all new parser and introduced a cost-based optimizer that is compatible with Hive syntax and semantics and utilizes optimizers based on a variety of rules (Rbo), introducing and developing accurate optimizer components based on statistical data and adding a collection of all new rules.

The following is end-to-end operation performance data from TPC-H benchmark comparing Maxcompute2.0 offline computing using its new compiler and cost-based optimizer with the community developed Hive2.0 on Tez under optimal operation settings:

2

Test environment:

1.Cluster scale: 30 test clusters, of which 20 are computing nodes

2.Machine configuration: 22core96GGigabit full dual-channel network121TB SATA hard drives on each node

3.Software versions: MaxCompute2.0Sp24rc5 / hive2.0onTez / MaxCompute1.0Sp23s14 / hive2.0onMr

4.Data size: 1TB (zlib compression)

In order to ensure data rationality, the performance test data is made up of stable values produced by several rounds of testing of each test case. The performance evaluation utilizes independent clusters, each cleared and recovered to its original settings prior to each test. Tests are not run in parallel, rather each round of tests is executed after the entire previous test set is complete.

From the comparison, we can conclude that:

1.MaxCompute2.0 offline computing is faster than its counterpart Hive2.0 on Tez by more than 90%.

2.MaxCompute2.0 is faster than Hive in executing more than 95% of benchmark SQL statements. We also analyzed the internal execution details. With the scheduling time and other time consumption deducted, MaxCompute2.0 actually outrivals Hive2.0 in execution performance by more than 114%.

3.MaxCompute2.0 performance is improved by 68% when compared to MaxCompute1.0.

4.MaxCompute2.0 outperforms Hive2.0 on Mr by 190% in terms of overall performance. Specifically, the performance of 77% of benchmark SQL statements is more than tripled.

MaxCompute2.0 Performance Increase and Future Performance Expectations

Our goal is to have MaxCompute2.0 capable of all these features:

1.Compatible with community developed software, compatible with all Hive data types, follows SQL 2003, supports multidimensional grouping

2.Developed whole-stage code generation and, by condensing code that can slow down the entire search into one function, reduced the number of function calls and took advantage of CPU registers to store intermediate data and other drains on performance

3.More optimization rules as well as updates and upgrades to existing rules, default activation of join reordering, and support for range partitioning.

Apart from the huge performance improvements over the Apsara-based first generation execution engine, MaxCompute2.0 has also launched the following RBO and CBO execution performance optimizations:

1.Trimming rules: column trimming, partition trimming, sub-query trimming

2.Push-down/merge rules: predicate push-down

3.Deduplication rules: project deduplication, exchange deduplication, and sort deduplication

4.Constant folding/predicate derivation

5.Relation optimization: auto MapJoin, Skew Join; implement BroadcastHashJoin, ShuffleHashJoin, and MergeJoin; Join Reordering

6.Aggregate optimization: HashAggregate, SortedAggregate and deduplicate

7.Processing optimization: GroupBy push-down, exchange push-down, and sort push-down

Alibaba Cloud Big Data and AI Products

Alibaba Cloud launched eight new Big Data and artificial intelligence (AI) products at the Mobile World Congress 2018 in Barcelona, Spain. These products, along with MaxCompute2.0, will meet the surging demand for powerful and reliable cloud computing services as well as advanced AI solutions among enterprises. Read more about the launch on the official press release.

相关实践学习
基于MaxCompute的热门话题分析
本实验围绕社交用户发布的文章做了详尽的分析,通过分析能得到用户群体年龄分布,性别分布,地理位置分布,以及热门话题的热度。
SaaS 模式云数据仓库必修课
本课程由阿里云开发者社区和阿里云大数据团队共同出品,是SaaS模式云原生数据仓库领导者MaxCompute核心课程。本课程由阿里云资深产品和技术专家们从概念到方法,从场景到实践,体系化的将阿里巴巴飞天大数据平台10多年的经过验证的方法与实践深入浅出的讲给开发者们。帮助大数据开发者快速了解并掌握SaaS模式的云原生的数据仓库,助力开发者学习了解先进的技术栈,并能在实际业务中敏捷的进行大数据分析,赋能企业业务。 通过本课程可以了解SaaS模式云原生数据仓库领导者MaxCompute核心功能及典型适用场景,可应用MaxCompute实现数仓搭建,快速进行大数据分析。适合大数据工程师、大数据分析师 大量数据需要处理、存储和管理,需要搭建数据仓库?学它! 没有足够人员和经验来运维大数据平台,不想自建IDC买机器,需要免运维的大数据平台?会SQL就等于会大数据?学它! 想知道大数据用得对不对,想用更少的钱得到持续演进的数仓能力?获得极致弹性的计算资源和更好的性能,以及持续保护数据安全的生产环境?学它! 想要获得灵活的分析能力,快速洞察数据规律特征?想要兼得数据湖的灵活性与数据仓库的成长性?学它! 出品人:阿里云大数据产品及研发团队专家 产品 MaxCompute 官网 https://www.aliyun.com/product/odps 
目录
相关文章
|
13天前
|
算法 数据挖掘 测试技术
文献解读-Processing UMI Datasets at High Accuracy and Efficiency with the Sentieon ctDNA Analysis Pipeline
Sentieon ctDNA分析流程通过创新的算法设计和高效的软件实现,为高深度、大panel的ctDNA测序数据分析提了一个快速而准确的解决方案。它在多个数据集上均展现出优于或等同于现有方法的性能,同时大幅提高了处理速度。这一进展有望推动ctDNA技术在临床肿瘤学中的广泛应用,特别是在早期癌症检测和最小残留病监测等领域。
36 8
|
2月前
|
算法 数据挖掘 数据处理
文献解读-Sentieon DNAscope LongRead – A highly Accurate, Fast, and Efficient Pipeline for Germline Variant Calling from PacBio HiFi reads
PacBio® HiFi 测序是第一种提供经济、高精度长读数测序的技术,其平均读数长度超过 10kb,平均碱基准确率达到 99.8% 。在该研究中,研究者介绍了一种准确、高效的 DNAscope LongRead 管道,用于从 PacBio® HiFi 读数中调用胚系变异。DNAscope LongRead 是对 Sentieon 的 DNAscope 工具的修改和扩展,该工具曾获美国食品药品管理局(FDA)精密变异调用奖。
30 2
文献解读-Sentieon DNAscope LongRead – A highly Accurate, Fast, and Efficient Pipeline for Germline Variant Calling from PacBio HiFi reads
|
3月前
|
机器学习/深度学习 自然语言处理 数据可视化
分布式表示(Distributed Representation)
分布式表示(Distributed Representation)
218 15
|
消息中间件 分布式计算 大数据
大数据Spark Continuous Processing
大数据Spark Continuous Processing
142 0
sbs
|
SQL 存储 监控
One SQL to Rule Them All: An Efficient and Syntactically Idiomatic Approach to Management of Streams and Tables 论文翻译
One SQL to Rule Them All: An Efficient and Syntactically Idiomatic Approach to Management of Streams and Tables[文件: One SQL to Rule Them All- An Efficient and Syntactically Idiomatic Approach to Manag
sbs
215 0
One SQL to Rule Them All: An Efficient and Syntactically Idiomatic Approach to Management of Streams and Tables 论文翻译
|
SQL 存储 算法
《Optimization of Common Table Expressions in MPP Database Systems》论文导读
Optimization of Common Table Expressions in MPP Database Systems
《Optimization of Common Table Expressions in MPP Database Systems》论文导读
|
机器学习/深度学习 运维 Ubuntu
Elastic:机器学习的实践 - single metric job
在 Elasticsearch 中,可以将机器学习视为搜索和分析的自然扩展。它是对时间序列数据的分析。 Elasticsearch 支持的机器学习功能可以通过运行 metric 任务来自动分析时间序列数据,该 metric 任务包含一个或多个定义了将要分析的字段的检测器。 它可以帮助我们识别单变量时间序列数据中的异常,并向我们显示正常情况。在 Elasticsearch 中,我们可以通过机器学习来检测时间系列中的异常情况。
2790 0
Elastic:机器学习的实践 - single metric job
|
分布式计算 安全 MaxCompute
MaxCompute2.0 Empowers the Rapid Expansion of ZhongAn Insurance
At the Alibaba Cloud MaxCompute session during the 2017 Computing Conference held in Hangzhou.
1848 0
|
SQL 分布式计算 Hadoop
Optimizing Complex Data Distribution in MaxCompute
For a long time, data distribution has been an issue in the field of Big Data processing. Unfortunately, the Big Data processing systems that are popular today do not satisfactorily solve the issue.
1817 0