What is the Actual Performance of HANA?
Introduction
I covered the topic of the actual performance of HANA versus competitive databases in the article Which if Faster, HANA or Oracle 12C. In this article I will cover the various database benchmarks on HANA and its competitors in more detail.
Who Performs Benchmark Testing in Databases?
The first thing to establish is that there is no independent body – such as a Consumer Reports for database benchmarking. This means that vendors performed the benchmarks that I reviewed. This is obviously a major issue.
Let us enumerate the problems with having no independent source for benchmarking as it relates to databases.
Selective Release: A vendor would never release a benchmark, which showed it as losing to a competing vendor across the board. The result of the benchmark would have to be positive for the vendor in some dimension, and more positive than negative for the results to be released. This brings up the issue that pharmaceutical companies drug testing shows that negative studies tend to go unpublished. “…studies about antidepressants made the drugs appear to work much better than they really did. Of 74 antidepressant studies registered with the FDA, 37 studies that showed positive results ended up being published. By contrast, studies that showed iffy or negative results mostly ended up going unpublished or had their data distorted to appear positive, Turner found. The missing or skewed studies helped create the impression that 94 percent of antidepressant trials had produced positive results, according to Turner's analysis, published in the New England Journal of Medicine. In reality, all the studies together showed just 51-percent positive results.” For instance, a past analysis of clinical trials supporting new drugs approved by the FDA showed that just 43 percent of more than 900 trials on 90 new drugs ended up being published. In other words, about 60 percent of the related studies remained unpublished even five years after the FDA had approved the drugs for market. That meant physicians were prescribing the drugs and patients were taking them without full knowledge of how well the treatments worked. - LifeScience. We will address this topic directly as it appears that SAP is doing the same thing with its OLTP benchmarks for HANA.
Skill Familiarity Bias: Because a vendor will always have more skills in their solutions than in a competitor solution and because databases can be “tuned up” and because of differences in hardware that is selected as well as a number of other differences, even if a vendor were 100% above board, they would still tend to observe better performance in their solution than a competing solution.
Hardware Bias: The vendors spare no expense in hardware for these tests. The customers will often purchase hardware that is lower in its specification than that used by the vendor.
The Laboratory Environment Bias: The hardware and database is run in a “lab” environment. It has no other batch jobs pulling its resources – which are of course unrealistic. Therefore the performance of the benchmark would normally not be attainable in a production setting. I see the benchmark results are more comparable between different benchmarks than between the benchmark and a production environment.
Sales Bias: Every benchmark paper I looked at had one clear purpose. That was to improve the sales of the product benchmarked by the vendor that wrote the paper.
Interpretational Bias: The benchmarks that are released are then viewed through the prism of bias. That is, people that have an incentive to prefer a particular software vendor. One entity that has published inaccurate information about benchmarks that have been right in line with their financial bias has been the consulting firm Bluefin, which is overall one of the least reliable providers of information on HANA.
The Benchmark Tests
The following benchmarks were reviewed that were performed for these databases.
SAP OLTP Benchmark: This is a benchmark for transaction processing. So things that ERP systems tend to do the most like recording journal entries, decrementing inventory when performing a goods issue, etc..
SAP BW-EML (Business Warehouse Mixed Workload) Benchmark: This is an analytics benchmark.
SAP’s Missing Benchmarks
For years SAP would release an OLTP benchmark for databases. However with HANA, SAP stopped releasing this benchmark. Database design would predict that HANA would perform poorly in this benchmark and this is the most likely reason why SAP never produced this benchmark. However, the consulting firm Bluefin has the following way of covering this up:
“The SAP HANA platform was designed to be a data platform on which to build the business applications of the future. One of the interesting impacts of this is that the benchmarks of the past (e.g. Sales/Distribution) were not the right metric by which to measure SAP HANA.” – Behind the SAP BW EML Benchmark
At no point in this article by John Appleby does he declare the fact that he has a quota, or leads a group with a quota to sell HANA. John Appleby presents himself as if he is some disinterested third party. So that is problem number one. But the second problem is that Appleby is speaking what amounts to gibberish in this quotation.
S4 has a Sales module.
This sales module will be performing the same functions as the current ECC SD module. Will there be analytics involved in the Sales module? Of course. However, there will also be transactions or OLTP performed.
S4 Sales will record sales orders, update sales orders, etc…
Therefore it is demonstrably untrue that an OLTP benchmark is now irrelevant because “the platform was designed to be a data platform on which to build business applications of the future.” That sentence is just a straight up lie, and it’s hard to twist oneself up into a pretzel to try to defend it. The person seems to be preparing to run for political office.
Appleby’s interpretation of the BW-EML benchmark contains other nonsense like
“the configuration used by published results is the stock installation…there are not performance constructs like additional indexes or aggregates in use.”
The reason this is nonsensical is that column-oriented databases don’t use indexes. They don’t need them. Why Appleby is impressed by this is a head scratcher. How many times has it been established that the primary reason for the reduction in the size of the database footprint is due to the removal of indexes? If so, and if this is widely accepted, why is it surprising to Appleby that the BW-EML benchmark for a column-oriented database does not have indexes???
On the topic of aggregates, HANA does use aggregates, but does not call them aggregates. So what Appleby is saying there is incorrect. Although there are fewer aggregates. Hasso Plattner has had an obsession with eliminating aggregates for some time and he rails against aggregates in his articles and his books, but in many cases aggregates are beneficial. Unlike what Hasso Plattner states, not everything needs to be constantly recalculated. And not everything needs to be recalculated every time it is accessed. This is just a waste of processing cycles. Let us take an example. Lets say we want to see a report of all the sales orders that a company has processed for the past 3 months. This report was processed and aggregated along different dimensional attributes yesterday. Under Hasso Plattner’s logic this aggregate is worthless because it is pre-calculated. However let us look at that statement in detail.
Let us say that the aggregate was calculated yesterday exactly 24 hours prior.
1 day is roughly 1/90th of a 3 month period.
If we look back 90 days in the report, we would show say 100,000 sales orders. That is an average of 1111 sales orders per day (yes, weekends would be less than work days, but as a average 1111 sales orders)
Now let us say that the day that drops off if we run the report anew had 1500 sales orders created (so a high day). And let us say that the day that was added, which is yesterday plus the hours up until the present hour are 700 sales orders (a low day).
So instead of looking at 100,000 sales orders, we are now looking at 99,200 sales orders. 1500 - 700 = 800 fewer sales orders. That is a change of .08% in the number of sales orders. Is that a real problem? Is the last 24 hours more representative than the 24 hour period from 3 months ago? Probably not. But if it is, how much more should the company be willing to spend to get rid of all aggregates? And are there other investments that might be a better use of that money?
There are an unlimited number of scenarios that could be imagined to determine the importance of the removal of aggregates. For instance, if just two days of sales orders were reviewed, then the company would receive a much larger variation. However generally, the needs for instantly recalculated information are greatly overestimated in vendor marketing documentation and in analytics vendor documentation in particular. I have a future article I am preparing which describes the testing of a long held belief that forecasting information must be frequently updated with the most recent sales history to obtain the highest forecast accuracy. I have been testing actual client data, and from a client with difficult to forecast sales history, and will show that as with the tests I performed at previous clients, this is actually not all that important and contributes little to forecast accuracy.
So, while there can be scenarios where getting the most up to date information is critical, SAP tends to take these few scenarios and generalize them to the be "normal," when in fact they tend to be the exceptions. Hasso Plattner has a way of presenting things that are often quite grey as black or white. And of course, all of Hasso Plattner’s examples have the peculiar and consistent outcome of handing over more money to SAP. I don't make more money if I can exaggerate the way that Hasso Plattner does, and therefore his proposals tend to come off as sales fluff...at least to me.
Logic for The Improved Analytical Performance of Column Oriented Databases
I found this quotation from IDC to be a very good explanation of my column-oriented databases is so effective for analytics.
"The established approach to setting up a query/reporting database (ODS, data mart, data warehouse) has involved establishing indexes for all columns that might have value lookup operations in the queries. Many organizations now use columnar databases, which have the same relational characteristics as row-oriented databases but store the data in blocks of column rather than row data for speed of retrieval. This obviates the need for indexes and, in some cases, for cubes and materialized views." - IDC
"If live data is to be queried and updated at the same time, the queries must be very fast in order to avoid consuming resources on the database server and slowing down transactions. A number of vendors have created database technologies that optimize query performance by combining two key elements: query-optimized columnar organization for the data and memory-optimized database operations. In the case discussed here, however, there is an additional challenge, which is to maintain that data in a form that also supports a high-performance transactional database." – IDC
“Database In-Memory leverages a unique “dual-format” architecture that enables tables to be in memory simultaneously in a traditional row format and a new in-memory column format. The Oracle SQL Optimizer automatically routes analytic queries to the column format and OLTP queries to the row format, transparently delivering best-of-both-worlds performance. Oracle Database 12 c automatically maintains full transactional consistency between the row and the column formats, just as it maintains consistency between tables and indexes today.
· Access only the columns that are needed.
· Scan and filter data in a compressed format.
· Prune out any unnecessary data within each column.
· Use SIMD to apply filter predicates.” – Oracle
However, this does not mean, and Oracle is not implying that a column-oriented database is better for applications outside of analytics. And as far as I can determine from reading the perspective of different database vendors on this topic, SAP is the only database vendor that proposes that a column oriented design is better for all types of applications.
Some of the Results
For instance in the Oracle benchmark paper released in 2015, the benchmark was tested on hardware similar to what SAP used in its BW-EML benchmark, but leaves out the topic of how many customers would use this hardware configuration. I don’t know myself as I have not recorded the hardware specification of many clients, but the hardware used by Oracle appeared quite advanced. At one point SAP’s benchmark used a machine with 1536 GB of RAM. I have personally never heard of this much RAM being used on a server at any account that I have worked on. It probably exists as there are very advanced companies out there doing scientific computing. But it is a small number. At one point Oracle points out that the monster machine used by SAP beat Oracle’s BW-EML benchmark, but needed 3 times the amount of memory to do this. Things bring up the question of whether SAP’s hardware was simply reengineered to beat the Oracle benchmark. So did SAP first try the machine with 1000 GB of RAM, and then add 200 GB or RAM and then test again, and then add another 200 and test again, etc until it finally beat the Oracle score? In another benchmark SAP installed 100 IBM servers in a SAP HANA cluster. Furthermore, if no one outside of the NSA, Amazon AWS (which resells portions of its hardware over the cloud) or a scientific computing center will be willing to buy this size of hardware how relevant are these benchmarks to the majority of HANA customers?
The Impact of Marketing on SAP Benchmarking
SAP needs to get marketing out of the process of releasing benchmarking information. In the benchmark publication SAP HANA Performance: Efficient Speed and Scale-Out for Real-Time Business Intelligence, I don’t need to see a cover plastered with stock photograph imagery of a man pulling a “fly” snowboarding maneuver, and then an image of a bunch of men rowing together, along with a marketing written introduction that uses a word salad of terms like NetWeaver components. This should be a scientific paper that is not word-smithed and couched in the deceptive marketing language. SAP marketing must acknowledge that not every paper produced by SAP needs have their fingerprints on it. This is the type of BS writing that I am referring to.
“The drill-down queries (276 to 483 milliseconds) demonstrate SAP HANA’s aggressive support for ad hoc joins and, therefore, to provide unrestricted ability for users to “slice and dice” with- out having to first involve the technical staff to provide indexes to support it (as would be the case with a conventional database).”
Please do not use the term “slice and dice” in a technical paper, or the term “unrestricted,” or the colorful “HANA’s aggressive support.” This is not scientific terminology. SAP’s benchmarking paper needs to be completely rewritten just using the original data. Then at the end SAP has quotations like the following:
“We have seen massive system speed improvements and increased ability to analyze the most detailed levels of customers and products.” – Colgate Palmolive
So this is an anecdote, and it sounds like it was written by Donald Trump (except it use the word massive instead of tremendous.) What is an anecdote doing in a benchmarking study!? Does SAP Marketing have any idea of what a study like this is actually supposed to contain?
Conclusion
When one compares what Bill McDermott, Hasso Plattner, SAP marketing, Bluefin, Deloitte and others say about the game changing aspects of HANA to the technical benchmarks there is absolutely no correspondence. SAP invests comparatively little in benchmarking, but its marketing spending on HANA is off the charts. This is reminiscent to pharmaceutical companies. Pharmaceutical companies spend far more on marketing than research, and the research is mostly just running clinical trials, which is based upon research that is performed by universities and is publicly funded. I call this the illusion of innovation.
Oracle has provided compelling evidence that its 12c database outperforms SAP HANA. I say this acknowledging the fact that there is no independent body that performs database benchmarking. Oracle invests much more into database benchmarking and its benchmarking studies are more transparent and make the case far better than SAP’s. For all of the talk of HANA’s performance, SAP produces a single benchmark to support these supposed claims of superiority over Oracle 12c and others. While we do not have independent verification, sifting through the results it seems more likely than not that Oracle 12c is not only a little bit faster, but far faster than HANA. And secondly, while SAP has placed speed as the first priority in the design of its database, Oracle’s orientation is far more holistic, placing reliability first. Secondly, given 12c’s design, it will almost certainly easily beat HANA for OLTP processing.
This article did not review the benchmarking of other database vendors. However, I find it more likely than not that vendors like Teradata, given the database talent that they have, not also have a solution that is superior to HANA in performance. And the list of other database vendors that can also beat HANA is likely more than just Oracle and Teradata.
I was not paid or otherwise compensated by any vendor or other entity to write this article.
References
Memory-Optimized Transactions and Analytics in One Platform: Achieving Business Agility with Oracle Database (IDC Sponsored by Oracle)
OracleVoice: Oracle Challenges SAP On In-Memory Database Claims, John Soat, Oracle.
Benchmark Results Reveal the Benefits of Oracle Database In-Memory for SAP Applications, Oracle White Paper, September 2015
http://www.livescience.com/8365-dark-side-medical-research-widespread-bias-omissions.html
http://www.livescience.com/5815-fraud-errors-misconceptions-medical-research.html
Behind the SAP BW EML Benchmark, John Appleby, Bluefin, March 19, 2015.
