Data is the currency of the digital world. How your organization stores, organizes, analyzes, and uses the data within its confines will largely determine how successful it is. Enterprises deal with large quantities of data, typically at petabyte scale, and they look to glean maximum value from all this data.
Cloud computing has been a game changer in this respect. What would be cost-prohibitive with traditional servers is now much more accessible with the economic and powerful solutions offered by cloud computing vendors.
Case in point: Data warehouse solutions hosted completely in the cloud. Thanks to cloud-based data lakes, what would have been impossible a few years ago is now made possible by the plummeting costs of data storage disks, and more powerful compute instances. This post explains how to use data warehouses in the cloud, and compares popular options on major public cloud platforms.
Data Warehouse Basics
A data warehouse is a centralized data store that’s used by multiple applications within your organization. If you’re looking to analyze small quantities of data that are a couple of GB in size, a data warehouse is too complex for your needs. A data warehouse makes sense only once you’ve scaled to a few hundred GB of data. At that point, you can’t function at the same speed and agility you used to, and you need a data warehouse.
The first thing to know about a data warehouse is that it is architected differently from small-scale database infrastructure. Rather than having databases that are restricted to hardware servers, a data warehouse is made of multiple servers that work together as a single unit.
Alibaba Cloud MaxCompute
Alibaba Cloud’s MaxCompute is a large-scale data analysis platform that is purpose-built for running big data workloads. Coming from the house of Alibaba, it was built out of necessity, when Alibaba needed a way to manage their ever-growing data that Oracle servers could no longer handle. It is battle-tested internally at Alibaba, where it has run a cluster of 10,000 server nodes. On a daily basis, 14,000 developers at Alibaba run three million jobs on it, and it stores 99% of all of Alibaba’s data. It is the first database service to scale to 100 TB of data at 7,000 BigBench Query-per-minute (BBQpm).
MaxCompute makes data migration simple with a variety of options. You can use Alibaba Cloud’s own tools like the MaxCompute client, or DataWorks, or even popular external tools like Flume, Logstash, or Fluentd. The uploaded data is stored in an SQL database, and can easily be scaled up to petabytes in size.
The most recent version of MaxCompute supports SQL 2.0, and interestingly allows for querying of unstructured data like images and video content. Despite the large quantities of data, and some of it being unstructured, MaxCompute is especially well-suited for real-time analysis. And the best part is that it is extremely easy to use and maintain. MaxCompute handles the difficulty of managing a distributed data store by having unique processes for clustering, indexing, and join optimization which all help with better data storage and retrieval at large scale.
With its recent US launch, MaxCompute is ready to change the way Big Data is processed across the world. With aggressive pricing, it is ready to take on similar services from the two other big cloud vendors—AWS and Azure.
AWS Redshift
AWS Redshift is one of the early services from the AWS stable. Similar to MaxCompute, it stores and analyzes data at petabyte scale. You can load data into Redshift using many AWS services such as S3, DynamoDB, or an SSH-enabled host on EC2. It leverages AWS IAM for security and access permissions. Further, you can encrypt your data using KMS, either on server-side or within the AWS cloud.
A unique feature of the service is Redshift Spectrum, which lets you query data that’s already in AWS S3. This means you don’t have to load your data into Redshift or transform your data. Instead, you can get to querying the data directly. However, if you’d rather have your data in Redshift and you have a lot of it, AWS Glue is an ETL service that makes data loading easy.
AWS recently announced new DC2 nodes which replace DC1 nodes at the same cost. They’re based on Intel’s Broadwell chips and offer twice the performance of the previous DC1 nodes and 30% better storage utilization.
With a variety of options for usage, AWS Redshift is an attractive option for data warehousing in the cloud.
Azure SQL Data Warehouse
Azure SQL data warehouse is the Big Data analysis solution from Microsoft. With Microsoft’s big footprint among the Fortune 500 enterprises, many of its customers would be interested in this service. Azure provides two flavors of this service—one optimized for elasticity, and the other optimized for compute. You could separate workloads across these two tiers, and it makes for an interesting choice. You can allocate or measure usage in the form of Data Warehouse Units (DWUs). There are two types of DWUs—a regular DWU, and a cDWU which is optimized for compute. Azure provides a level of service for each database which is measured in the form of Database Transaction Units (DTUs).
Azure has a tool called PolyBase, which is used to query external data without requiring the user to know Hadoop. PolyBase lets you import and export data to and from Hadoop, Azure Blob Storage, or Azure Data Lake Store, or query the data without moving it in and out of SQL Data Warehouse. SQL Data Warehouse is also well integrated with PowerShell, which lets you use scripting to automate common tasks.
Conclusion
In conclusion, all three data warehouse services mentioned here are powerful tools that take a different approach to the same challenge—analyzing big data in real time. If you have broader commitments that require you to choose Redshift or SQL Data Warehouse, it’s not a bad spot to be in. However, if you’re curious to try a powerful new option that is also cost-effective, MaxCompute is the way to go. Alibaba Cloud is offering a $300 credit for new users, making it easy to get a feel for what the platform has to offer before going all in. Try MaxCompute and start unlocking value from all your data in real time.
Bio
Twain Taylor
Twain began his career at Google, where, among other things, he was involved in technical support for the AdWords team. His work involved reviewing stack traces, and resolving issues affecting both customers and the Support team, and handling escalations. Later, he built branded social media applications, and automation scripts to help startups better manage their marketing operations. Today, as a technology journalist he helps IT magazines, and startups change the way teams build and ship applications
