Big Data is the new corporate currency. If used correctly, there is immense value to be extracted. Revenues from Big Data services, software and hardware are predicted to reach USD $187BN in 2019, representing an increase of more than 50 percent over a five-year period.
Much of this data will pass through the cloud, with 50 percent of organizations predicted to embrace a cloud-first policy in 2018 for Big Data and analytics. Enterprises are clearly demanding more flexibility and control over costs than on-premises solutions can deliver.
As the maturity of cloud-based technologies and the surge of Big Data converge, it is impossible to ignore the competitive edge a data processing and warehousing solution that is infinitely scalable and equally elastic brings to the enterprise. The tipping point for Big Data is here.
But why are we seeing a surge in cloud demand now? One major reason is the fact that technologies powering the cloud have not just increased in sophistication but concerns about security in the cloud have also diminished.
From complex, secured APIs to robust authentication and best practices, cloud platforms are investing in a range of features and support to ensure greater security and scalability. This strategy is paying off with the total number of organizations who distrust cloud dropping from 50 percent to 29 percent within just 12 months.
As a major cloud and big data infrastructure provider, Alibaba Cloud provides an expanding suite of cloud-based products to manage commercial big data problems, including Alibaba Cloud Data Integration, which has just recently been launched for the international market.
Data Integration is an all-in-one data synchronization platform that supports online real-time and offline data exchange between all data sources, networks, and locations. Based on an advanced distribution architecture with multiple modules (such as dirty data processing and flow control distributed system), the service provides data transmission, data conversion and synchronization services. It also supports multiple features, including support for multiple data sources, fast transmission, high reliability, scalability, and mass synchronization. Below we will take a closer look at the features and benefits of this new product and how your organization can add Data Integration to fulfill your Big Data processing needs.
Support for Multiple Disparate Data Sources
Data Integration supports data synchronization between more than 400 pairs of disparate data sources (including RDS databases, semi-structured storage, non-structured storage (such as audio, video, and images), NoSQL databases, and big data storage). This also includes important support for real-time data reading and writing between data sources such as Oracle, MySQL, and DataHub.
Scheduled Tasks
Data Integration allows you to schedule offline tasks by setting a specific trigger time (including year, month, day, hour, and minute). It only requires a few steps to configure periodical incremental data extraction. Data Integration works perfectly with DataWorks data modeling. The entire workflow is an integration of operations and maintenance.
Mass Upload to Cloud
Data Integration leverages the computing capability of Hadoop clusters to synchronize the HDFS data from clusters to MaxCompute, known as Mass Cloud Upload. Data Integration can transmit up to 5TB of data per day and the maximum transmission rate is 2GB/s.
Monitoring and Alarms
With 19 built-in monitoring rules, Data Integration applies to most monitoring scenarios. You can set alarm rules based on these monitoring rules. Additionally, you can pre-define the task failure notification mode for Data Integration.
Data Source Management
By leveraging the data sources and datasets that define the source and destination of data, Data Integration provides two data management plug-ins. The Reader plug-in is used to read data and the Writer plug-in is used to write data. Based on this framework, a set of simplified intermediate data transmission formats is developed to exchange data between arbitrary structured and semi-structured data sources.
Local Data Collection
Data Integration supports data synchronization in Alibaba Cloud classic networks and VPCs (virtual private cloud), as well as data collection in local IDCs.
Full Database Migration
Data Integration provides a full database migration tool which allows the creation of multiple data synchronization tasks and imports all data tables in a MySQL database to MaxCompute. By using full database migration, you no longer need to create synchronization tasks one at a time.
Incremental Synchronization
By using the WHERE clause, Data Integration supports business data filtering by date. Data with different dates is synchronized to the relevant MaxCompute partition tables. By setting the synchronization interval to 1 hour or 10 minutes, Data Integration is capable of performing quasi-real-time incremental synchronization.
To learn more about Data Integration, visit the product page at Alibaba Cloud today.
