AI 助理
备案控制台
开发者社区 数据库 文章 正文

【大数据开发运维解决方案】Oracle通过sqoop同步数据到hive

2023-03-24 471
版权
版权声明：
本文内容由阿里云实名注册用户自发贡献，版权归原作者所有，阿里云开发者社区不拥有其著作权，亦不承担相应法律责任。具体规则请查看《 阿里云开发者社区用户服务协议》和 《阿里云开发者社区知识产权保护指引》。如果您发现本社区中有涉嫌抄袭的内容，填写 侵权投诉表单进行举报，一经查实，本社区将立刻删除涉嫌侵权内容。
本文涉及的产品
注册配置 MSE Nacos/ZooKeeper，118元/月
云原生大数据计算服务 MaxCompute，5000CU*H 100GB 3个月
云原生网关 MSE Higress，422元/月
简介： 将关系型数据库ORACLE的数据导入到HDFS中，可以通过Sqoop、OGG来实现，相比较ORACLE GOLDENGATE，Sqoop不仅不需要复杂的安装配置，而且传输效率很高，同时也能实现增量数据同步。本文档将在以上两个文章的基础上操作，是对第二篇文章环境的一个简单使用测试，使用过程中出现的错误亦可以验证暴漏第二篇文章安装的问题出现的错误,至于sqoop增量同步到hive请看本人在这篇文章之后写的测试文档。

一、介绍

将关系型数据库ORACLE的数据导入到HDFS中，可以通过Sqoop、OGG来实现，相比较ORACLE GOLDENGATE，Sqoop不仅不需要复杂的安装配置，而且传输效率很高，同时也能实现增量数据同步。
本文档将在以上两个文章的基础上操作，是对第二篇文章环境的一个简单使用测试，使用过程中出现的错误亦可以验证暴漏第二篇文章安装的问题出现的错误,至于sqoop增量同步到hive请看本人在这篇文章之后写的测试文档。

二、环境配置

image.png

三、实验过程

1、Oracle源端创建测试用表并初始化

--scott用户下创建此表
create table ora_hive(
empno number  primary key,
ename varchar2(30),
hiredate date
);
--简单初始化出近1000天数据
insert into ora_hive
  select level, dbms_random.string('u', 20), sysdate - level
    from dual
  connect by level <= 1000;
commit;
--现在表中存在2019，2018，2017，2016四年的数据

2、hive创建目标表

切换到hive目录：
[root@hadoop ~]# cd /hadoop/hive/
[root@hadoop hive]# cd bin
[root@hadoop bin]# ./hive
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/hadoop/hive/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/imp
l/StaticLoggerBinder.class]SLF4J: Found binding in [jar:file:/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/
org/slf4j/impl/StaticLoggerBinder.class]SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]

Logging initialized using configuration in jar:file:/hadoop/hive/lib/hive-common-2.3.2.jar!/
hive-log4j2.properties Async: trueHive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider
 using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.hive> show databases;
OK
default
sbux
Time taken: 1.508 seconds, Fetched: 2 row(s)
hive> create database oracle;
OK
Time taken: 1.729 seconds
hive> show databases;
OK
default
oracle
sbux
Time taken: 0.026 seconds, Fetched: 3 row(s)
hive> use oracle;
OK
Time taken: 0.094 seconds
hive> create table ora_hive(
    > empno int,
    > ename string,
    > hiredate date
    > );
OK
Time taken: 0.744 seconds
hive> show tables;

3、导入数据：

[root@hadoop bin]# sqoop import --connect jdbc:oracle:thin:@192.168.1.6:1521:orcl --username scott --password tiger --table ORA_HIVE -m 1 --hive-import --hive-database oracle
Warning: /hadoop/sqoop/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
Warning: /hadoop/sqoop/../zookeeper does not exist! Accumulo imports will fail.
Please set $ZOOKEEPER_HOME to the root of your Zookeeper installation.
19/03/12 14:38:11 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7
19/03/12 14:38:11 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
19/03/12 14:38:11 INFO tool.BaseSqoopTool: Using Hive-specific delimiters for output. You can override
19/03/12 14:38:11 INFO tool.BaseSqoopTool: delimiters with --fields-terminated-by, etc.
19/03/12 14:38:12 INFO oracle.OraOopManagerFactory: Data Connector for Oracle and Hadoop is disabled.
19/03/12 14:38:12 INFO manager.SqlManager: Using default fetchSize of 1000
19/03/12 14:38:12 INFO tool.CodeGenTool: Beginning code generation
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/hadoop/hbase/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/hadoop/hive/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
19/03/12 14:38:12 INFO manager.OracleManager: Time zone has been set to GMT
19/03/12 14:38:12 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM ORA_HIVE t WHERE 1=0
19/03/12 14:38:12 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /hadoop
Note: /tmp/sqoop-root/compile/e198fef468abdfcbc531a36035dd6646/ORA_HIVE.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
19/03/12 14:38:16 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-root/compile/e198fef468abdfcbc531a36035dd6646/ORA_HIVE.jar
19/03/12 14:38:16 INFO manager.OracleManager: Time zone has been set to GMT
19/03/12 14:38:16 INFO manager.OracleManager: Time zone has been set to GMT
19/03/12 14:38:16 INFO mapreduce.ImportJobBase: Beginning import of ORA_HIVE
19/03/12 14:38:17 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar
19/03/12 14:38:17 INFO manager.OracleManager: Time zone has been set to GMT
19/03/12 14:38:18 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
19/03/12 14:38:18 INFO client.RMProxy: Connecting to ResourceManager at /192.168.1.66:8032
19/03/12 14:38:21 INFO db.DBInputFormat: Using read commited transaction isolation
19/03/12 14:38:21 INFO mapreduce.JobSubmitter: number of splits:1
19/03/12 14:38:22 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1552371714699_0001
19/03/12 14:38:23 INFO impl.YarnClientImpl: Submitted application application_1552371714699_0001
19/03/12 14:38:23 INFO mapreduce.Job: The url to track the job: http://hadoop:8088/proxy/application_1552371714699_0001/
19/03/12 14:38:23 INFO mapreduce.Job: Running job: job_1552371714699_0001
19/03/12 14:38:35 INFO mapreduce.Job: Job job_1552371714699_0001 running in uber mode : false
19/03/12 14:38:35 INFO mapreduce.Job:  map 0% reduce 0%
19/03/12 14:38:42 INFO mapreduce.Job:  map 100% reduce 0%
19/03/12 14:38:43 INFO mapreduce.Job: Job job_1552371714699_0001 completed successfully
19/03/12 14:38:44 INFO mapreduce.Job: Counters: 30
    File System Counters
        FILE: Number of bytes read=0
        FILE: Number of bytes written=140340
        FILE: Number of read operations=0
        FILE: Number of large read operations=0
        FILE: Number of write operations=0
        HDFS: Number of bytes read=87
        HDFS: Number of bytes written=46893
        HDFS: Number of read operations=4
        HDFS: Number of large read operations=0
        HDFS: Number of write operations=2
    Job Counters 
        Launched map tasks=1
        Other local map tasks=1
        Total time spent by all maps in occupied slots (ms)=4625
        Total time spent by all reduces in occupied slots (ms)=0
        Total time spent by all map tasks (ms)=4625
        Total vcore-milliseconds taken by all map tasks=4625
        Total megabyte-milliseconds taken by all map tasks=4736000
    Map-Reduce Framework
        Map input records=1000
        Map output records=1000
        Input split bytes=87
        Spilled Records=0
        Failed Shuffles=0
        Merged Map outputs=0
        GC time elapsed (ms)=101
        CPU time spent (ms)=2730
        Physical memory (bytes) snapshot=203558912
        Virtual memory (bytes) snapshot=2137956352
        Total committed heap usage (bytes)=100139008
    File Input Format Counters 
        Bytes Read=0
    File Output Format Counters 
        Bytes Written=46893
19/03/12 14:38:44 INFO mapreduce.ImportJobBase: Transferred 45.7939 KB in 25.921 seconds (1.7667 KB/sec)
19/03/12 14:38:44 INFO mapreduce.ImportJobBase: Retrieved 1000 records.
19/03/12 14:38:44 INFO mapreduce.ImportJobBase: Publishing Hive/Hcat import job data to Listeners for table ORA_HIVE
19/03/12 14:38:44 INFO manager.OracleManager: Time zone has been set to GMT
19/03/12 14:38:44 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM ORA_HIVE t WHERE 1=0
19/03/12 14:38:44 WARN hive.TableDefWriter: Column EMPNO had to be cast to a less precise type in Hive
19/03/12 14:38:44 WARN hive.TableDefWriter: Column HIREDATE had to be cast to a less precise type in Hive
19/03/12 14:38:44 INFO hive.HiveImport: Loading uploaded data into Hive
19/03/12 14:38:44 INFO conf.HiveConf: Found configuration file file:/hadoop/hive/conf/hive-site.xml
2019-03-12 06:38:47,077 main ERROR Could not register mbeans java.security.AccessControlException: access denied ("javax.management.MBeanTrustPermission" "register")
    at java.security.AccessControlContext.checkPermission(AccessControlContext.java:472)
    at java.lang.SecurityManager.checkPermission(SecurityManager.java:585)
    at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.checkMBeanTrustPermission(DefaultMBeanServerInterceptor.java:1848)
    at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:322)
    at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
    at org.apache.logging.log4j.core.jmx.Server.register(Server.java:380)
    at org.apache.logging.log4j.core.jmx.Server.reregisterMBeansAfterReconfigure(Server.java:165)
    at org.apache.logging.log4j.core.jmx.Server.reregisterMBeansAfterReconfigure(Server.java:138)
    at org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:507)
    at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:249)
    at org.apache.logging.log4j.core.async.AsyncLoggerContext.start(AsyncLoggerContext.java:86)
    at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:239)
    at org.apache.logging.log4j.core.config.Configurator.initialize(Configurator.java:157)
    at org.apache.logging.log4j.core.config.Configurator.initialize(Configurator.java:130)
    at org.apache.logging.log4j.core.config.Configurator.initialize(Configurator.java:100)
    at org.apache.logging.log4j.core.config.Configurator.initialize(Configurator.java:187)
    at org.apache.hadoop.hive.common.LogUtils.initHiveLog4jDefault(LogUtils.java:154)
    at org.apache.hadoop.hive.common.LogUtils.initHiveLog4jCommon(LogUtils.java:90)
    at org.apache.hadoop.hive.common.LogUtils.initHiveLog4jCommon(LogUtils.java:82)
    at org.apache.hadoop.hive.common.LogUtils.initHiveLog4j(LogUtils.java:65)
    at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:702)
    at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:686)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.sqoop.hive.HiveImport.executeScript(HiveImport.java:331)
    at org.apache.sqoop.hive.HiveImport.importTable(HiveImport.java:241)
    at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:537)
    at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:628)
    at org.apache.sqoop.Sqoop.run(Sqoop.java:147)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
    at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)
    at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234)
    at org.apache.sqoop.Sqoop.runTool(Sqoop.java:243)
    at org.apache.sqoop.Sqoop.main(Sqoop.java:252)


Logging initialized using configuration in jar:file:/hadoop/hive/lib/hive-common-2.3.2.jar!/hive-log4j2.properties Async: true
19/03/12 14:38:47 INFO SessionState: 
Logging initialized using configuration in jar:file:/hadoop/hive/lib/hive-common-2.3.2.jar!/hive-log4j2.properties Async: true
19/03/12 14:38:47 INFO session.SessionState: Created HDFS directory: /tmp/hive/root/0a864287-81f1-47f6-80fb-8ea3cf0b2faf
19/03/12 14:38:47 INFO session.SessionState: Created local directory: /hadoop/hive/tmp/root/0a864287-81f1-47f6-80fb-8ea3cf0b2faf
19/03/12 14:38:47 INFO session.SessionState: Created HDFS directory: /tmp/hive/root/0a864287-81f1-47f6-80fb-8ea3cf0b2faf/_tmp_space.db
19/03/12 14:38:47 INFO conf.HiveConf: Using the default value passed in for log id: 0a864287-81f1-47f6-80fb-8ea3cf0b2faf
19/03/12 14:38:47 INFO session.SessionState: Updating thread name to 0a864287-81f1-47f6-80fb-8ea3cf0b2faf main
19/03/12 14:38:47 INFO conf.HiveConf: Using the default value passed in for log id: 0a864287-81f1-47f6-80fb-8ea3cf0b2faf
19/03/12 14:38:47 INFO ql.Driver: Compiling command(queryId=root_20190312063847_496038e3-f0ee-4f95-b35b-bfd1746cfd09): CREATE TABLE IF NOT EXISTS `oracle`.`ORA_HIVE` ( `EMPNO` DOUBLE, `ENAM
E` STRING, `HIREDATE` STRING) COMMENT 'Imported by sqoop on 2019/03/12 06:38:44' ROW FORMAT DELIMITED FIELDS TERMINATED BY '\001' LINES TERMINATED BY '\012' STORED AS TEXTFILE19/03/12 14:38:50 INFO hive.metastore: Trying to connect to metastore with URI thrift://192.168.1.66:9083
19/03/12 14:38:50 INFO hive.metastore: Opened a connection to metastore, current connections: 1
19/03/12 14:38:50 INFO hive.metastore: Connected to metastore.
19/03/12 14:38:50 INFO parse.CalcitePlanner: Starting Semantic Analysis
19/03/12 14:38:50 INFO parse.CalcitePlanner: Creating table oracle.ORA_HIVE position=27
19/03/12 14:38:51 INFO ql.Driver: Semantic Analysis Completed
19/03/12 14:38:51 INFO ql.Driver: Returning Hive schema: Schema(fieldSchemas:null, properties:null)
19/03/12 14:38:51 INFO ql.Driver: Completed compiling command(queryId=root_20190312063847_496038e3-f0ee-4f95-b35b-bfd1746cfd09); Time taken: 3.385 seconds
19/03/12 14:38:51 INFO ql.Driver: Concurrency mode is disabled, not creating a lock manager
19/03/12 14:38:51 INFO ql.Driver: Executing command(queryId=root_20190312063847_496038e3-f0ee-4f95-b35b-bfd1746cfd09): CREATE TABLE IF NOT EXISTS `oracle`.`ORA_HIVE` ( `EMPNO` DOUBLE, `ENAM
E` STRING, `HIREDATE` STRING) COMMENT 'Imported by sqoop on 2019/03/12 06:38:44' ROW FORMAT DELIMITED FIELDS TERMINATED BY '\001' LINES TERMINATED BY '\012' STORED AS TEXTFILE19/03/12 14:38:51 INFO sqlstd.SQLStdHiveAccessController: Created SQLStdHiveAccessController for session context : HiveAuthzSessionContext [sessionString=0a864287-81f1-47f6-80fb-8ea3cf0b2fa
f, clientType=HIVECLI]19/03/12 14:38:51 WARN session.SessionState: METASTORE_FILTER_HOOK will be ignored, since hive.security.authorization.manager is set to instance of HiveAuthorizerFactory.
19/03/12 14:38:51 INFO hive.metastore: Mestastore configuration hive.metastore.filter.hook changed from org.apache.hadoop.hive.metastore.DefaultMetaStoreFilterHookImpl to org.apache.hadoop.
hive.ql.security.authorization.plugin.AuthorizationMetaStoreFilterHook19/03/12 14:38:51 INFO hive.metastore: Closed a connection to metastore, current connections: 0
19/03/12 14:38:51 INFO hive.metastore: Trying to connect to metastore with URI thrift://192.168.1.66:9083
19/03/12 14:38:51 INFO hive.metastore: Opened a connection to metastore, current connections: 1
19/03/12 14:38:51 INFO hive.metastore: Connected to metastore.
19/03/12 14:38:51 INFO ql.Driver: Completed executing command(queryId=root_20190312063847_496038e3-f0ee-4f95-b35b-bfd1746cfd09); Time taken: 0.134 seconds
OK
19/03/12 14:38:51 INFO ql.Driver: OK
Time taken: 3.534 seconds
19/03/12 14:38:51 INFO CliDriver: Time taken: 3.534 seconds
19/03/12 14:38:51 INFO conf.HiveConf: Using the default value passed in for log id: 0a864287-81f1-47f6-80fb-8ea3cf0b2faf
19/03/12 14:38:51 INFO session.SessionState: Resetting thread name to  main
19/03/12 14:38:51 INFO conf.HiveConf: Using the default value passed in for log id: 0a864287-81f1-47f6-80fb-8ea3cf0b2faf
19/03/12 14:38:51 INFO session.SessionState: Updating thread name to 0a864287-81f1-47f6-80fb-8ea3cf0b2faf main
19/03/12 14:38:51 INFO ql.Driver: Compiling command(queryId=root_20190312063851_88f08fef-b111-439a-ac51-62b91c507927): 
LOAD DATA INPATH 'hdfs://192.168.1.66:9000/user/root/ORA_HIVE' INTO TABLE `oracle`.`ORA_HIVE`
19/03/12 14:38:51 INFO ql.Driver: Semantic Analysis Completed
19/03/12 14:38:51 INFO ql.Driver: Returning Hive schema: Schema(fieldSchemas:null, properties:null)
19/03/12 14:38:51 INFO ql.Driver: Completed compiling command(queryId=root_20190312063851_88f08fef-b111-439a-ac51-62b91c507927); Time taken: 0.451 seconds
19/03/12 14:38:51 INFO ql.Driver: Concurrency mode is disabled, not creating a lock manager
19/03/12 14:38:51 INFO ql.Driver: Executing command(queryId=root_20190312063851_88f08fef-b111-439a-ac51-62b91c507927): 
LOAD DATA INPATH 'hdfs://192.168.1.66:9000/user/root/ORA_HIVE' INTO TABLE `oracle`.`ORA_HIVE`
19/03/12 14:38:51 INFO ql.Driver: Starting task [Stage-0:MOVE] in serial mode
19/03/12 14:38:51 INFO hive.metastore: Closed a connection to metastore, current connections: 0
Loading data to table oracle.ora_hive
19/03/12 14:38:51 INFO exec.Task: Loading data to table oracle.ora_hive from hdfs://192.168.1.66:9000/user/root/ORA_HIVE
19/03/12 14:38:51 INFO hive.metastore: Trying to connect to metastore with URI thrift://192.168.1.66:9083
19/03/12 14:38:51 INFO hive.metastore: Opened a connection to metastore, current connections: 1
19/03/12 14:38:51 INFO hive.metastore: Connected to metastore.
19/03/12 14:38:51 ERROR hdfs.KeyProviderCache: Could not find uri with key [dfs.encryption.key.provider.uri] to create a keyProvider !!
19/03/12 14:38:52 ERROR exec.TaskRunner: Error in executeTask
java.lang.NoSuchMethodError: com.fasterxml.jackson.databind.ObjectMapper.readerFor(Ljava/lang/Class;)Lcom/fasterxml/jackson/databind/ObjectReader;
    at org.apache.hadoop.hive.common.StatsSetupConst$ColumnStatsAccurate.<clinit>(StatsSetupConst.java:165)
    at org.apache.hadoop.hive.common.StatsSetupConst.parseStatsAcc(StatsSetupConst.java:300)
    at org.apache.hadoop.hive.common.StatsSetupConst.clearColumnStatsState(StatsSetupConst.java:261)
    at org.apache.hadoop.hive.ql.metadata.Hive.loadTable(Hive.java:2032)
    at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:360)
    at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:199)
    at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100)
    at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2183)
    at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1839)
    at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1526)
    at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1237)
    at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1227)
    at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:233)
    at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:184)
    at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:403)
    at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:336)
    at org.apache.hadoop.hive.cli.CliDriver.processReader(CliDriver.java:474)
    at org.apache.hadoop.hive.cli.CliDriver.processFile(CliDriver.java:490)
    at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:793)
    at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:759)
    at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:686)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.sqoop.hive.HiveImport.executeScript(HiveImport.java:331)
    at org.apache.sqoop.hive.HiveImport.importTable(HiveImport.java:241)
    at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:537)
    at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:628)
    at org.apache.sqoop.Sqoop.run(Sqoop.java:147)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
    at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)
    at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234)
    at org.apache.sqoop.Sqoop.runTool(Sqoop.java:243)
    at org.apache.sqoop.Sqoop.main(Sqoop.java:252)
FAILED: Execution Error, return code -101 from org.apache.hadoop.hive.ql.exec.MoveTask. com.fasterxml.jackson.databind.ObjectMapper.readerFor(Ljava/lang/Class;)Lcom/fasterxml/jackson/databi
nd/ObjectReader;19/03/12 14:38:52 ERROR ql.Driver: FAILED: Execution Error, return code -101 from org.apache.hadoop.hive.ql.exec.MoveTask. com.fasterxml.jackson.databind.ObjectMapper.readerFor(Ljava/lang/C
lass;)Lcom/fasterxml/jackson/databind/ObjectReader;19/03/12 14:38:52 INFO ql.Driver: Completed executing command(queryId=root_20190312063851_88f08fef-b111-439a-ac51-62b91c507927); Time taken: 0.497 seconds
19/03/12 14:38:52 INFO conf.HiveConf: Using the default value passed in for log id: 0a864287-81f1-47f6-80fb-8ea3cf0b2faf
19/03/12 14:38:52 INFO session.SessionState: Resetting thread name to  main
19/03/12 14:38:52 INFO conf.HiveConf: Using the default value passed in for log id: 0a864287-81f1-47f6-80fb-8ea3cf0b2faf
19/03/12 14:38:52 INFO session.SessionState: Deleted directory: /tmp/hive/root/0a864287-81f1-47f6-80fb-8ea3cf0b2faf on fs with scheme hdfs
19/03/12 14:38:52 INFO session.SessionState: Deleted directory: /hadoop/hive/tmp/root/0a864287-81f1-47f6-80fb-8ea3cf0b2faf on fs with scheme file
19/03/12 14:38:52 INFO hive.metastore: Closed a connection to metastore, current connections: 0
19/03/12 14:38:52 ERROR tool.ImportTool: Import failed: java.io.IOException: Hive CliDriver exited with status=-101
    at org.apache.sqoop.hive.HiveImport.executeScript(HiveImport.java:355)
    at org.apache.sqoop.hive.HiveImport.importTable(HiveImport.java:241)
    at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:537)
    at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:628)
    at org.apache.sqoop.Sqoop.run(Sqoop.java:147)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
    at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)
    at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234)
    at org.apache.sqoop.Sqoop.runTool(Sqoop.java:243)
    at org.apache.sqoop.Sqoop.main(Sqoop.java:252)

发现报错，先看第一个报错内容：

2019-03-12 06:38:47,077 main ERROR Could not register mbeans java.security.AccessControlException: access denied ("javax.management.MBeanTrustPermission" "register")

这个报错内容是Oracle jdbc驱动连接数据库时报的错，解决办法为：

修改$JAVA_HOME/jre/lib/security/java.policy文件,添加一行:
permission javax.management.MBeanTrustPermission "register";

来看第二个错误：

java.lang.NoSuchMethodError: com.fasterxml.jackson.databind.ObjectMapper.readerFor(Ljava/lang/Class;)Lcom/fasterxml/jackson/databind/ObjectReader;

这个报错原因是hive+sqoop的jackson版本不一致导致的问题，解决办法为：
将$SQOOP_HOME/lib/jackson*.jar 文件bak，再把$HIVE_HOME/lib/jackson*.jar 拷贝替换至 $SQOOP_HOME/lib 目录中。

[root@hadoop bin]# cd /hadoop/sqoop/lib/
[root@hadoop lib]# mkdir /hadoop/bak
[root@hadoop lib]# mv jackson*.jar /hadoop/bak/
[root@hadoop lib]# cp $HIVE_HOME/lib/jackson*.jar .

这时候先删除一次已经创建的ORA_HIVE再执行一次导入：

[root@hadoop ~]# hadoop fs -rmr ORA_HIVE
rmr: DEPRECATED: Please use 'rm -r' instead.
19/03/12 14:54:21 INFO fs.TrashPolicyDefault: Namenode trash configuration: Deletion interval = 0 minut
es, Emptier interval = 0 minutes.Deleted ORA_HIVE
[root@hadoop ~]# sqoop import --connect jdbc:oracle:thin:@192.168.1.6:1521:orcl --username scott --password tiger --table ORA_HIVE -m 1 --hive-import --hive-database oracle
Warning: /hadoop/sqoop/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
Warning: /hadoop/sqoop/../zookeeper does not exist! Accumulo imports will fail.
Please set $ZOOKEEPER_HOME to the root of your Zookeeper installation.
19/03/12 14:55:21 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7
19/03/12 14:55:21 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
19/03/12 14:55:21 INFO tool.BaseSqoopTool: Using Hive-specific delimiters for output. You can override
19/03/12 14:55:21 INFO tool.BaseSqoopTool: delimiters with --fields-terminated-by, etc.
19/03/12 14:55:21 INFO oracle.OraOopManagerFactory: Data Connector for Oracle and Hadoop is disabled.
19/03/12 14:55:21 INFO manager.SqlManager: Using default fetchSize of 1000
19/03/12 14:55:21 INFO tool.CodeGenTool: Beginning code generation
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/hadoop/hbase/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/hadoop/hive/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
19/03/12 14:55:22 INFO manager.OracleManager: Time zone has been set to GMT
19/03/12 14:55:22 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM ORA_HIVE t WHERE 1=0
19/03/12 14:55:22 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /hadoop
Note: /tmp/sqoop-root/compile/a53ff4a6266b4f7f9f659e0a43fa9e7e/ORA_HIVE.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
19/03/12 14:55:24 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-root/compile/a53ff4a6266b4f7f9f659e0a43fa9e7e/ORA_HIVE.jar
19/03/12 14:55:24 INFO manager.OracleManager: Time zone has been set to GMT
19/03/12 14:55:24 INFO manager.OracleManager: Time zone has been set to GMT
19/03/12 14:55:24 INFO mapreduce.ImportJobBase: Beginning import of ORA_HIVE
19/03/12 14:55:25 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar
19/03/12 14:55:25 INFO manager.OracleManager: Time zone has been set to GMT
19/03/12 14:55:26 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
19/03/12 14:55:26 INFO client.RMProxy: Connecting to ResourceManager at /192.168.1.66:8032
19/03/12 14:55:29 INFO db.DBInputFormat: Using read commited transaction isolation
19/03/12 14:55:29 INFO mapreduce.JobSubmitter: number of splits:1
19/03/12 14:55:29 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1552371714699_0002
19/03/12 14:55:30 INFO impl.YarnClientImpl: Submitted application application_1552371714699_0002
19/03/12 14:55:30 INFO mapreduce.Job: The url to track the job: http://hadoop:8088/proxy/application_1552371714699_0002/
19/03/12 14:55:30 INFO mapreduce.Job: Running job: job_1552371714699_0002
19/03/12 14:55:39 INFO mapreduce.Job: Job job_1552371714699_0002 running in uber mode : false
19/03/12 14:55:39 INFO mapreduce.Job:  map 0% reduce 0%
19/03/12 14:55:46 INFO mapreduce.Job:  map 100% reduce 0%
19/03/12 14:55:47 INFO mapreduce.Job: Job job_1552371714699_0002 completed successfully
19/03/12 14:55:47 INFO mapreduce.Job: Counters: 30
    File System Counters
        FILE: Number of bytes read=0
        FILE: Number of bytes written=143639
        FILE: Number of read operations=0
        FILE: Number of large read operations=0
        FILE: Number of write operations=0
        HDFS: Number of bytes read=87
        HDFS: Number of bytes written=46893
        HDFS: Number of read operations=4
        HDFS: Number of large read operations=0
        HDFS: Number of write operations=2
    Job Counters 
        Launched map tasks=1
        Other local map tasks=1
        Total time spent by all maps in occupied slots (ms)=4765
        Total time spent by all reduces in occupied slots (ms)=0
        Total time spent by all map tasks (ms)=4765
        Total vcore-milliseconds taken by all map tasks=4765
        Total megabyte-milliseconds taken by all map tasks=4879360
    Map-Reduce Framework
        Map input records=1000
        Map output records=1000
        Input split bytes=87
        Spilled Records=0
        Failed Shuffles=0
        Merged Map outputs=0
        GC time elapsed (ms)=152
        CPU time spent (ms)=2670
        Physical memory (bytes) snapshot=201306112
        Virtual memory (bytes) snapshot=2138767360
        Total committed heap usage (bytes)=101187584
    File Input Format Counters 
        Bytes Read=0
    File Output Format Counters 
        Bytes Written=46893
19/03/12 14:55:47 INFO mapreduce.ImportJobBase: Transferred 45.7939 KB in 21.7019 seconds (2.1101 KB/sec)
19/03/12 14:55:47 INFO mapreduce.ImportJobBase: Retrieved 1000 records.
19/03/12 14:55:47 INFO mapreduce.ImportJobBase: Publishing Hive/Hcat import job data to Listeners for table ORA_HIVE
19/03/12 14:55:47 INFO manager.OracleManager: Time zone has been set to GMT
19/03/12 14:55:47 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM ORA_HIVE t WHERE 1=0
19/03/12 14:55:47 WARN hive.TableDefWriter: Column EMPNO had to be cast to a less precise type in Hive
19/03/12 14:55:47 WARN hive.TableDefWriter: Column HIREDATE had to be cast to a less precise type in Hive
19/03/12 14:55:47 INFO hive.HiveImport: Loading uploaded data into Hive
19/03/12 14:55:47 INFO conf.HiveConf: Found configuration file file:/hadoop/hive/conf/hive-site.xml

Logging initialized using configuration in jar:file:/hadoop/hive/lib/hive-common-2.3.2.jar!/hive-log4j2.properties Async: true
19/03/12 14:55:50 INFO SessionState: 
Logging initialized using configuration in jar:file:/hadoop/hive/lib/hive-common-2.3.2.jar!/hive-log4j2.properties Async: true
19/03/12 14:55:50 INFO session.SessionState: Created HDFS directory: /tmp/hive/root/4a5f4f66-31eb-4a12-95d2-bf22d45ecde4
19/03/12 14:55:50 INFO session.SessionState: Created local directory: /hadoop/hive/tmp/root/4a5f4f66-31eb-4a12-95d2-bf22d45ecde4
19/03/12 14:55:50 INFO session.SessionState: Created HDFS directory: /tmp/hive/root/4a5f4f66-31eb-4a12-95d2-bf22d45ecde4/_tmp_space.db
19/03/12 14:55:50 INFO conf.HiveConf: Using the default value passed in for log id: 4a5f4f66-31eb-4a12-95d2-bf22d45ecde4
19/03/12 14:55:50 INFO session.SessionState: Updating thread name to 4a5f4f66-31eb-4a12-95d2-bf22d45ecde4 main
19/03/12 14:55:50 INFO conf.HiveConf: Using the default value passed in for log id: 4a5f4f66-31eb-4a12-95d2-bf22d45ecde4
19/03/12 14:55:51 INFO ql.Driver: Compiling command(queryId=root_20190312065551_050b64c3-f5a1-4073-addd-a838c3585502): CREATE TABLE IF NOT EXISTS `oracle`.`ORA_HIVE` ( `EMPNO` DOUBLE, `ENAM
E` STRING, `HIREDATE` STRING) COMMENT 'Imported by sqoop on 2019/03/12 06:55:47' ROW FORMAT DELIMITED FIELDS TERMINATED BY '\001' LINES TERMINATED BY '\012' STORED AS TEXTFILE19/03/12 14:55:53 INFO hive.metastore: Trying to connect to metastore with URI thrift://192.168.1.66:9083
19/03/12 14:55:53 INFO hive.metastore: Opened a connection to metastore, current connections: 1
19/03/12 14:55:53 INFO hive.metastore: Connected to metastore.
19/03/12 14:55:53 INFO parse.CalcitePlanner: Starting Semantic Analysis
19/03/12 14:55:53 INFO parse.CalcitePlanner: Creating table oracle.ORA_HIVE position=27
19/03/12 14:55:53 INFO ql.Driver: Semantic Analysis Completed
19/03/12 14:55:53 INFO ql.Driver: Returning Hive schema: Schema(fieldSchemas:null, properties:null)
19/03/12 14:55:53 INFO ql.Driver: Completed compiling command(queryId=root_20190312065551_050b64c3-f5a1-4073-addd-a838c3585502); Time taken: 2.75 seconds
19/03/12 14:55:53 INFO ql.Driver: Concurrency mode is disabled, not creating a lock manager
19/03/12 14:55:53 INFO ql.Driver: Executing command(queryId=root_20190312065551_050b64c3-f5a1-4073-addd-a838c3585502): CREATE TABLE IF NOT EXISTS `oracle`.`ORA_HIVE` ( `EMPNO` DOUBLE, `ENAM
E` STRING, `HIREDATE` STRING) COMMENT 'Imported by sqoop on 2019/03/12 06:55:47' ROW FORMAT DELIMITED FIELDS TERMINATED BY '\001' LINES TERMINATED BY '\012' STORED AS TEXTFILE19/03/12 14:55:53 INFO sqlstd.SQLStdHiveAccessController: Created SQLStdHiveAccessController for session context : HiveAuthzSessionContext [sessionString=4a5f4f66-31eb-4a12-95d2-bf22d45ecde
4, clientType=HIVECLI]19/03/12 14:55:53 WARN session.SessionState: METASTORE_FILTER_HOOK will be ignored, since hive.security.authorization.manager is set to instance of HiveAuthorizerFactory.
19/03/12 14:55:53 INFO hive.metastore: Mestastore configuration hive.metastore.filter.hook changed from org.apache.hadoop.hive.metastore.DefaultMetaStoreFilterHookImpl to org.apache.hadoop.
hive.ql.security.authorization.plugin.AuthorizationMetaStoreFilterHook19/03/12 14:55:53 INFO hive.metastore: Closed a connection to metastore, current connections: 0
19/03/12 14:55:53 INFO hive.metastore: Trying to connect to metastore with URI thrift://192.168.1.66:9083
19/03/12 14:55:53 INFO hive.metastore: Opened a connection to metastore, current connections: 1
19/03/12 14:55:53 INFO hive.metastore: Connected to metastore.
19/03/12 14:55:53 INFO ql.Driver: Completed executing command(queryId=root_20190312065551_050b64c3-f5a1-4073-addd-a838c3585502); Time taken: 0.088 seconds
OK
19/03/12 14:55:53 INFO ql.Driver: OK
Time taken: 2.851 seconds
19/03/12 14:55:53 INFO CliDriver: Time taken: 2.851 seconds
19/03/12 14:55:53 INFO conf.HiveConf: Using the default value passed in for log id: 4a5f4f66-31eb-4a12-95d2-bf22d45ecde4
19/03/12 14:55:53 INFO session.SessionState: Resetting thread name to  main
19/03/12 14:55:53 INFO conf.HiveConf: Using the default value passed in for log id: 4a5f4f66-31eb-4a12-95d2-bf22d45ecde4
19/03/12 14:55:53 INFO session.SessionState: Updating thread name to 4a5f4f66-31eb-4a12-95d2-bf22d45ecde4 main
19/03/12 14:55:53 INFO ql.Driver: Compiling command(queryId=root_20190312065553_81be5e55-9c13-4ad8-86e0-3ec286bea2e0): 
LOAD DATA INPATH 'hdfs://192.168.1.66:9000/user/root/ORA_HIVE' INTO TABLE `oracle`.`ORA_HIVE`
19/03/12 14:55:54 INFO ql.Driver: Semantic Analysis Completed
19/03/12 14:55:54 INFO ql.Driver: Returning Hive schema: Schema(fieldSchemas:null, properties:null)
19/03/12 14:55:54 INFO ql.Driver: Completed compiling command(queryId=root_20190312065553_81be5e55-9c13-4ad8-86e0-3ec286bea2e0); Time taken: 0.414 seconds
19/03/12 14:55:54 INFO ql.Driver: Concurrency mode is disabled, not creating a lock manager
19/03/12 14:55:54 INFO ql.Driver: Executing command(queryId=root_20190312065553_81be5e55-9c13-4ad8-86e0-3ec286bea2e0): 
LOAD DATA INPATH 'hdfs://192.168.1.66:9000/user/root/ORA_HIVE' INTO TABLE `oracle`.`ORA_HIVE`
19/03/12 14:55:54 INFO ql.Driver: Starting task [Stage-0:MOVE] in serial mode
19/03/12 14:55:54 INFO hive.metastore: Closed a connection to metastore, current connections: 0
Loading data to table oracle.ora_hive
19/03/12 14:55:54 INFO exec.Task: Loading data to table oracle.ora_hive from hdfs://192.168.1.66:9000/user/root/ORA_HIVE
19/03/12 14:55:54 INFO hive.metastore: Trying to connect to metastore with URI thrift://192.168.1.66:9083
19/03/12 14:55:54 INFO hive.metastore: Opened a connection to metastore, current connections: 1
19/03/12 14:55:54 INFO hive.metastore: Connected to metastore.
19/03/12 14:55:54 ERROR hdfs.KeyProviderCache: Could not find uri with key [dfs.encryption.key.provider.uri] to create a keyProvider !!
19/03/12 14:55:55 INFO ql.Driver: Starting task [Stage-1:STATS] in serial mode
19/03/12 14:55:55 INFO exec.StatsTask: Executing stats task
19/03/12 14:55:55 INFO hive.metastore: Closed a connection to metastore, current connections: 0
19/03/12 14:55:55 INFO hive.metastore: Trying to connect to metastore with URI thrift://192.168.1.66:9083
19/03/12 14:55:55 INFO hive.metastore: Opened a connection to metastore, current connections: 1
19/03/12 14:55:55 INFO hive.metastore: Connected to metastore.
19/03/12 14:55:55 INFO hive.metastore: Closed a connection to metastore, current connections: 0
19/03/12 14:55:55 INFO hive.metastore: Trying to connect to metastore with URI thrift://192.168.1.66:9083
19/03/12 14:55:55 INFO hive.metastore: Opened a connection to metastore, current connections: 1
19/03/12 14:55:55 INFO hive.metastore: Connected to metastore.
19/03/12 14:55:55 INFO exec.StatsTask: Table oracle.ora_hive stats: [numFiles=2, numRows=0, totalSize=93786, rawDataSize=0]
19/03/12 14:55:55 INFO ql.Driver: Completed executing command(queryId=root_20190312065553_81be5e55-9c13-4ad8-86e0-3ec286bea2e0); Time taken: 1.186 seconds
OK
19/03/12 14:55:55 INFO ql.Driver: OK
Time taken: 1.6 seconds
19/03/12 14:55:55 INFO CliDriver: Time taken: 1.6 seconds
19/03/12 14:55:55 INFO conf.HiveConf: Using the default value passed in for log id: 4a5f4f66-31eb-4a12-95d2-bf22d45ecde4
19/03/12 14:55:55 INFO session.SessionState: Resetting thread name to  main
19/03/12 14:55:55 INFO conf.HiveConf: Using the default value passed in for log id: 4a5f4f66-31eb-4a12-95d2-bf22d45ecde4
19/03/12 14:55:55 INFO session.SessionState: Deleted directory: /tmp/hive/root/4a5f4f66-31eb-4a12-95d2-bf22d45ecde4 on fs with scheme hdfs
19/03/12 14:55:55 INFO session.SessionState: Deleted directory: /hadoop/hive/tmp/root/4a5f4f66-31eb-4a12-95d2-bf22d45ecde4 on fs with scheme file
19/03/12 14:55:55 INFO hive.metastore: Closed a connection to metastore, current connections: 0
19/03/12 14:55:55 INFO hive.HiveImport: Hive import complete.
19/03/12 14:55:55 INFO hive.HiveImport: Export directory is contains the _SUCCESS file only, removing the directory.

这次导入成功了，去hive验证一下：

[root@hadoop bin]# ./hive
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/hadoop/hive/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]

Logging initialized using configuration in jar:file:/hadoop/hive/lib/hive-common-2.3.2.jar!/hive-log4j2.properties Async: true
Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
hive> use oracle;
OK
Time taken: 1.111 seconds
hive> select * from ORA_HIVE;
。。。。。。。
997    XAOOEHXITLWEFBZFCNAB    NULL
998    IZVWHVGTEJHCJWJZTDXK    NULL
999    YMBFLJTTWPENEBXEWVIJ    NULL
1000    QFKDIGEYFWQBZBGTJPPD    NULL
Time taken: 2.092 seconds, Fetched: 2000 row(s)
已经导入进来了
文章标签：
云原生大数据计算服务 MaxCompute
微服务引擎
Java
SQL
关系型数据库
运维
Oracle
HIVE
数据库
大数据
数据库连接
关键词：
云原生大数据计算服务 MaxCompute数据
运维解决方案
Oracle数据库一键上云数据
云原生大数据计算服务 MaxCompute同步
开发运维
相关实践学习
基于MaxCompute的热门话题分析
本实验围绕社交用户发布的文章做了详尽的分析，通过分析能得到用户群体年龄分布，性别分布，地理位置分布，以及热门话题的热度。
SaaS 模式云数据仓库必修课
本课程由阿里云开发者社区和阿里云大数据团队共同出品，是SaaS模式云原生数据仓库领导者MaxCompute核心课程。本课程由阿里云资深产品和技术专家们从概念到方法，从场景到实践，体系化的将阿里巴巴飞天大数据平台10多年的经过验证的方法与实践深入浅出的讲给开发者们。帮助大数据开发者快速了解并掌握SaaS模式的云原生的数据仓库，助力开发者学习了解先进的技术栈，并能在实际业务中敏捷的进行大数据分析，赋能企业业务。 通过本课程可以了解SaaS模式云原生数据仓库领导者MaxCompute核心功能及典型适用场景，可应用MaxCompute实现数仓搭建，快速进行大数据分析。适合大数据工程师、大数据分析师 大量数据需要处理、存储和管理，需要搭建数据仓库？学它！ 没有足够人员和经验来运维大数据平台，不想自建IDC买机器，需要免运维的大数据平台？会SQL就等于会大数据？学它！ 想知道大数据用得对不对，想用更少的钱得到持续演进的数仓能力？获得极致弹性的计算资源和更好的性能，以及持续保护数据安全的生产环境？学它！ 想要获得灵活的分析能力，快速洞察数据规律特征？想要兼得数据湖的灵活性与数据仓库的成长性？学它！ 出品人：阿里云大数据产品及研发团队专家 产品 MaxCompute 官网 https://www.aliyun.com/product/odps&nbsp;
赵延东的一亩三分地
目录
相关文章
游客vsgxb64qlj7cg
|
18天前
|
机器学习/深度学习 人工智能 运维
智能化运维：AI与大数据在IT运维中的应用探索####
本文旨在探讨人工智能（AI）与大数据分析技术如何革新传统IT运维模式，提升运维效率与服务质量。通过具体案例分析，揭示AI算法在故障预测、异常检测及自动化修复等方面的实际应用成效，同时阐述大数据如何助力实现精准运维管理，降低运营成本，提升用户体验。文章还将简要讨论实施智能化运维面临的挑战与未来发展趋势，为IT管理者提供决策参考。 ####
游客vsgxb64qlj7cg
55 4
蓝易云
|
1月前
|
存储 运维 安全
Spring运维之boot项目多环境(yaml 多文件 proerties)及分组管理与开发控制
通过以上措施，可以保证Spring Boot项目的配置管理在专业水准上，并且易于维护和管理，符合搜索引擎收录标准。
蓝易云
42 2
131王
|
2月前
|
机器学习/深度学习 人工智能 运维
智能运维：大数据与AI的融合之道###
【10月更文挑战第20天】 运维领域正经历一场静悄悄的变革，大数据与人工智能的深度融合正重塑着传统的运维模式。本文探讨了智能运维如何借助大数据分析和机器学习算法，实现从被动响应到主动预防的转变，提升系统稳定性和效率的同时，降低了运维成本。通过实例解析，揭示智能运维在现代IT架构中的核心价值，为读者提供一份关于未来运维趋势的深刻洞察。 ###
131王
115 10
Dylaniou
|
2月前
|
运维 Java Linux
【运维基础知识】掌握VI编辑器：提升你的Java开发效率
本文详细介绍了VI编辑器的常用命令，包括模式切换、文本编辑、搜索替换及退出操作，帮助Java开发者提高在Linux环境下的编码效率。掌握这些命令，将使你在开发过程中更加得心应手。
Dylaniou
39 2
武子康
|
2月前
|
SQL 分布式计算 关系型数据库
Hadoop-21 Sqoop 数据迁移工具 简介与环境配置 云服务器 ETL工具 MySQL与Hive数据互相迁移 导入导出
Hadoop-21 Sqoop 数据迁移工具 简介与环境配置 云服务器 ETL工具 MySQL与Hive数据互相迁移 导入导出
武子康
82 3
ZCOM
|
2月前
|
存储 运维 监控
实时计算Flink版在稳定性、性能、开发运维、安全能力等等跟其他引擎及自建Flink集群比较。
实时计算Flink版在稳定性、性能、开发运维和安全能力等方面表现出色。其自研的高性能状态存储引擎GeminiStateBackend显著提升了作业稳定性，状态管理优化使性能提升40%以上。核心性能较开源Flink提升2-3倍，资源利用率提高100%。提供一站式开发管理、自动化运维和丰富的监控告警功能，支持多语言开发和智能调优。安全方面，具备访问控制、高可用保障和全链路容错能力，确保企业级应用的安全与稳定。
ZCOM
51 0
武子康
|
2月前
|
SQL 分布式计算 Java
大数据-96 Spark 集群 SparkSQL Scala编写SQL操作SparkSQL的数据源：JSON、CSV、JDBC、Hive
大数据-96 Spark 集群 SparkSQL Scala编写SQL操作SparkSQL的数据源：JSON、CSV、JDBC、Hive
武子康
59 0
武子康
|
2月前
|
SQL 分布式计算 关系型数据库
Hadoop-24 Sqoop迁移 MySQL到Hive 与 Hive到MySQL SQL生成数据 HDFS集群 Sqoop import jdbc ETL MapReduce
Hadoop-24 Sqoop迁移 MySQL到Hive 与 Hive到MySQL SQL生成数据 HDFS集群 Sqoop import jdbc ETL MapReduce
武子康
104 0
请看我回答~
|
4月前
|
运维 Devops 持续交付
自动化运维之路：从脚本到DevOps探索后端开发：从基础到高级实践
【8月更文挑战第28天】在数字化时代的浪潮中，企业对于IT运维的要求越来越高。从最初的手动执行脚本，到如今的自动化运维和DevOps实践，本文将带你领略运维的演变之旅。我们将探索如何通过编写简单的自动化脚本来提升效率，进而介绍DevOps文化的兴起及其对现代运维的影响。文章将为你揭示，通过持续集成、持续部署和微服务架构的实践，如何构建一个高效、可靠的运维体系。准备好让你的运维工作变得更加智能化和自动化了吗？让我们一起踏上这段旅程。 【8月更文挑战第28天】 本文旨在为初学者和有一定经验的开发者提供一个深入浅出的后端开发之旅。我们将一起探索后端开发的多个方面，包括语言选择、框架应用、数据库设计
请看我回答~
100 3
pbeskoyq7ffm4
|
4月前
|
运维 Oracle 前端开发
Oracle 11g RAC集群日常运维命令总结
Oracle 11g RAC集群日常运维命令总结
pbeskoyq7ffm4
109 2

热门文章

最新文章

  • 1
    丰富、连接、待集成—MaxCompute 生态再出发
  • 2
    日交易笔百万级，Ping++的大数据平台架构
  • 3
    计世资讯：用户对大数据安全分析需求逐渐迫切
  • 4
    科学家组建大数据平台解决科研资源共享难题
  • 5
    2020年M2M收入达160亿美金 运营商需加强“大数据”能力
  • 6
    Mysql5.7使用DTS增量同步数据到MaxCompute
  • 7
    云计算＆大数据 “下一幕”智能变革之力
  • 8
    北京探索“大数据”惠民 政府公布数据供社会开发
  • 9
    《离线和实时大数据开发实战》（二）大数据平台架构 & 技术概览1
  • 10
    MyBatis Plus 解决大数据量查询慢问题
  • 1
    解决Toad for Oracle显示乱中文码问题
    99
  • 2
    【服务器】python通过JDBC连接到位于Linux远程服务器上的Oracle数据库
    102
  • 3
    零基础入门 Oracle数据库：轻松上手
    220
  • 4
    java操作多数据源将oracle数据同步达梦数据库
    300
  • 5
    SuSE linux server 11通过SAP来安装oracle11g
    111
  • 6
    oracle11g SAP测试机归档日志暴增排查（二）
    338
  • 7
    oracle11g SAP测试机归档日志暴增排查（一）
    86
  • 8
    修改oracle11g的awr快照参数
    54
  • 9
    发现oracle10gSYSAUX空间没有了进行处理
    48
  • 10
    在centos7虚拟机上安装docker oracle11g
    295

    • 相关课程

    更多
  • 大数据Spark2020最新课程（知识精讲与实战演练）第二阶段
  • 大数据Spark2020版（知识精讲与实战演练）第三阶段
  • 大数据Spark2020版（知识精讲与实战演练）第四阶段
  • 2020版大数据实战项目之DMP广告系统（第四阶段）
  • 2020版大数据实战项目之DMP广告系统（第五阶段）
  • 2020版大数据实战项目之DMP广告系统（第六阶段）

    • 相关电子书

    更多
  • Data+AI时代大数据平台应该如何建设
  • 大数据AI一体化的解读
  • 极氪大数据 Serverless 应用实践

    • 相关实验场景

    更多
  • 基于Hologres+Flink搭建GitHub实时数据大屏
  • 基于Hologres轻松玩转一站式实时仓库
  • 使用操作系统智能助手OS Copilot解锁操作系统运维与编程
  • 基于OpenSearch向量检索版和MaxCompute快速搭建图搜服务
  • 倚天大数据电商数据分析快速实践
  • 玩转MaxCompute SQL! 30分钟搞定数据分析挖掘

    • 推荐镜像

    更多
  • oracle
  • mysql
  • postgresql
    • 下一篇
    DataWorks