冬季实战营第五期学习报告-阿里云开发者社区

冬季实战营第五期学习报告

2022-03-02 303

版权

本文内容由阿里云实名注册用户自发贡献，版权归原作者所有，阿里云开发者社区不拥有其著作权，亦不承担相应法律责任。具体规则请查看《阿里云开发者社区用户服务协议》和《阿里云开发者社区知识产权保护指引》。如果您发现本社区中有涉嫌抄袭的内容，填写侵权投诉表单进行举报，一经查实，本社区将立刻删除涉嫌侵权内容。

本文涉及的产品

Elasticsearch Serverless通用抵扣包，测试体验金 200元

简介： 第一个场景里u.txt文件各行分隔符不统一，有tab的，有空格的，导入以后大量空列，需要编辑后才能正常体验。

1 动手实战-基于EMR离线数据分析

1.1 创建资源，连接EMR集群

场景申请到的资源

屏幕截图(735).png

登陆ram子账号，找到主节点公网IP地址

屏幕截图(737).png

连接EMR集群，场景中的终端操作起来不太方便，使用本地putty终端也可以连接到主节点，完成后面的操作。

1.2 导入数据至EMR集群

在HDFS上创建目录，将编辑的文件放到HFDS文件系统上

[root@emr-header-1 ~]hdfs dfs -mkdir -p /data/student

[root@emr-header-1 ~]vim u.txt

[root@emr-header-1 ~] hdfs dfs -put u.txt /data/student

显示放入的文件和文件内容

[root@emr-header-1 ~]# hdfs dfs -ls /data/studentFound 1 items
-rw-r-----2 root hadoop       23912022-02-2809:30 /data/student/u.txt
[root@emr-header-1 ~]# hdfs dfs -cat /data/student/u.txt1962423881250949186302389171774222377187888711624451288060692316634618863975962984744884182806115265288117148825346558916284673054513886324817

登陆hive,创建表，导入数据

[root@emr-header-1 ~]# hiveLogging initialized using configuration in file:/etc/ecm/hive-conf-2.3.2-1.0.1/hive-log4j2.properties Async: trueHive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
    hive> CREATE TABLE emrusers (
        userid INT,
        movieid INT,
        rating INT,
        unixtime STRING )
        ROW FORMAT DELIMITED
        FIELDS TERMINATED BY '\t'        ;
    OK
    Time taken: 1.053 seconds
hive>  LOAD DATA INPATH '/data/student/u.txt' INTO TABLE emrusers;
    Loading data to table default.emrusers
    OK
    Time taken: 0.459 seconds

1.3 查询表，在表上运行统计分析sql语句

查看表的前五行数据，sql语句被转成了map-reduce任务，花费的时间较长。

hive>select*from emrusers limit5;    OK
196242388125094918630238917177422237718788871162445128806069231663461886397596Time taken:0.069 seconds, Fetched:5 row(s)

查询表的总行数，sql语句被转成了map-reduce任务，花费的时间较长。

hive>selectcount(*)from emrusers;    WARNING: Hive-on-MR is deprecated in Hive 2and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez)or using Hive 1.X releases.
    Query ID = root_20220228110103_9aec542e-2d15-49de-b0fe-388ee617b755
    Total jobs =1    Launching Job 1 out of 1    Number of reduce tasks determined at compile time:1Inorder to change the average load for a reducer (in bytes):set hive.exec.reducers.bytes.per.reducer=<number>Inorder to limit the maximum number of reducers:set hive.exec.reducers.max=<number>Inorder to set a constant number of reducers:set mapreduce.job.reduces=<number>    Starting Job = job_1646010854736_0005, Tracking URL = http://emr-header-1.cluster-286405:20888/proxy/application_1646010854736_0005/    Kill Command =/usr/lib/hadoop-current/bin/hadoop job  -kill job_1646010854736_0005
    Hadoop job information for Stage-1: number of mappers:1; number of reducers:12022-02-2811:01:11,438 Stage-1 map =0%,  reduce =0%2022-02-2811:01:16,722 Stage-1 map =100%,  reduce =0%, Cumulative CPU 0.99 sec
2022-02-2811:01:22,891 Stage-1 map =100%,  reduce =100%, Cumulative CPU 2.28 sec
    MapReduce Total cumulative CPU time:2 seconds 280 msec
    Ended Job = job_1646010854736_0005
    MapReduce Jobs Launched:    Stage-Stage-1: Map:1  Reduce:1   Cumulative CPU:2.28 sec   HDFS Read:10079 HDFS Write:103 SUCCESS
    Total MapReduce CPU Time Spent:2 seconds 280 msec
    OK
106Time taken:20.893 seconds, Fetched:1 row(s)

查询数据表中评级最高的三个电影，sql语句被转成了map-reduce任务，花费的时间较长。

hive>select movieid,sum(rating)as rat from emrusers groupby movieid orderby rat desclimit3;    WARNING: Hive-on-MR is deprecated in Hive 2and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez)or using Hive 1.X releases.
    Query ID = root_20220228110213_6733e92a-00ed-4d71-b289-5be55aaa26af
    Total jobs =2    Launching Job 1 out of 2    Number of reduce tasks not specified. Estimated from input data size:1Inorder to change the average load for a reducer (in bytes):set hive.exec.reducers.bytes.per.reducer=<number>Inorder to limit the maximum number of reducers:set hive.exec.reducers.max=<number>Inorder to set a constant number of reducers:set mapreduce.job.reduces=<number>    Starting Job = job_1646010854736_0006, Tracking URL = http://emr-header-1.cluster-286405:20888/proxy/application_1646010854736_0006/    Kill Command =/usr/lib/hadoop-current/bin/hadoop job  -kill job_1646010854736_0006
    Hadoop job information for Stage-1: number of mappers:1; number of reducers:12022-02-2811:02:21,418 Stage-1 map =0%,  reduce =0%2022-02-2811:02:25,532 Stage-1 map =100%,  reduce =0%, Cumulative CPU 1.0 sec
2022-02-2811:02:30,664 Stage-1 map =100%,  reduce =100%, Cumulative CPU 2.0 sec
    MapReduce Total cumulative CPU time:2 seconds 0 msec
    Ended Job = job_1646010854736_0006
    Launching Job 2 out of 2    Number of reduce tasks determined at compile time:1Inorder to change the average load for a reducer (in bytes):set hive.exec.reducers.bytes.per.reducer=<number>Inorder to limit the maximum number of reducers:set hive.exec.reducers.max=<number>Inorder to set a constant number of reducers:set mapreduce.job.reduces=<number>    Starting Job = job_1646010854736_0007, Tracking URL = http://emr-header-1.cluster-286405:20888/proxy/application_1646010854736_0007/    Kill Command =/usr/lib/hadoop-current/bin/hadoop job  -kill job_1646010854736_0007
    Hadoop job information for Stage-2: number of mappers:1; number of reducers:12022-02-2811:02:38,922 Stage-2 map =0%,  reduce =0%2022-02-2811:02:43,038 Stage-2 map =100%,  reduce =0%, Cumulative CPU 1.12 sec
2022-02-2811:02:48,162 Stage-2 map =100%,  reduce =100%, Cumulative CPU 2.14 sec
    MapReduce Total cumulative CPU time:2 seconds 140 msec
    Ended Job = job_1646010854736_0007
    MapReduce Jobs Launched:    Stage-Stage-1: Map:1  Reduce:1   Cumulative CPU:2.0 sec   HDFS Read:9642 HDFS Write:2131 SUCCESS
    Stage-Stage-2: Map:1  Reduce:1   Cumulative CPU:2.14 sec   HDFS Read:7869 HDFS Write:143 SUCCESS
    Total MapReduce CPU Time Spent:4 seconds 140 msec
    OK
14413274103049Time taken:36.114 seconds, Fetched:3 row(s)