默认分隔符\t
遇到的问题
hdfs上传文件报错(解决办法适用于之前上传的数据不重要)
[root@hadoop soft]# hdfs dfs -put test.txt /wordcount/input 2023-11-10 11:01:30,339 WARN hdfs.DataStreamer: DataStreamer Exception org.apache.hadoop.ipc.RemoteException(java.io.IOException): File /wordcount/input/wcdata.txt._COPYING_ could only be written to 0 of the 1 minReplication nodes. There are 0 datanode(s) running and 0 node(s) are excluded in this operation.
如果查看磁盘空间情况发现磁盘为0，那么你和我应该原因一样
https://stackoverflow.com/questions/11889261/datanode-process-not-running-in-hadoop
原因:多次执行NameNode格式化,导致namenode cluster id 和datanode cluster id 不相同。
查看日志
cat hadoop-root-datanode-hadoop.log
然后查看节点信息
hadoop dfsadmin -report
解决办法就是停止服务然后删除自己设置的数据保存路径
stop-all.sh
进入hadoop文件夹，然后查看你的文件保存路径core-site.xml中配置的hadoop.tmp.dir对应文件
cat etc/hadoop/core-site.xml
删除该路径下内容
然后格式化并启动服务
# 格式化文件系统 hdfs namenode -format start-all.sh
然后查看
hadoop dfsadmin -report
如果卡在INFO mapreduce.Job: Running job: job_1699694634855_0001
https://blog.csdn.net/m0_55901666/article/details/121987499
先看自己磁盘是否够，因为中间数据在磁盘保存，
然后在设置根据自己虚拟机情况设置内存，三台都需要
<property> <name>yarn.nodemanager.resource.memory-mb</name> <value>4096</value> </property>
改完记得重启------根据自己服务器情况(是否适合重启服务器)，如果不适合尝试其他方法
reboot
!!!!!也可能需要清理磁盘
# 查看磁盘 [root@hadoop2 ~]# df -h 文件系统 容量 已用 可用 已用% 挂载点 devtmpfs 1.9G 0 1.9G 0% /dev tmpfs 1.9G 28K 1.9G 1% /dev/shm tmpfs 1.9G 12M 1.9G 1% /run tmpfs 1.9G 0 1.9G 0% /sys/fs/cgroup /dev/mapper/centos-root 17G 17G 567M 97% / /dev/sda1 1014M 234M 781M 24% /boot tmpfs 378M 0 378M 0% /run/user/0 [root@hadoop2 ~]# cd / [root@hadoop2 /]# du -h -x --max-depth=1 38M ./etc 2.2G ./root 3.3G ./var 64K ./tmp 5.6G ./usr 18M ./home 0 ./media 0 ./mnt 5.3G ./opt 0 ./srv 17G . [root@hadoop2 /]# cd /usr/ [root@hadoop2 usr]# du -h -x --max-depth=1 350M ./bin 45M ./sbin 793M ./lib 175M ./lib64 370M ./share 0 ./etc 0 ./games 30M ./include 183M ./libexec 3.7G ./local 0 ./src 5.6G . [root@hadoop2 usr]# cd /lo -bash: cd: /lo: 没有那个文件或目录 [root@hadoop2 usr]# cd local/ [root@hadoop2 local]# ls bin etc gitlab include lib64 mysql python3 share develop games hadoop lib libexec openssl sbin src [root@hadoop2 local]# du -h -x --max-depth=1 0 ./bin 0 ./etc 0 ./games 0 ./include 0 ./lib 0 ./lib64 0 ./libexec 0 ./sbin 0 ./share 0 ./src 1.7G ./mysql 1.1G ./gitlab 643M ./develop 24M ./openssl 250M ./python3 1.9M ./hadoop 3.7G . [root@hadoop2 local]# cd gitlab/ [root@hadoop2 gitlab]# ls gitlab-ce-15.7.3-ce.0.el7.x86_64.rpm policycoreutils-python-2.5-22.amzn2.x86_64.rpm [root@hadoop2 gitlab]# cd .. [root@hadoop2 local]# rm -rf gitlab/ [root@hadoop2 local]# cd mysql/ [root@hadoop2 mysql]# ls mysql-8.0.31-1.el7.x86_64.rpm-bundle.tar mysql-community-client-8.0.31-1.el7.x86_64.rpm mysql-community-client-plugins-8.0.31-1.el7.x86_64.rpm mysql-community-common-8.0.31-1.el7.x86_64.rpm mysql-community-debuginfo-8.0.31-1.el7.x86_64.rpm mysql-community-devel-8.0.31-1.el7.x86_64.rpm mysql-community-embedded-compat-8.0.31-1.el7.x86_64.rpm mysql-community-icu-data-files-8.0.31-1.el7.x86_64.rpm mysql-community-libs-8.0.31-1.el7.x86_64.rpm mysql-community-libs-compat-8.0.31-1.el7.x86_64.rpm mysql-community-server-8.0.31-1.el7.x86_64.rpm mysql-community-server-debug-8.0.31-1.el7.x86_64.rpm mysql-community-test-8.0.31-1.el7.x86_64.rpm [root@hadoop2 mysql]# cd .. [root@hadoop2 local]# rm -rf mysql/ [root@hadoop2 local]# ls bin etc hadoop lib libexec python3 share develop games include lib64 openssl sbin src [root@hadoop2 local]# df -h 文件系统 容量 已用 可用 已用% 挂载点 devtmpfs 1.9G 0 1.9G 0% /dev tmpfs 1.9G 28K 1.9G 1% /dev/shm tmpfs 1.9G 16M 1.9G 1% /run tmpfs 1.9G 0 1.9G 0% /sys/fs/cgroup /dev/mapper/centos-root 17G 14G 3.4G 81% / /dev/sda1 1014M 234M 781M 24% /boot tmpfs 378M 0 378M 0% /run/user/0 [root@hadoop2 local]# cd /usr/local/ [root@hadoop2 local]# ls bin etc hadoop lib libexec python3 share develop games include lib64 openssl sbin src [root@hadoop2 local]# cd /opt/soft/ [root@hadoop2 soft]# ls apache-zookeeper-3.8.3-bin.tar.gz hadoop-3.3.5.tar.gz skywalking docker-compose jdk-8 wcdata.txt hadoop-3 jdk-8u391-linux-x64.tar.gz zookeeper-3 [root@hadoop2 soft]# rm apache-zookeeper-3.8.3-bin.tar.gz rm：是否删除普通文件 "apache-zookeeper-3.8.3-bin.tar.gz"？y [root@hadoop2 soft]# rm hadoop-3.3.5.tar.gz rm：是否删除普通文件 "hadoop-3.3.5.tar.gz"？y [root@hadoop2 soft]# rm jdk-8u391-linux-x64.tar.gz rm：是否删除普通文件 "jdk-8u391-linux-x64.tar.gz"？y [root@hadoop2 soft]# df -h 文件系统 容量 已用 可用 已用% 挂载点 devtmpfs 1.9G 0 1.9G 0% /dev tmpfs 1.9G 28K 1.9G 1% /dev/shm tmpfs 1.9G 17M 1.9G 1% /run tmpfs 1.9G 0 1.9G 0% /sys/fs/cgroup /dev/mapper/centos-root 17G 13G 4.2G 76% / /dev/sda1 1014M 234M 781M 24% /boot tmpfs 378M 0 378M 0% /run/user/0
org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot create directory /user/root/QuasiMonteCarlo_1699695721627_1361889091/in. Name node is in safe mode.
关闭安全模式
hdfs dfsadmin -safemode leave
ERROR conf.Configuration: error parsing conf core-site.xml com.ctc.wstx.exc.WstxParsingException: Illegal processing instruction target ("xml"); xml (case insensitive) is reserved by the specs
- ！注意xml配置头部不要出现空行
- 查看自己xml是否缺少部分</>
hadoop2: journalnode is running as process 60632. Stop it first and ensure /tmp/hadoop-root-journalnode.pid file is empty before retry.
启动集群时报错的，
问题原因：我在三个节点都启动start-all.sh了，只需要在一个节点启动即可
2023-11-11 22:11:15,224 WARN fs.FileSystem: Failed to initialize fileystem hdfs://hadoopService: java.lang.IllegalArgumentException: java.net.UnknownHostException: hadoopService-mkdir: java.net.UnknownHostException: hadoopService
这个是使用下列命令遇到的
hdfs dfs -mkdir -p /wordcount/input
问题原因：
hdfs.site.xml的这里配置错了，之前写的与服务名不一致
Browse Directory Operation category READ is not supported in state standby. Visit https://s.apache.org/sbnn-error
这是因为突然关机或者故障导致节点状态变成standby
先说原因
namenode一般是两个
- active--对外提供服务
- standby--对外不提供服务
这里可以看到我们的nn1节点状态变成standby了
我们只需要转换节点状态即可
hdfs haadmin -transitionToStandby --forcemanual nn2
然后查看
成功了
hadoop1: WARNING: resourcemanager did not stop gracefully after 5 seconds: Trying to kill with kill -9
查看日志,看见有webapp，不知道是不是这个报错
cd /opt/soft/hadoop-3/logs
Hadoop安装
单机---3.3.5版本
服务器配置
如果是vm虚拟机，那么涉及ip的都要一致，并且在范围内
下载解压
配置文件
！注意xml配置头部不要出现空行
配置文件路径
- hadoop-3(你的hadoop文件夹)/etc/hadoop
框内为你解压hadoop的路径
hadoop-env.sh
移至末尾添加如下
export JAVA_HOME=/opt/soft/jdk-8 # export set JAVA_OPTS="--add-opens java.base/java.lang=ALL-UNNAMED" export HDFS_NAMENODE_USER=root export HDFS_SECONDARYNAMENODE_USER=root export HDFS_DATANODE_USER=root export HDFS_ZKFC_USER=root export HDFS_JOURNALNODE_USER=root export HADOOP_SHELL_EXECNAME=root export YARN_RESOURCEMANAGER_USER=root export YARN_NODEMANAGER_USER=root
core-site.xml
默认
需要清空内容并覆盖，或者粘贴configuration
vim清空方式
命令模式下
- gg跳至文件首行
- dG删除光标所在行到末尾行内容
<?xml version="1.0" encoding="UTF-8"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <configuration> <property> <name>fs.defaultFS</name> <value>hdfs://hadoop:9000</value> </property> <property> <name>hadoop.tmp.dir</name> <value>/home/hadoop_data</value> </property> <property> <name>hadoop.http.staticuser.user</name> <value>root</value> </property> <property> <name>dfs.permissions.enabled</name> <value>false</value> </property> <property> <name>hadoop.proxyuser.root.hosts</name> <value>*</value> </property> <property> <name>hadoop.proxyuser.root.groups</name> <value>*</value> </property> </configuration>
hdfs.site.xml
<?xml version="1.0" encoding="UTF-8"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <configuration> <property> <name>dfs.replication</name> <value>1</value> </property> <property> <name>dfs.namenode.secondary.http-address</name> <value>hadoop:50090</value> </property> </configuration>
workers
注意：
hadoop2.x中该文件名为slaves
hadoop3.x中该文件名为workers
打开后默认
改为你的节点
mapred-site.xml
同core-site.xml，默认
<?xml version="1.0"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <configuration> <property> <name>mapreduce.job.tracker</name> <value>hdfs://hadoop:8001</value> <final>true</final> </property> <property> <name>mapreduce.application.classpath</name> <value>$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*</value> </property> </configuration>
yarn-site.xml
<?xml version="1.0"?> <configuration> <property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> </property> <property> <name>yarn.nodemanager.env-whitelist</name> <value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_HOME,PATH,LANG,TZ,HADOOP_MAPRED_HOME</value> </property> <!-- 内存 --> <property> <name>yarn.nodemanager.resource.memory-mb</name> <value>4096</value> </property> </configuration>
启动服务
第一次启动时一定要格式化文件系统，且不要重复执行: hdfs namenode -format
原因:多次执行NameNode格式化,导致namenode cluster id 和datanode cluster id 不相同。
依次启动
# 格式化文件系统 hdfs namenode -format # 启动 NameNode DataNode SecondaryNameNode start-dfs.sh # 启动 ResourceManager daemon 和 NodeManager start-yarn.sh # 启动 JobHistoryServer mapred --daemon start historyserver
统一启动
# 格式化文件系统 hdfs namenode -format start-all.sh stop-all.sh
关闭服务
# 关机之前 依关闭服务 # Hadoop102 mapred --daemon stop historyserver # hadoop103 stop-yarn.sh # hadoop101 stop-dfs.sh
访问页面
9870端口
ip:9870，这里是http://192.168.171.10:9870/
这里要查看你的datanodes，如果没有数据要考虑是否之前多次格式化文件了具体问题查看
博客开头的hdfs上传文件报错(解决办法适用于之前上传的数据不重要)
如果你需要域名例如http://hadoop:9870/
在本地hosts进行添加即可，url会先经过hosts的解析
192.168.171.10 hadoop
9864端口
19888端口
测试HDFS命令
hadoop jar $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.3.5.jar pi 10 10
这里可能遇到几个问题
- job卡住
- INFO mapreduce.Job: Running job: job_1699694634855_0001
- 解决办法，mapred-site.xml这个配置文件的mapreduce.job.tracker
集群--3.3.5版本
主要就是几点
- 主机名设置以及相互之间ssh连接
- 防火墙
- 配置文件的更改
集群服务器准备
我这里用三台192.168.80的
- 192.168.80.10---hadoop0
- 192.168.80.11---hadoop1
- 192.168.80.12---hadoop2
克隆虚拟机，并修改网卡信息
# 改ens33即可 vi /etc/sysconfig/network-scripts/ifcfg-ens33 vi /etc/sysconfig/network-scripts/ifcfg-ens34 cd /etc/sysconfig/network-scripts vi ifcfg-ens33 # 文件信息介绍 key value IPADDR 想要设定的ip地址，与虚拟机在相同网段 NETMASK 子网掩码 GATEWAY 网关，必须与前面的“网关IP”相同 BOOTPROTO 网络分配方式（有dhcp及static） ONBOOT 设置开机启动 # 克隆的记得改网卡的uuid,或者改最后一位数也可以 # 如果两个虚拟机网卡一致会导致克隆后的虚拟机无法访问 [root@localhost ~]# uuidgen 2738dea3-922f-4572-a3b1-8f3680924f34 # NAT # 只需要修改static，然后加上最下面的配置，如果是克隆的，记得把uuid改一下 TYPE=Ethernet PROXY_METHOD=none BROWSER_ONLY=no BOOTPROTO=static DEFROUTE=yes IPV4_FAILURE_FATAL=no IPV6INIT=yes IPV6_AUTOCONF=yes IPV6_DEFROUTE=yes IPV6_FAILURE_FATAL=no IPV6_ADDR_GEN_MODE=stable-privacy NAME=ens33 UUID=f37e0253-c7e2-4cdc-9a9a-3330758b71e8 DEVICE=ens33 ONBOOT=yes IPV6_PRIVACY=no AUTOCONNECT_PRIORITY=1 IPADDR=192.168.80.15 PREFIX=24 NETMASK=255.255.255.0 NETWORK=192.168.80.0 GATEWAY=192.168.80.2 DNS1=223.5.5.5 # 桥接 # 如果是克隆的，记得改uuid TYPE=Ethernet PROXY_METHOD=none BROWSER_ONLY=no BOOTPROTO=dhcp DEFROUTE=yes IPV4_FAILURE_FATAL=no IPV6INIT=yes IPV6_AUTOCONF=yes IPV6_DEFROUTE=yes IPV6_FAILURE_FATAL=no IPV6_ADDR_GEN_MODE=stable-privacy NAME=ens34 UUID=cca230b0-d9d3-4e11-b4b1-fa0870830458 DEVICE=ens34 ONBOOT=yes IPV6_PRIVACY=no AUTOCONNECT_PRIORITY=2 # 重启网络服务或者重启 service network restart # 其他常用命令 # 改名 vim /etc/hostname # 暂时改名 hostname aaa # 显示网卡信息 ip addr # 关机 poweroff # 重启 reboot
修改主机名
三台服务器分别改为
- hadoop0
- hadoop1
- hadoop2
vim /etc/hostname
修改hosts文件
三台都要改
vim /etc/hosts 192.168.80.10 hadoop0 192.168.80.11 hadoop1 192.168.80.12 hadoop2
1.搭建zookeeper集群
- 下载并解压安装
- 设置环境变量
- 配置zookeeper机器节点id
下载解压
三台都需要
wget https://dlcdn.apache.org/zookeeper/zookeeper-3.8.3/apache-zookeeper-3.8.3-bin.tar.gz # 这个是你保存的地方 cd /opt/soft tar -zxvf apache-zookeeper-3.8.3-bin.tar.gz mv apache-zookeeper-3.8.3-bin zookeeper-3
更改环境变量
三台都需要
vim /etc/profile.d/my_env.sh
export JAVA_HOME=/opt/soft/jdk-8 # export set JAVA_OPTS="--add-opens java.base/java.lang=ALL-UNNAMED" export ZOOKEEPER_HOME=/opt/soft/zookeeper-3 export HDFS_NAMENODE_USER=root export HDFS_SECONDARYNAMENODE_USER=root export HDFS_DATANODE_USER=root export HDFS_ZKFC_USER=root export HDFS_JOURNALNODE_USER=root export HADOOP_SHELL_EXECNAME=root export YARN_RESOURCEMANAGER_USER=root export YARN_NODEMANAGER_USER=root export HADOOP_HOME=/opt/soft/hadoop-3 export HADOOP_INSTALL=$HADOOP_HOME export HADOOP_MAPRED_HOME=$HADOOP_HOME export HADOOP_COMMON_HOME=$HADOOP_HOME export HADOOP_HDFS_HOME=$HADOOP_HOME export YARN_HOME=$HADOOP_HOME export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop export PATH=$PATH:$JAVA_HOME/bin:$ZOOKEEPER_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
加载
source /etc/profile
配置zookeeper配置并创建目录
三台都需要
# 编辑配置文件 cd $ZOOKEEPER_HOME/conf vim zoo.cfg
# 设置zookeeper # 心跳单位，2s tickTime=2000 # zookeeper-3初始化的同步超时时间，10个心跳单位，也即20s initLimit=10 # 普通同步：发送一个请求并得到响应的超时时间，5个心跳单位也即10s syncLimit=5 # 内存快照数据的存储位置 dataDir=/home/zookeeper-3/data # 事务日志的存储位置 dataLogDir=/home/zookeeper-3/datalog # 当前zookeeper-3节点的端口 clientPort=2181 # 单个客户端到集群中单个节点的并发连接数，通过ip判断是否同一个客户端，默认60 maxClientCnxns=1000 # 保留7个内存快照文件在dataDir中，默认保留3个 autopurge.snapRetainCount=7 # 清除快照的定时任务，默认1小时，如果设置为0，标识关闭清除任务 autopurge.purgeInterval=1 #允许客户端连接设置的最小超时时间，默认2个心跳单位 minSessionTimeout=4000 #允许客户端连接设置的最大超时时间，默认是20个心跳单位，也即40s， maxSessionTimeout=300000 #zookeeper-3 3.5.5启动默认会把AdminService服务启动，这个服务默认是8080端口 admin.serverPort=9001 #集群地址配置 server.1=hadoop0:2888:3888 server.2=hadoop1:2888:3888 server.3=hadoop2:2888:3888
创建目录是根据刚刚的配置
三台都需要
mkdir -p /home/zookeeper-3/data /home/zookeeper-3/datalog
然后依次分配节点id在三台服务器
hadoop0
echo 1 > /home/zookeeper-3/data/myid more /home/zookeeper-3/data/myid
hadoop1
echo 2 > /home/zookeeper-3/data/myid more /home/zookeeper-3/data/myid
hadoop2
echo 3 > /home/zookeeper-3/data/myid more /home/zookeeper-3/data/myid
编写启动脚本
三台都需要
cd /etc/systemd/system vim zookeeper.service
[Unit] Description=zookeeper After=syslog.target network.target [Service] Type=forking # 指定zookeeper-3 日志文件路径，也可以在zkServer.sh 中定义 Environment=ZOO_LOG_DIR=/home/zookeeper-3/datalog # 指定JDK路径，也可以在zkServer.sh 中定义 Environment=JAVA_HOME=/opt/soft/jdk-8 ExecStart=/opt/soft/zookeeper-3/bin/zkServer.sh start ExecStop=/opt/soft/zookeeper-3/bin/zkServer.sh stop Restart=always User=root Group=root [Install] WantedBy=multi-user.target
然后在三台服务器重新加载某个服务的配置文件
systemctl daemon-reload
systemctl start zookeeper systemctl enable zookeeper systemctl status zookeeper
三台服务器如下
如果失败
常见问题
- 是否有文件夹路径未创建
- 是否未配置conf路径下的zoo.cfg
2.搭建hadoop集群
因为之前单机已经下载了，这里只需要改配置文件即可
配置文件
三个都需要配置
- hadoop-env.sh
- core-site.xml
- hdfs-site.xml
- workers
- mapred-site.xml
- yarn-site.xml
core-site.xml
vim core-site.xml
<?xml version="1.0" encoding="UTF-8"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <!-- Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. See accompanying LICENSE file. --> <!-- Put site-specific property overrides in this file. --> <configuration> <property> <name>fs.defaultFS</name> <value>hdfs://hadoopService</value> </property> <property> <name>hadoop.tmp.dir</name> <value>/home/hadoop/data</value> </property> <property> <name>ha.zookeeper.quorum</name> <value>hadoop0:2181,hadoop1:2181,hadoop2:2181</value> </property> <property> <name>hadoop.http.staticuser.user</name> <value>root</value> </property> <property> <name>dfs.permissions.enabled</name> <value>false</value> </property> <property> <name>hadoop.proxyuser.root.hosts</name> <value>*</value> </property> <property> <name>hadoop.proxyuser.root.groups</name> <value>*</value> </property> </configuration>
同时创建文件夹
mkdir -p /home/hadoop/data
hdfs-site.xml
https://hadoop.apache.org/docs/r2.9.1/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml
vim hdfs-site.xml
<?xml version="1.0" encoding="UTF-8"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <!-- Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. See accompanying LICENSE file. --> <!-- Put site-specific property overrides in this file. --> <configuration> <property> <name>dfs.nameservices</name> <value>hadoopService</value> </property> <property> <name>dfs.ha.namenodes.hadoopService</name> <value>nn1,nn2</value> </property> <property> <name>dfs.namenode.rpc-address.hadoopService.nn1</name> <value>hadoop0:8020</value> </property> <property> <name>dfs.namenode.rpc-address.hadoopService.nn2</name> <value>hadoop1:8020</value> </property> <property> <name>dfs.namenode.http-address.hadoopService.nn1</name> <value>hadoop0:9870</value> </property> <property> <name>dfs.namenode.http-address.hadoopService.nn2</name> <value>hadoop1:9870</value> </property> <property> <name>dfs.namenode.shared.edits.dir</name> <value>qjournal://hadoop0:8485;hadoop1:8485;hadoop2:8485/hadoopService</value> </property> <property> <name>dfs.client.failover.proxy.provider.hadoopService</name> <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value> </property> <property> <name>dfs.ha.fencing.methods</name> <value>sshfence</value> </property> <property> <name>dfs.ha.fencing.ssh.private-key-files</name> <value>/root/.ssh/id_rsa</value> </property> <property> <name>dfs.journalnode.edits.dir</name> <value>/home/hadoop/journalnode/data</value> </property> <property> <name>dfs.ha.automatic-failover.enabled</name> <value>true</value> </property> <property> <name>dfs.safemode.threshold.pct</name> <value>1</value> </property> </configuration>
workers
vim workers
hadoop0 hadoop1 hadoop2
mapred-site.xml
vim mapred-site.xml
<?xml version="1.0"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <!-- Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. See accompanying LICENSE file. --> <!-- Put site-specific property overrides in this file. --> <configuration> <property> <name>mapreduce.framework.name</name> <value>yarn</value> </property> <property> <name>mapreduce.application.classpath</name> <value>$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*</value> </property> <!-- yarn历史服务端口 --> <property> <name>mapreduce.jobhistory.address</name> <value>hadoop0:10020</value> </property> <!-- yarn历史服务web访问端口 --> <property> <name>mapreduce.jobhistory.webapp.address</name> <value>hadoop0:19888</value> </property> </configuration>
yarn-site.xml
vim yarn-site.xml
<?xml version="1.0"?> <!-- Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. See accompanying LICENSE file. --> <configuration> <!-- Site specific YARN configuration properties --> <property> <name>yarn.resourcemanager.ha.enabled</name> <value>true</value> </property> <property> <name>yarn.resourcemanager.cluster-id</name> <value>cluster1</value> </property> <property> <name>yarn.resourcemanager.ha.rm-ids</name> <value>rm1,rm2</value> </property> <property> <name>yarn.resourcemanager.hostname.rm1</name> <value>hadoop0</value> </property> <property> <name>yarn.resourcemanager.hostname.rm2</name> <value>hadoop1</value> </property> <property> <name>yarn.resourcemanager.webapp.address.rm1</name> <value>hadoop0:8088</value> </property> <property> <name>yarn.resourcemanager.webapp.address.rm2</name> <value>hadoop1:8088</value> </property> <property> <name>yarn.resourcemanager.zk-address</name> <value>hadoop0:2181,hadoop1:2181,hadoop2:2181</value> </property> <property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> </property> <property> <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name> <value>org.apache.hadoop.mapred.ShuffleHandler</value> </property> <property> <name>yarn.nodemanager.env-whitelist</name> <value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME</value> </property> <!-- 是否将对容器实施物理内存限制 --> <property> <name>yarn.nodemanager.pmem-check-enabled</name> <value>false</value> </property> <!-- 是否将对容器实施虚拟内存限制。 --> <property> <name>yarn.nodemanager.vmem-check-enabled</name> <value>false</value> </property> <!-- 开启日志聚集 --> <property> <name>yarn.log-aggregation-enable</name> <value>true</value> </property> <!-- 设置yarn历史服务器地址 --> <property> <name>yarn.log.server.url</name> <value>http://hadoop0:19888/jobhistory/logs</value> </property> <!-- 保存的时间7天 --> <property> <name>yarn.log-aggregation.retain-seconds</name> <value>604800</value> </property> </configuration>
免密处理
在一号机器上操作
ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa ssh-copy-id root@hadoop0 ssh-copy-id root@hadoop1 ssh-copy-id root@hadoop2
测试连接
ssh root@hadoop1 # 退出 exit
利用scp命令将一号机的密钥拷贝到其他两台
scp -r ~/.ssh/ root@hadoop1:~/ scp -r ~/.ssh/ root@hadoop2:~/
在三台服务器上加载配置
source /etc/profile
启动hadoop
# 格式化 hdfs --daemon start journalnode
然后在（hadoop0、hadoop0）其中一个namenode进行格式化
注意
- namenode是hdfs-site.xml配置的n1和n2
- 如果你是单机拷贝过来的，要注意之前是否格式化文件系统了，要把(core-site.xml对应的数据持久化路径)所在路径下数据都删除
/tmp/hadoop/dfs/journalnode/hadoopService
hdfs namenode -format
然后启动
hdfs --daemon start namenode
然后把这个初始化数据通过命令拷贝到另一个节点
只需要在另一个namenode节点(hadoop1)执行
hdfs namenode -bootstrapStandby
然后也启动
hdfs --daemon start namenode
在其中一个namenode（hadoop0）上初始化
hdfs zkfc -formatZK
到这初始化完成了
在namenode（hadoop0、hadoop1）停止服务
stop-dfs.sh
然后在一个节点启动即可
start-all.sh
启动前
启动后
查看namenodes节点
[root@hadoop0 ~]# hdfs getconf -namenodes hadoop0 hadoop1
# 启动历史服务 mapred --daemon start historyserver
访问页面
http://hadoop0:9870/
http://hadoop0:8088
测试HDFS命令
# 创建文件夹 hdfs dfs -mkdir -p /wordcount/input # 查看 HDFS 目录结构 hdfs dfs -ls / hdfs dfs -ls /wordcount hdfs dfs -ls /wordcount/input # 上传本地测试文件 wcdata.txt 到 HDFS 上 /wordcount/input hdfs dfs -put wcdata.txt /wordcount/input # 检查文件是否上传成功 hdfs dfs -ls /wordcount/input hdfs dfs -cat /wordcount/input/wcdata.txt # 测试单词统计 hadoop jar $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.3.5.jar wordcount /wordcount/input/wcdata.txt /wordcount/result hdfs dfs -ls /wordcount/result hdfs dfs -cat /wordcount/result/part-r-00000
查看元数据
cd /home/hadoop_data/dfs/name/current ls
查看数据并将数据写入指定文件
# 查看数据,文件名根据你的生成fsimage情况 hdfs oiv -p XML -i fsimage_0000000000000000099
将元数据内容按照指定格式读取后写入到新文件中
hdfs oiv -p XML -i fsimage_0000000000000000099 -o /opt/soft/fsimage.xml hdfs oev -p XML -i edits_inprogress_0000000000000000673 -o /opt/soft/edit.xml
测试(利用hadoop自带的jar包算PI)
注意
- 保证hadoop1节点是active的
- 保证足够的磁盘内存，否则卡在running job不动
hadoop jar $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.3.5.jar pi 10 10
demo数据清理
https://gitee.com/htu-soft/bigdata2023/tree/master/Hadoop/code/hadoop
需要mysql就先上传到该路径下
$HADOOP_HOME/share/hadoop/common/lib/
然后拷贝到其他节点
scp $HADOOP_HOME/share/hadoop/common/lib/mysql-connector-j-8.0.33.jar root@hadoop1:$HADOOP_HOME/share/hadoop/common/lib/
Last login: Sun Nov 12 11:30:33 2023 [root@hadoop0 ~]# start-all.sh Starting namenodes on [hadoop0 hadoop1] 上一次登录：日 11月 12 12:10:18 CST 2023从 192.168.80.1pts/0 上 Starting datanodes 上一次登录：日 11月 12 12:10:32 CST 2023pts/0 上 Starting journal nodes [hadoop0 hadoop2 hadoop1] 上一次登录：日 11月 12 12:10:35 CST 2023pts/0 上 Starting ZK Failover Controllers on NN hosts [hadoop0 hadoop1] 上一次登录：日 11月 12 12:10:49 CST 2023pts/0 上 Starting resourcemanagers on [ hadoop0 hadoop1] 上一次登录：日 11月 12 12:10:55 CST 2023pts/0 上 Starting nodemanagers 上一次登录：日 11月 12 12:11:15 CST 2023pts/0 上 [root@hadoop0 ~]# hdfs dfs -mkdir -p /video/data [root@hadoop0 ~]# cd /opt/soft/ [root@hadoop0 soft]# ls data.csv docker-compose fsimage.xml hadoop.jar skywalking zookeeper-3 data.json edit.xml hadoop-3 jdk-8 wcdata.txt [root@hadoop0 soft]# hdfs dfs -put data.json /video/data [root@hadoop0 soft]# hdfs dfs -cat /video/data/data.json {"rank":1,"title":"《逃出大英博物馆》第二集","dzl":"77.8","bfl":"523.9","zfl":"39000","type":"影视","time":"3-5"} {"rank":2,"title":"兄弟们这一期真的要看到最后！","dzl":"89.6","bfl":"636.1","zfl":"4246","type":"搞笑","time":"3-5"} {"rank":3,"title":"全世界学生都痛恨的一种教育方式","dzl":"27.3","bfl":"313.6","zfl":"13000","type":"搞笑","time":"5-8"} {"rank":4,"title":"这是我玩过最抽象的宝可梦了3","dzl":"29.7","bfl":"193.6","zfl":"4631","type":"搞笑","time":"10+"} {"rank":5,"title":"我们又又又改造公司了","dzl":"25.0","bfl":"269.7","zfl":"3084","type":"搞笑","time":"10+"} {"rank":6,"title":"我在尝试当一种很新的美食博主","dzl":"48.0","bfl":"258.7","zfl":"6596","type":"美食","time":"0-3"} {"rank":100,"title":"很听劝我和男装的自己搞在一起","dzl":"18.7","bfl":"87.8","zfl":"4077","type":"搞笑","time":"0-3"} {"rank":99,"title":"探秘中国最贵的面！70只大闸蟹做一碗面！到底什么味道？","dzl":"8.8","bfl":"88.2","zfl":"4527","type":"美食","time":"5-8"} {"rank":7,"title":"土布","dzl":"26.4","bfl":"224.8","zfl":"3291","type":"生活","time":"5-8"} {"rank":34,"title":"我的工作周报vs我实际的工作","dzl":"18.2","bfl":"162.6","zfl":"21000","type":"生活","time":"3-5"} {"rank":8,"title":"麒麟9000S性能分析：华为Mate60 Pro能效如何？","dzl":"18.7","bfl":"151.7","zfl":"34000","type":"知识","time":"10+"} {"rank":98,"title":"Time Vent！十秒带你穿越2023-2009！感受令和与新平成假面骑士的力量叭！","dzl":"14.4","bfl":"124.3","zfl":"6918","type":"动画","time":"0-3"} {"rank":35,"title":"魔术师的三个阶段","dzl":"12.2","bfl":"180.0","zfl":"3841","type":"影视","time":"3-5"} {"rank":9,"title":"高斯一败涂地","dzl":"21.3","bfl":"181.3","zfl":"3424","type":"搞笑","time":"10+"} {"rank":97,"title":"千匹之战！中国车VS特斯拉！","dzl":"7.2","bfl":"117.4","zfl":"15000","type":"知识","time":"10+"} {"rank":96,"title":"真实事件改编没想到一个综艺节目这么敢拍孩子需要被改造吗？","dzl":"11.2","bfl":"441.4","zfl":"1640","type":"影视","time":"5-8"} {"rank":36,"title":"哈哈哈哈哈谁是卧底音乐版","dzl":"14.2","bfl":"232.9","zfl":"16000","type":"生活","time":"0-3"} {"rank":10,"title":"偷子","dzl":"58.1","bfl":"699.6","zfl":"20000","type":"搞笑","time":"10+"} {"rank":37,"title":"感谢还有读书这条路能让我摆脱这样的原生家庭","dzl":"11.2","bfl":"162.4","zfl":"13000","type":"生活","time":"10+"} {"rank":11,"title":"“当 代 热 门 游 戏 现 状”","dzl":"26.3","bfl":"205.0","zfl":"2511","type":"游戏","time":"0-3"} {"rank":95,"title":"欧洲资本积累到底有多肮脏？揭秘罪恶的黑奴贸易历史书为何只字不提？","dzl":"8.1","bfl":"77.8","zfl":"3752","type":"生活","time":"10+"} {"rank":38,"title":"永不停息的制裁华为能否王者归来？","dzl":"16.0","bfl":"128.1","zfl":"3785","type":"知识","time":"8-10"} {"rank":12,"title":"章鱼哥疯了","dzl":"10.2","bfl":"425.1","zfl":"1550","type":"游戏","time":"10+"} {"rank":13,"title":"当你以游戏的方式打开军训","dzl":"18.6","bfl":"146.8","zfl":"11000","type":"生活","time":"8-10"} {"rank":14,"title":"这下终于听懂秀才的小曲了","dzl":"17.7","bfl":"233.5","zfl":"9093","type":"搞笑","time":"0-3"} {"rank":15,"title":"【毕导】这个视频里说的都是真的但你却永远无法证明","dzl":"32.3","bfl":"436.4","zfl":"35000","type":"知识","time":"10+"} {"rank":16,"title":"【冰冰vlog.011】总要在烟雨青天去趟江南吧","dzl":"15.7","bfl":"150.3","zfl":"15000","type":"生活","time":"8-10"} {"rank":17,"title":"【深度科普】长期摆烂如何恢复体能？","dzl":"9.9","bfl":"135.9","zfl":"12000","type":"运动","time":"5-8"} {"rank":18,"title":"恋与提瓦特空桃篇：谁让你是我命定的另一半","dzl":"16.3","bfl":"122.1","zfl":"9901","type":"动画","time":"3-5"} {"rank":19,"title":"帮唐山一位身残志坚的大姐卖冰激凌","dzl":"19.6","bfl":"134.2","zfl":"2845","type":"生活","time":"3-5"} {"rank":20,"title":"350元一桶巨型泡面！新晋“天价网红泡面”值不值？？？","dzl":"13.5","bfl":"270.4","zfl":"1517","type":"美食","time":"10+"} {"rank":21,"title":"深夜抢救我爸被喷一脸血怕得病猛搓脸找药吃","dzl":"21.1","bfl":"239.2","zfl":"1103","type":"生活","time":"5-8"} {"rank":22,"title":"新铁锅怎么开锅更好用酒店大厨教你诀窍保证不锈不粘锅","dzl":"22.2","bfl":"425.3","zfl":"7099","type":"美食","time":"0-3"} {"rank":94,"title":"九小时畅享听完一整本书不是一剪梅！书荒丨小说","dzl":"2.0","bfl":"119.8","zfl":"88","type":"动画","time":"10+"} {"rank":23,"title":"潮汕大妈在北美说唱圈里的生意经","dzl":"21.9","bfl":"417.8","zfl":"1106","type":"生活","time":"3-5"} {"rank":93,"title":"再次出发","dzl":"13.7","bfl":"104.6","zfl":"3630","type":"知识","time":"0-3"} {"rank":24,"title":"万人雪书的长视频终于来啦！架不住你们天天催啊花了好多时间呜呜呜～","dzl":"25.6","bfl":"376.5","zfl":"4181","type":"生活","time":"8-10"} {"rank":92,"title":"2023年8月热歌榜TOP50今夕是何年？难以置信这是2023年？","dzl":"6.7","bfl":"92.1","zfl":"3226","type":"音乐","time":"10+"} {"rank":25,"title":"《明日方舟》SideStory「不义之财」活动宣传PV","dzl":"14.1","bfl":"172.7","zfl":"36000","type":"游戏","time":"5-8"} {"rank":26,"title":"当中二青年来买单！(猜不到结局版）","dzl":"22.3","bfl":"235.5","zfl":"18000","type":"影视","time":"3-5"} {"rank":91,"title":"日本排海为何中国网友互喷？日本政府是如何正当化排海的？","dzl":"6.4","bfl":"70.7","zfl":"7570","type":"知识","time":"10+"} {"rank":90,"title":"她似了又活了。她没似他似了所以她也似了。她又活了可他们似了所以她又似了。","dzl":"8.0","bfl":"46.5","zfl":"7960","type":"搞笑","time":"3-5"} {"rank":28,"title":"宿舍规则怪谈","dzl":"27.1","bfl":"205.1","zfl":"5178","type":"搞笑","time":"10+"} {"rank":89,"title":"光头强家里捡的","dzl":"0.4","bfl":"320.9","zfl":"14","type":"动画","time":"8-10"} {"rank":88,"title":"全网首拆！麒麟 5G 确认回归 | 华为 Mate 60 Pro","dzl":"56.2","bfl":"703.5","zfl":"90000","type":"知识","time":"10+"} {"rank":29,"title":"穷小子强行与富少爷交换人生本想荣华富贵哪知少爷家里更是离谱！","dzl":"17.7","bfl":"288.5","zfl":"2355","type":"动画","time":"3-5"} {"rank":87,"title":"我精神状态完全没有出问题的啦！！！","dzl":"16.4","bfl":"127.9","zfl":"4622","type":"动画","time":"0-3"} {"rank":30,"title":"一个疯子却获得了诺贝尔奖真实故事改编高分电影《美丽心灵》","dzl":"12.5","bfl":"329.1","zfl":"3109","type":"影视","time":"10+"} {"rank":86,"title":"最诡异的一局","dzl":"11","bfl":"101.4","zfl":"1702","type":"游戏","time":"3-5"} {"rank":85,"title":"拥有几十台能上路的经典老车是什么样的体验？","dzl":"9.8","bfl":"163.4","zfl":"1669","type":"知识","time":"3-5"} {"rank":31,"title":"这款游戏出现在21世纪还是太迷幻了","dzl":"12.5","bfl":"114.6","zfl":"1791","type":"游戏","time":"10+"} {"rank":84,"title":"史上最好的原神模组！！（boss篇）","dzl":"8.7","bfl":"104.8","zfl":"8998","type":"游戏","time":"5-8"} {"rank":32,"title":"关于下半身的生理知识没人告诉你这些！｜拉撒保姆级教程","dzl":"10.9","bfl":"100.1","zfl":"7921","type":"知识","time":"10+"} {"rank":83,"title":"努力有错吗？学习有罪吗？高考没有一个韩国人敢退出的战争","dzl":"8.2","bfl":"168.7","zfl":"9989","type":"知识","time":"10+"} {"rank":33,"title":"宝可梦日语OP大全【挪威小哥Pellek｜中字】","dzl":"11.7","bfl":"77.4","zfl":"3124","type":"音乐","time":"8-10"} {"rank":82,"title":"不同年代的上班族白领都是怎么办公的？最后真的别演我工作状态哈哈哈哈！","dzl":"15.6","bfl":"147.5","zfl":"1688","type":"生活","time":"0-3"} {"rank":81,"title":"我们分手了","dzl":"8.1","bfl":"241.4","zfl":"7008","type":"运动","time":"5-8"} {"rank":39,"title":"老鼠们每天用捕鼠夹健身只为发动鼠界第三次世界大战","dzl":"19.2","bfl":"627.6","zfl":"10000","type":"影视","time":"10+"} {"rank":40,"title":"大意了！这些不是全国统一的","dzl":"17.0","bfl":"219.7","zfl":"1325","type":"生活","time":"5-8"} {"rank":41,"title":"青春没有售价米哈游贴脸开大","dzl":"28.6","bfl":"147.1","zfl":"29000","type":"游戏","time":"0-3"} {"rank":42,"title":"早自习补作业事件","dzl":"29.0","bfl":"331.6","zfl":"5866","type":"影视","time":"0-3"} {"rank":43,"title":"凌晨12.00教100个仙人掌唱歌没想到邻居找上门来了","dzl":"10.7","bfl":"330.0","zfl":"11000","type":"生活","time":"3-5"} {"rank":44,"title":"这一次我们重新定义砖块","dzl":"24.7","bfl":"309.7","zfl":"5792","type":"生活","time":"0-3"} {"rank":45,"title":"抗塔两分钟还是满血我没开玩笑！","dzl":"12.9","bfl":"279.4","zfl":"2197","type":"游戏","time":"5-8"} {"rank":67,"title":"世界纪录保持者的拍摄间该怎么提升？","dzl":"9.3","bfl":"117.2","zfl":"2266","type":"知识","time":"10+"} {"rank":68,"title":"一口气看完海贼王真人版第1季！1-8集！真人版符合你的预期吗？","dzl":"4.0","bfl":"238.6","zfl":"11000","type":"影视","time":"10+"} {"rank":69,"title":"叶问之摆烂宗师","dzl":"9.2","bfl":"139.3","zfl":"10000","type":"搞笑","time":"5-8"} {"rank":70,"title":"甲方花钱助我出道","dzl":"10.5","bfl":"78.4","zfl":"4665","type":"生活","time":"3-5"} {"rank":71,"title":"秀 才 小 曲 最 骚 版 本","dzl":"12.9","bfl":"144.9","zfl":"3663","type":"搞笑","time":"0-3"} {"rank":72,"title":"还原一下著名的《潘博文事件》","dzl":"21.8","bfl":"499.1","zfl":"5908","type":"生活","time":"3-5"} {"rank":73,"title":"手持烟火以谋生 心怀诗意以谋爱","dzl":"15.3","bfl":"120.8","zfl":"5448","type":"动画","time":"0-3"} {"rank":74,"title":"大家散伙！唐僧闹分手悟空为何大开杀戒？","dzl":"6.8","bfl":"101.4","zfl":"2224","type":"影视","time":"10+"} {"rank":75,"title":"数学揭示万物密码 当然这还不够我们要去探访一下永恒。","dzl":"8.4","bfl":"132.2","zfl":"1487","type":"知识","time":"5-8"} {"rank":76,"title":"如果2077的CV来配音《赛博朋克：边缘行者》（第八话）","dzl":"7.5","bfl":"57.5","zfl":"4243","type":"动画","time":"10+"} {"rank":77,"title":"自己做的游戏终于发布了！赚了多少钱！？","dzl":"17.6","bfl":"155.7","zfl":"3834","type":"游戏","time":"0-3"} {"rank":78,"title":"《要有自己的颜色》","dzl":"21.2","bfl":"199.9","zfl":"1782","type":"生活","time":"0-3"} {"rank":79,"title":"买一块几十斤巨大安格斯上脑又被坑惨了涮麻辣火锅却爽翻了","dzl":"11.2","bfl":"136.6","zfl":"803","type":"美食","time":"8-10"} {"rank":80,"title":"鹦鹉螺：我不是活化石","dzl":"14.3","bfl":"199.4","zfl":"1950","type":"知识","time":"10+"} {"rank":27,"title":"【洛天依游学记原创曲】歌行四方 | AI歌手X非遗音乐","dzl":"10.4","bfl":"200.7","zfl":"5512","type":"音乐","time":"3-5"} {"rank":46,"title":"我居然穿越回了10年前的B站！！！","dzl":"7.7","bfl":"116.6","zfl":"3811","type":"生活","time":"8-10"} {"rank":47,"title":"陈Sir的工作日","dzl":"13.5","bfl":"110.8","zfl":"26000","type":"生活","time":"0-3"} {"rank":48,"title":"干嘛啊","dzl":"24.0","bfl":"266.1","zfl":"7128","type":"生活","time":"0-3"} {"rank":49,"title":"你看你又不敢对峙命运了吧！！！","dzl":"14.0","bfl":"97.8","zfl":"696","type":"游戏","time":"3-5"} {"rank":50,"title":"我花1万块重庆打车到上海却被全国网友说成老赖","dzl":"15.3","bfl":"140.3","zfl":"10000","type":"生活","time":"10+"} {"rank":51,"title":"摸 气 挑 战 （2）","dzl":"26.3","bfl":"247.7","zfl":"9562","type":"生活","time":"0-3"} {"rank":52,"title":"仙人揉腹操九式详解版！做完大拉特拉","dzl":"7.9","bfl":"69.7","zfl":"14000","type":"运动","time":"0-3"} {"rank":53,"title":"看着徒弟一点点长大逝去才发现长生是苦【我和徒弟03】","dzl":"5.6","bfl":"150.4","zfl":"618","type":"动画","time":"10+"} {"rank":54,"title":"祝妹妹一路顺风","dzl":"25.4","bfl":"170.8","zfl":"18000","type":"生活","time":"0-3"} {"rank":55,"title":"300w粉丝特别节目！拍了一些大家想看的但是也太怪了？","dzl":"12.3","bfl":"66.5","zfl":"710","type":"知识","time":"5-8"} {"rank":56,"title":"爆肝几个星期我建出了最细节的海岛小镇！！！","dzl":"24.3","bfl":"214.8","zfl":"5545","type":"游戏","time":"5-8"} {"rank":57,"title":"让巴黎看到国人的美","dzl":"24.4","bfl":"186.1","zfl":"784","type":"生活","time":"0-3"} {"rank":58,"title":"村里来了新成员漠叔好心劝上学做社会有用的人","dzl":"11.9","bfl":"119.5","zfl":"1510","type":"美食","time":"10+"} {"rank":59,"title":"《原神》角色演示-「菲米尼：海露幽响」","dzl":"11.7","bfl":"110.7","zfl":"7387","type":"游戏","time":"5-8"} {"rank":61,"title":"当你碰到经验就会「瞬间暴毙」!!？","dzl":"10.0","bfl":"105.6","zfl":"554","type":"游戏","time":"10+"} {"rank":62,"title":"大学开学时的各种人|大学学习","dzl":"9.0","bfl":"294.9","zfl":"224","type":"搞笑","time":"3-5"} {"rank":63,"title":"都什么年代谁还用传统方式结义？！！","dzl":"9.0","bfl":"60.6","zfl":"3278","type":"搞笑","time":"10+"} {"rank":64,"title":"【闽南婚宴】中式流水席天花板吃过一辈子忘不掉。","dzl":"39.1","bfl":"393.2","zfl":"6.5","type":"美食","time":"10+"} {"rank":65,"title":"口腔溃疡为什么是白色的？","dzl":"18.1","bfl":"318.9","zfl":"3562","type":"知识","time":"3-5"} {"rank":66,"title":"我今年拼过最牛的积木！（上）","dzl":"7.8","bfl":"172.8","zfl":"8298","type":"动画","time":"10+"} {"rank":60,"title":"【太君の噩梦】打服日本一战成名比你想象的更夸张！苏联军神朱可夫","dzl":"10.7","bfl":"130.1","zfl":"1218","type":"知识","time":"10+"} [root@hadoop0 soft]# hdfs dfs -cat /video/data/data.json {"rank":1,"title":"《逃出大英博物馆》第二集","dzl":"77.8","bfl":"523.9","zfl":"39000","type":"影视","time":"3-5"} {"rank":2,"title":"兄弟们这一期真的要看到最后！","dzl":"89.6","bfl":"636.1","zfl":"4246","type":"搞笑","time":"3-5"} {"rank":3,"title":"全世界学生都痛恨的一种教育方式","dzl":"27.3","bfl":"313.6","zfl":"13000","type":"搞笑","time":"5-8"} {"rank":4,"title":"这是我玩过最抽象的宝可梦了3","dzl":"29.7","bfl":"193.6","zfl":"4631","type":"搞笑","time":"10+"} {"rank":5,"title":"我们又又又改造公司了","dzl":"25.0","bfl":"269.7","zfl":"3084","type":"搞笑","time":"10+"} {"rank":6,"title":"我在尝试当一种很新的美食博主","dzl":"48.0","bfl":"258.7","zfl":"6596","type":"美食","time":"0-3"} {"rank":100,"title":"很听劝我和男装的自己搞在一起","dzl":"18.7","bfl":"87.8","zfl":"4077","type":"搞笑","time":"0-3"} {"rank":99,"title":"探秘中国最贵的面！70只大闸蟹做一碗面！到底什么味道？","dzl":"8.8","bfl":"88.2","zfl":"4527","type":"美食","time":"5-8"} {"rank":7,"title":"土布","dzl":"26.4","bfl":"224.8","zfl":"3291","type":"生活","time":"5-8"} {"rank":34,"title":"我的工作周报vs我实际的工作","dzl":"18.2","bfl":"162.6","zfl":"21000","type":"生活","time":"3-5"} {"rank":8,"title":"麒麟9000S性能分析：华为Mate60 Pro能效如何？","dzl":"18.7","bfl":"151.7","zfl":"34000","type":"知识","time":"10+"} {"rank":98,"title":"Time Vent！十秒带你穿越2023-2009！感受令和与新平成假面骑士的力量叭！","dzl":"14.4","bfl":"124.3","zfl":"6918","type":"动画","time":"0-3"} {"rank":35,"title":"魔术师的三个阶段","dzl":"12.2","bfl":"180.0","zfl":"3841","type":"影视","time":"3-5"} {"rank":9,"title":"高斯一败涂地","dzl":"21.3","bfl":"181.3","zfl":"3424","type":"搞笑","time":"10+"} {"rank":97,"title":"千匹之战！中国车VS特斯拉！","dzl":"7.2","bfl":"117.4","zfl":"15000","type":"知识","time":"10+"} {"rank":96,"title":"真实事件改编没想到一个综艺节目这么敢拍孩子需要被改造吗？","dzl":"11.2","bfl":"441.4","zfl":"1640","type":"影视","time":"5-8"} {"rank":36,"title":"哈哈哈哈哈谁是卧底音乐版","dzl":"14.2","bfl":"232.9","zfl":"16000","type":"生活","time":"0-3"} {"rank":10,"title":"偷子","dzl":"58.1","bfl":"699.6","zfl":"20000","type":"搞笑","time":"10+"} {"rank":37,"title":"感谢还有读书这条路能让我摆脱这样的原生家庭","dzl":"11.2","bfl":"162.4","zfl":"13000","type":"生活","time":"10+"} {"rank":11,"title":"“当 代 热 门 游 戏 现 状”","dzl":"26.3","bfl":"205.0","zfl":"2511","type":"游戏","time":"0-3"} {"rank":95,"title":"欧洲资本积累到底有多肮脏？揭秘罪恶的黑奴贸易历史书为何只字不提？","dzl":"8.1","bfl":"77.8","zfl":"3752","type":"生活","time":"10+"} {"rank":38,"title":"永不停息的制裁华为能否王者归来？","dzl":"16.0","bfl":"128.1","zfl":"3785","type":"知识","time":"8-10"} {"rank":12,"title":"章鱼哥疯了","dzl":"10.2","bfl":"425.1","zfl":"1550","type":"游戏","time":"10+"} {"rank":13,"title":"当你以游戏的方式打开军训","dzl":"18.6","bfl":"146.8","zfl":"11000","type":"生活","time":"8-10"} {"rank":14,"title":"这下终于听懂秀才的小曲了","dzl":"17.7","bfl":"233.5","zfl":"9093","type":"搞笑","time":"0-3"} {"rank":15,"title":"【毕导】这个视频里说的都是真的但你却永远无法证明","dzl":"32.3","bfl":"436.4","zfl":"35000","type":"知识","time":"10+"} {"rank":16,"title":"【冰冰vlog.011】总要在烟雨青天去趟江南吧","dzl":"15.7","bfl":"150.3","zfl":"15000","type":"生活","time":"8-10"} {"rank":17,"title":"【深度科普】长期摆烂如何恢复体能？","dzl":"9.9","bfl":"135.9","zfl":"12000","type":"运动","time":"5-8"} {"rank":18,"title":"恋与提瓦特空桃篇：谁让你是我命定的另一半","dzl":"16.3","bfl":"122.1","zfl":"9901","type":"动画","time":"3-5"} {"rank":19,"title":"帮唐山一位身残志坚的大姐卖冰激凌","dzl":"19.6","bfl":"134.2","zfl":"2845","type":"生活","time":"3-5"} {"rank":20,"title":"350元一桶巨型泡面！新晋“天价网红泡面”值不值？？？","dzl":"13.5","bfl":"270.4","zfl":"1517","type":"美食","time":"10+"} {"rank":21,"title":"深夜抢救我爸被喷一脸血怕得病猛搓脸找药吃","dzl":"21.1","bfl":"239.2","zfl":"1103","type":"生活","time":"5-8"} {"rank":22,"title":"新铁锅怎么开锅更好用酒店大厨教你诀窍保证不锈不粘锅","dzl":"22.2","bfl":"425.3","zfl":"7099","type":"美食","time":"0-3"} {"rank":94,"title":"九小时畅享听完一整本书不是一剪梅！书荒丨小说","dzl":"2.0","bfl":"119.8","zfl":"88","type":"动画","time":"10+"} {"rank":23,"title":"潮汕大妈在北美说唱圈里的生意经","dzl":"21.9","bfl":"417.8","zfl":"1106","type":"生活","time":"3-5"} {"rank":93,"title":"再次出发","dzl":"13.7","bfl":"104.6","zfl":"3630","type":"知识","time":"0-3"} {"rank":24,"title":"万人雪书的长视频终于来啦！架不住你们天天催啊花了好多时间呜呜呜～","dzl":"25.6","bfl":"376.5","zfl":"4181","type":"生活","time":"8-10"} {"rank":92,"title":"2023年8月热歌榜TOP50今夕是何年？难以置信这是2023年？","dzl":"6.7","bfl":"92.1","zfl":"3226","type":"音乐","time":"10+"} {"rank":25,"title":"《明日方舟》SideStory「不义之财」活动宣传PV","dzl":"14.1","bfl":"172.7","zfl":"36000","type":"游戏","time":"5-8"} {"rank":26,"title":"当中二青年来买单！(猜不到结局版）","dzl":"22.3","bfl":"235.5","zfl":"18000","type":"影视","time":"3-5"} {"rank":91,"title":"日本排海为何中国网友互喷？日本政府是如何正当化排海的？","dzl":"6.4","bfl":"70.7","zfl":"7570","type":"知识","time":"10+"} {"rank":90,"title":"她似了又活了。她没似他似了所以她也似了。她又活了可他们似了所以她又似了。","dzl":"8.0","bfl":"46.5","zfl":"7960","type":"搞笑","time":"3-5"} {"rank":28,"title":"宿舍规则怪谈","dzl":"27.1","bfl":"205.1","zfl":"5178","type":"搞笑","time":"10+"} {"rank":89,"title":"光头强家里捡的","dzl":"0.4","bfl":"320.9","zfl":"14","type":"动画","time":"8-10"} {"rank":88,"title":"全网首拆！麒麟 5G 确认回归 | 华为 Mate 60 Pro","dzl":"56.2","bfl":"703.5","zfl":"90000","type":"知识","time":"10+"} {"rank":29,"title":"穷小子强行与富少爷交换人生本想荣华富贵哪知少爷家里更是离谱！","dzl":"17.7","bfl":"288.5","zfl":"2355","type":"动画","time":"3-5"} {"rank":87,"title":"我精神状态完全没有出问题的啦！！！","dzl":"16.4","bfl":"127.9","zfl":"4622","type":"动画","time":"0-3"} {"rank":30,"title":"一个疯子却获得了诺贝尔奖真实故事改编高分电影《美丽心灵》","dzl":"12.5","bfl":"329.1","zfl":"3109","type":"影视","time":"10+"} {"rank":86,"title":"最诡异的一局","dzl":"11","bfl":"101.4","zfl":"1702","type":"游戏","time":"3-5"} {"rank":85,"title":"拥有几十台能上路的经典老车是什么样的体验？","dzl":"9.8","bfl":"163.4","zfl":"1669","type":"知识","time":"3-5"} {"rank":31,"title":"这款游戏出现在21世纪还是太迷幻了","dzl":"12.5","bfl":"114.6","zfl":"1791","type":"游戏","time":"10+"} {"rank":84,"title":"史上最好的原神模组！！（boss篇）","dzl":"8.7","bfl":"104.8","zfl":"8998","type":"游戏","time":"5-8"} {"rank":32,"title":"关于下半身的生理知识没人告诉你这些！｜拉撒保姆级教程","dzl":"10.9","bfl":"100.1","zfl":"7921","type":"知识","time":"10+"} {"rank":83,"title":"努力有错吗？学习有罪吗？高考没有一个韩国人敢退出的战争","dzl":"8.2","bfl":"168.7","zfl":"9989","type":"知识","time":"10+"} {"rank":33,"title":"宝可梦日语OP大全【挪威小哥Pellek｜中字】","dzl":"11.7","bfl":"77.4","zfl":"3124","type":"音乐","time":"8-10"} {"rank":82,"title":"不同年代的上班族白领都是怎么办公的？最后真的别演我工作状态哈哈哈哈！","dzl":"15.6","bfl":"147.5","zfl":"1688","type":"生活","time":"0-3"} {"rank":81,"title":"我们分手了","dzl":"8.1","bfl":"241.4","zfl":"7008","type":"运动","time":"5-8"} {"rank":39,"title":"老鼠们每天用捕鼠夹健身只为发动鼠界第三次世界大战","dzl":"19.2","bfl":"627.6","zfl":"10000","type":"影视","time":"10+"} {"rank":40,"title":"大意了！这些不是全国统一的","dzl":"17.0","bfl":"219.7","zfl":"1325","type":"生活","time":"5-8"} {"rank":41,"title":"青春没有售价米哈游贴脸开大","dzl":"28.6","bfl":"147.1","zfl":"29000","type":"游戏","time":"0-3"} {"rank":42,"title":"早自习补作业事件","dzl":"29.0","bfl":"331.6","zfl":"5866","type":"影视","time":"0-3"} {"rank":43,"title":"凌晨12.00教100个仙人掌唱歌没想到邻居找上门来了","dzl":"10.7","bfl":"330.0","zfl":"11000","type":"生活","time":"3-5"} {"rank":44,"title":"这一次我们重新定义砖块","dzl":"24.7","bfl":"309.7","zfl":"5792","type":"生活","time":"0-3"} {"rank":45,"title":"抗塔两分钟还是满血我没开玩笑！","dzl":"12.9","bfl":"279.4","zfl":"2197","type":"游戏","time":"5-8"} {"rank":67,"title":"世界纪录保持者的拍摄间该怎么提升？","dzl":"9.3","bfl":"117.2","zfl":"2266","type":"知识","time":"10+"} {"rank":68,"title":"一口气看完海贼王真人版第1季！1-8集！真人版符合你的预期吗？","dzl":"4.0","bfl":"238.6","zfl":"11000","type":"影视","time":"10+"} {"rank":69,"title":"叶问之摆烂宗师","dzl":"9.2","bfl":"139.3","zfl":"10000","type":"搞笑","time":"5-8"} {"rank":70,"title":"甲方花钱助我出道","dzl":"10.5","bfl":"78.4","zfl":"4665","type":"生活","time":"3-5"} {"rank":71,"title":"秀 才 小 曲 最 骚 版 本","dzl":"12.9","bfl":"144.9","zfl":"3663","type":"搞笑","time":"0-3"} {"rank":72,"title":"还原一下著名的《潘博文事件》","dzl":"21.8","bfl":"499.1","zfl":"5908","type":"生活","time":"3-5"} {"rank":73,"title":"手持烟火以谋生 心怀诗意以谋爱","dzl":"15.3","bfl":"120.8","zfl":"5448","type":"动画","time":"0-3"} {"rank":74,"title":"大家散伙！唐僧闹分手悟空为何大开杀戒？","dzl":"6.8","bfl":"101.4","zfl":"2224","type":"影视","time":"10+"} {"rank":75,"title":"数学揭示万物密码 当然这还不够我们要去探访一下永恒。","dzl":"8.4","bfl":"132.2","zfl":"1487","type":"知识","time":"5-8"} {"rank":76,"title":"如果2077的CV来配音《赛博朋克：边缘行者》（第八话）","dzl":"7.5","bfl":"57.5","zfl":"4243","type":"动画","time":"10+"} {"rank":77,"title":"自己做的游戏终于发布了！赚了多少钱！？","dzl":"17.6","bfl":"155.7","zfl":"3834","type":"游戏","time":"0-3"} {"rank":78,"title":"《要有自己的颜色》","dzl":"21.2","bfl":"199.9","zfl":"1782","type":"生活","time":"0-3"} {"rank":79,"title":"买一块几十斤巨大安格斯上脑又被坑惨了涮麻辣火锅却爽翻了","dzl":"11.2","bfl":"136.6","zfl":"803","type":"美食","time":"8-10"} {"rank":80,"title":"鹦鹉螺：我不是活化石","dzl":"14.3","bfl":"199.4","zfl":"1950","type":"知识","time":"10+"} {"rank":27,"title":"【洛天依游学记原创曲】歌行四方 | AI歌手X非遗音乐","dzl":"10.4","bfl":"200.7","zfl":"5512","type":"音乐","time":"3-5"} {"rank":46,"title":"我居然穿越回了10年前的B站！！！","dzl":"7.7","bfl":"116.6","zfl":"3811","type":"生活","time":"8-10"} {"rank":47,"title":"陈Sir的工作日","dzl":"13.5","bfl":"110.8","zfl":"26000","type":"生活","time":"0-3"} {"rank":48,"title":"干嘛啊","dzl":"24.0","bfl":"266.1","zfl":"7128","type":"生活","time":"0-3"} {"rank":49,"title":"你看你又不敢对峙命运了吧！！！","dzl":"14.0","bfl":"97.8","zfl":"696","type":"游戏","time":"3-5"} {"rank":50,"title":"我花1万块重庆打车到上海却被全国网友说成老赖","dzl":"15.3","bfl":"140.3","zfl":"10000","type":"生活","time":"10+"} {"rank":51,"title":"摸 气 挑 战 （2）","dzl":"26.3","bfl":"247.7","zfl":"9562","type":"生活","time":"0-3"} {"rank":52,"title":"仙人揉腹操九式详解版！做完大拉特拉","dzl":"7.9","bfl":"69.7","zfl":"14000","type":"运动","time":"0-3"} {"rank":53,"title":"看着徒弟一点点长大逝去才发现长生是苦【我和徒弟03】","dzl":"5.6","bfl":"150.4","zfl":"618","type":"动画","time":"10+"} {"rank":54,"title":"祝妹妹一路顺风","dzl":"25.4","bfl":"170.8","zfl":"18000","type":"生活","time":"0-3"} {"rank":55,"title":"300w粉丝特别节目！拍了一些大家想看的但是也太怪了？","dzl":"12.3","bfl":"66.5","zfl":"710","type":"知识","time":"5-8"} {"rank":56,"title":"爆肝几个星期我建出了最细节的海岛小镇！！！","dzl":"24.3","bfl":"214.8","zfl":"5545","type":"游戏","time":"5-8"} {"rank":57,"title":"让巴黎看到国人的美","dzl":"24.4","bfl":"186.1","zfl":"784","type":"生活","time":"0-3"} {"rank":58,"title":"村里来了新成员漠叔好心劝上学做社会有用的人","dzl":"11.9","bfl":"119.5","zfl":"1510","type":"美食","time":"10+"} {"rank":59,"title":"《原神》角色演示-「菲米尼：海露幽响」","dzl":"11.7","bfl":"110.7","zfl":"7387","type":"游戏","time":"5-8"} {"rank":61,"title":"当你碰到经验就会「瞬间暴毙」!!？","dzl":"10.0","bfl":"105.6","zfl":"554","type":"游戏","time":"10+"} {"rank":62,"title":"大学开学时的各种人|大学学习","dzl":"9.0","bfl":"294.9","zfl":"224","type":"搞笑","time":"3-5"} {"rank":63,"title":"都什么年代谁还用传统方式结义？！！","dzl":"9.0","bfl":"60.6","zfl":"3278","type":"搞笑","time":"10+"} {"rank":64,"title":"【闽南婚宴】中式流水席天花板吃过一辈子忘不掉。","dzl":"39.1","bfl":"393.2","zfl":"6.5","type":"美食","time":"10+"} {"rank":65,"title":"口腔溃疡为什么是白色的？","dzl":"18.1","bfl":"318.9","zfl":"3562","type":"知识","time":"3-5"} {"rank":66,"title":"我今年拼过最牛的积木！（上）","dzl":"7.8","bfl":"172.8","zfl":"8298","type":"动画","time":"10+"} {"rank":60,"title":"【太君の噩梦】打服日本一战成名比你想象的更夸张！苏联军神朱可夫","dzl":"10.7","bfl":"130.1","zfl":"1218","type":"知识","time":"10+"} ^C[root@hadoop0 soft]# hadoop jar hadoop.jar com.lihaozhe.video.DataFormat 2023-11-12 12:14:55,809 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm2 2023-11-12 12:14:56,382 WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this. 2023-11-12 12:14:56,470 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/root/.staging/job_1699762297160_0001 2023-11-12 12:14:57,975 INFO input.FileInputFormat: Total input files to process : 1 2023-11-12 12:14:58,263 INFO mapreduce.JobSubmitter: number of splits:1 2023-11-12 12:14:58,596 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1699762297160_0001 2023-11-12 12:14:58,598 INFO mapreduce.JobSubmitter: Executing with tokens: [] 2023-11-12 12:14:58,912 INFO conf.Configuration: resource-types.xml not found 2023-11-12 12:14:58,912 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'. 2023-11-12 12:14:59,770 INFO impl.YarnClientImpl: Submitted application application_1699762297160_0001 2023-11-12 12:14:59,945 INFO mapreduce.Job: The url to track the job: http://hadoop1:8088/proxy/application_1699762297160_0001/ 2023-11-12 12:14:59,946 INFO mapreduce.Job: Running job: job_1699762297160_0001 2023-11-12 12:15:21,595 INFO mapreduce.Job: Job job_1699762297160_0001 running in uber mode : false 2023-11-12 12:15:21,606 INFO mapreduce.Job: map 0% reduce 0% 2023-11-12 12:15:35,993 INFO mapreduce.Job: map 100% reduce 0% 2023-11-12 12:15:37,022 INFO mapreduce.Job: Job job_1699762297160_0001 completed successfully 2023-11-12 12:15:37,255 INFO mapreduce.Job: Counters: 33 File System Counters FILE: Number of bytes read=0 FILE: Number of bytes written=278833 FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations=0 HDFS: Number of bytes read=14367 HDFS: Number of bytes written=8161 HDFS: Number of read operations=7 HDFS: Number of large read operations=0 HDFS: Number of write operations=2 HDFS: Number of bytes read erasure-coded=0 Job Counters Launched map tasks=1 Data-local map tasks=1 Total time spent by all maps in occupied slots (ms)=11602 Total time spent by all reduces in occupied slots (ms)=0 Total time spent by all map tasks (ms)=11602 Total vcore-milliseconds taken by all map tasks=11602 Total megabyte-milliseconds taken by all map tasks=11880448 Map-Reduce Framework Map input records=100 Map output records=100 Input split bytes=106 Spilled Records=0 Failed Shuffles=0 Merged Map outputs=0 GC time elapsed (ms)=85 CPU time spent (ms)=1810 Physical memory (bytes) snapshot=190275584 Virtual memory (bytes) snapshot=2585636864 Total committed heap usage (bytes)=98041856 Peak Map Physical memory (bytes)=190275584 Peak Map Virtual memory (bytes)=2585636864 File Input Format Counters Bytes Read=14261 File Output Format Counters Bytes Written=8161 [root@hadoop0 soft]# hdfs dfs -cat /video/ods/part-m-00000 1,《逃出大英博物馆》第二集,77.8,523.9,39000,影视,3-5 2,兄弟们这一期真的要看到最后！,89.6,636.1,4246,搞笑,3-5 3,全世界学生都痛恨的一种教育方式,27.3,313.6,13000,搞笑,5-8 4,这是我玩过最抽象的宝可梦了3,29.7,193.6,4631,搞笑,10+ 5,我们又又又改造公司了,25.0,269.7,3084,搞笑,10+ 6,我在尝试当一种很新的美食博主,48.0,258.7,6596,美食,0-3 100,很听劝我和男装的自己搞在一起,18.7,87.8,4077,搞笑,0-3 99,探秘中国最贵的面！70只大闸蟹做一碗面！到底什么味道？,8.8,88.2,4527,美食,5-8 7,土布,26.4,224.8,3291,生活,5-8 34,我的工作周报vs我实际的工作,18.2,162.6,21000,生活,3-5 8,麒麟9000S性能分析：华为Mate60 Pro能效如何？,18.7,151.7,34000,知识,10+ 98,Time Vent！十秒带你穿越2023-2009！感受令和与新平成假面骑士的力量叭！,14.4,124.3,6918,动画,0-3 35,魔术师的三个阶段,12.2,180.0,3841,影视,3-5 9,高斯一败涂地,21.3,181.3,3424,搞笑,10+ 97,千匹之战！中国车VS特斯拉！,7.2,117.4,15000,知识,10+ 96,真实事件改编没想到一个综艺节目这么敢拍孩子需要被改造吗？,11.2,441.4,1640,影视,5-8 36,哈哈哈哈哈谁是卧底音乐版,14.2,232.9,16000,生活,0-3 10,偷子,58.1,699.6,20000,搞笑,10+ 37,感谢还有读书这条路能让我摆脱这样的原生家庭,11.2,162.4,13000,生活,10+ 11,“当 代 热 门 游 戏 现 状”,26.3,205.0,2511,游戏,0-3 95,欧洲资本积累到底有多肮脏？揭秘罪恶的黑奴贸易历史书为何只字不提？,8.1,77.8,3752,生活,10+ 38,永不停息的制裁华为能否王者归来？,16.0,128.1,3785,知识,8-10 12,章鱼哥疯了,10.2,425.1,1550,游戏,10+ 13,当你以游戏的方式打开军训,18.6,146.8,11000,生活,8-10 14,这下终于听懂秀才的小曲了,17.7,233.5,9093,搞笑,0-3 15,【毕导】这个视频里说的都是真的但你却永远无法证明,32.3,436.4,35000,知识,10+ 16,【冰冰vlog.011】总要在烟雨青天去趟江南吧,15.7,150.3,15000,生活,8-10 17,【深度科普】长期摆烂如何恢复体能？,9.9,135.9,12000,运动,5-8 18,恋与提瓦特空桃篇：谁让你是我命定的另一半,16.3,122.1,9901,动画,3-5 19,帮唐山一位身残志坚的大姐卖冰激凌,19.6,134.2,2845,生活,3-5 20,350元一桶巨型泡面！新晋“天价网红泡面”值不值？？？,13.5,270.4,1517,美食,10+ 21,深夜抢救我爸被喷一脸血怕得病猛搓脸找药吃,21.1,239.2,1103,生活,5-8 22,新铁锅怎么开锅更好用酒店大厨教你诀窍保证不锈不粘锅,22.2,425.3,7099,美食,0-3 94,九小时畅享听完一整本书不是一剪梅！书荒丨小说,2.0,119.8,88,动画,10+ 23,潮汕大妈在北美说唱圈里的生意经,21.9,417.8,1106,生活,3-5 93,再次出发,13.7,104.6,3630,知识,0-3 24,万人雪书的长视频终于来啦！架不住你们天天催啊花了好多时间呜呜呜～,25.6,376.5,4181,生活,8-10 92,2023年8月热歌榜TOP50今夕是何年？难以置信这是2023年？,6.7,92.1,3226,音乐,10+ 25,《明日方舟》SideStory「不义之财」活动宣传PV,14.1,172.7,36000,游戏,5-8 26,当中二青年来买单！(猜不到结局版）,22.3,235.5,18000,影视,3-5 91,日本排海为何中国网友互喷？日本政府是如何正当化排海的？,6.4,70.7,7570,知识,10+ 90,她似了又活了。她没似他似了所以她也似了。她又活了可他们似了所以她又似了。,8.0,46.5,7960,搞笑,3-5 28,宿舍规则怪谈,27.1,205.1,5178,搞笑,10+ 89,光头强家里捡的,0.4,320.9,14,动画,8-10 88,全网首拆！麒麟 5G 确认回归 | 华为 Mate 60 Pro,56.2,703.5,90000,知识,10+ 29,穷小子强行与富少爷交换人生本想荣华富贵哪知少爷家里更是离谱！,17.7,288.5,2355,动画,3-5 87,我精神状态完全没有出问题的啦！！！,16.4,127.9,4622,动画,0-3 30,一个疯子却获得了诺贝尔奖真实故事改编高分电影《美丽心灵》,12.5,329.1,3109,影视,10+ 86,最诡异的一局,11,101.4,1702,游戏,3-5 85,拥有几十台能上路的经典老车是什么样的体验？,9.8,163.4,1669,知识,3-5 31,这款游戏出现在21世纪还是太迷幻了,12.5,114.6,1791,游戏,10+ 84,史上最好的原神模组！！（boss篇）,8.7,104.8,8998,游戏,5-8 32,关于下半身的生理知识没人告诉你这些！｜拉撒保姆级教程,10.9,100.1,7921,知识,10+ 83,努力有错吗？学习有罪吗？高考没有一个韩国人敢退出的战争,8.2,168.7,9989,知识,10+ 33,宝可梦日语OP大全【挪威小哥Pellek｜中字】,11.7,77.4,3124,音乐,8-10 82,不同年代的上班族白领都是怎么办公的？最后真的别演我工作状态哈哈哈哈！,15.6,147.5,1688,生活,0-3 81,我们分手了,8.1,241.4,7008,运动,5-8 39,老鼠们每天用捕鼠夹健身只为发动鼠界第三次世界大战,19.2,627.6,10000,影视,10+ 40,大意了！这些不是全国统一的,17.0,219.7,1325,生活,5-8 41,青春没有售价米哈游贴脸开大,28.6,147.1,29000,游戏,0-3 42,早自习补作业事件,29.0,331.6,5866,影视,0-3 43,凌晨12.00教100个仙人掌唱歌没想到邻居找上门来了,10.7,330.0,11000,生活,3-5 44,这一次我们重新定义砖块,24.7,309.7,5792,生活,0-3 45,抗塔两分钟还是满血我没开玩笑！,12.9,279.4,2197,游戏,5-8 67,世界纪录保持者的拍摄间该怎么提升？,9.3,117.2,2266,知识,10+ 68,一口气看完海贼王真人版第1季！1-8集！真人版符合你的预期吗？,4.0,238.6,11000,影视,10+ 69,叶问之摆烂宗师,9.2,139.3,10000,搞笑,5-8 70,甲方花钱助我出道,10.5,78.4,4665,生活,3-5 71,秀 才 小 曲 最 骚 版 本,12.9,144.9,3663,搞笑,0-3 72,还原一下著名的《潘博文事件》,21.8,499.1,5908,生活,3-5 73,手持烟火以谋生 心怀诗意以谋爱,15.3,120.8,5448,动画,0-3 74,大家散伙！唐僧闹分手悟空为何大开杀戒？,6.8,101.4,2224,影视,10+ 75,数学揭示万物密码 当然这还不够我们要去探访一下永恒。,8.4,132.2,1487,知识,5-8 76,如果2077的CV来配音《赛博朋克：边缘行者》（第八话）,7.5,57.5,4243,动画,10+ 77,自己做的游戏终于发布了！赚了多少钱！？,17.6,155.7,3834,游戏,0-3 78,《要有自己的颜色》,21.2,199.9,1782,生活,0-3 79,买一块几十斤巨大安格斯上脑又被坑惨了涮麻辣火锅却爽翻了,11.2,136.6,803,美食,8-10 80,鹦鹉螺：我不是活化石,14.3,199.4,1950,知识,10+ 27,【洛天依游学记原创曲】歌行四方 | AI歌手X非遗音乐,10.4,200.7,5512,音乐,3-5 46,我居然穿越回了10年前的B站！！！,7.7,116.6,3811,生活,8-10 47,陈Sir的工作日,13.5,110.8,26000,生活,0-3 48,干嘛啊,24.0,266.1,7128,生活,0-3 49,你看你又不敢对峙命运了吧！！！,14.0,97.8,696,游戏,3-5 50,我花1万块重庆打车到上海却被全国网友说成老赖,15.3,140.3,10000,生活,10+ 51,摸 气 挑 战 （2）,26.3,247.7,9562,生活,0-3 52,仙人揉腹操九式详解版！做完大拉特拉,7.9,69.7,14000,运动,0-3 53,看着徒弟一点点长大逝去才发现长生是苦【我和徒弟03】,5.6,150.4,618,动画,10+ 54,祝妹妹一路顺风,25.4,170.8,18000,生活,0-3 55,300w粉丝特别节目！拍了一些大家想看的但是也太怪了？,12.3,66.5,710,知识,5-8 56,爆肝几个星期我建出了最细节的海岛小镇！！！,24.3,214.8,5545,游戏,5-8 57,让巴黎看到国人的美,24.4,186.1,784,生活,0-3 58,村里来了新成员漠叔好心劝上学做社会有用的人,11.9,119.5,1510,美食,10+ 59,《原神》角色演示-「菲米尼：海露幽响」,11.7,110.7,7387,游戏,5-8 61,当你碰到经验就会「瞬间暴毙」!!？,10.0,105.6,554,游戏,10+ 62,大学开学时的各种人|大学学习,9.0,294.9,224,搞笑,3-5 63,都什么年代谁还用传统方式结义？！！,9.0,60.6,3278,搞笑,10+ 64,【闽南婚宴】中式流水席天花板吃过一辈子忘不掉。,39.1,393.2,6.5,美食,10+ 65,口腔溃疡为什么是白色的？,18.1,318.9,3562,知识,3-5 66,我今年拼过最牛的积木！（上）,7.8,172.8,8298,动画,10+ 60,【太君の噩梦】打服日本一战成名比你想象的更夸张！苏联军神朱可夫,10.7,130.1,1218,知识,10+ [root@hadoop0 soft]# hadoop jar hadoop.jar com.lihaozhe.video.CategoryPercentJob 2023-11-12 12:16:16,089 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm2 2023-11-12 12:16:16,317 WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this. 2023-11-12 12:16:16,382 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/root/.staging/job_1699762297160_0002 2023-11-12 12:16:16,874 INFO input.FileInputFormat: Total input files to process : 1 2023-11-12 12:16:17,073 INFO mapreduce.JobSubmitter: number of splits:1 2023-11-12 12:16:17,323 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1699762297160_0002 2023-11-12 12:16:17,325 INFO mapreduce.JobSubmitter: Executing with tokens: [] 2023-11-12 12:16:17,564 INFO conf.Configuration: resource-types.xml not found 2023-11-12 12:16:17,564 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'. 2023-11-12 12:16:17,722 INFO impl.YarnClientImpl: Submitted application application_1699762297160_0002 2023-11-12 12:16:17,796 INFO mapreduce.Job: The url to track the job: http://hadoop1:8088/proxy/application_1699762297160_0002/ 2023-11-12 12:16:17,797 INFO mapreduce.Job: Running job: job_1699762297160_0002 2023-11-12 12:16:39,243 INFO mapreduce.Job: Job job_1699762297160_0002 running in uber mode : false 2023-11-12 12:16:39,250 INFO mapreduce.Job: map 0% reduce 0% 2023-11-12 12:16:52,214 INFO mapreduce.Job: map 100% reduce 0% 2023-11-12 12:17:00,304 INFO mapreduce.Job: map 100% reduce 100% 2023-11-12 12:17:01,332 INFO mapreduce.Job: Job job_1699762297160_0002 completed successfully 2023-11-12 12:17:01,495 INFO mapreduce.Job: Counters: 54 File System Counters FILE: Number of bytes read=123 FILE: Number of bytes written=559873 FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations=0 HDFS: Number of bytes read=8269 HDFS: Number of bytes written=86 HDFS: Number of read operations=8 HDFS: Number of large read operations=0 HDFS: Number of write operations=2 HDFS: Number of bytes read erasure-coded=0 Job Counters Launched map tasks=1 Launched reduce tasks=1 Data-local map tasks=1 Total time spent by all maps in occupied slots (ms)=9636 Total time spent by all reduces in occupied slots (ms)=5521 Total time spent by all map tasks (ms)=9636 Total time spent by all reduce tasks (ms)=5521 Total vcore-milliseconds taken by all map tasks=9636 Total vcore-milliseconds taken by all reduce tasks=5521 Total megabyte-milliseconds taken by all map tasks=9867264 Total megabyte-milliseconds taken by all reduce tasks=5653504 Map-Reduce Framework Map input records=100 Map output records=100 Map output bytes=1100 Map output materialized bytes=123 Input split bytes=108 Combine input records=100 Combine output records=9 Reduce input groups=9 Reduce shuffle bytes=123 Reduce input records=9 Reduce output records=9 Spilled Records=18 Shuffled Maps =1 Failed Shuffles=0 Merged Map outputs=1 GC time elapsed (ms)=144 CPU time spent (ms)=2050 Physical memory (bytes) snapshot=430870528 Virtual memory (bytes) snapshot=5175160832 Total committed heap usage (bytes)=337117184 Peak Map Physical memory (bytes)=265531392 Peak Map Virtual memory (bytes)=2586390528 Peak Reduce Physical memory (bytes)=165339136 Peak Reduce Virtual memory (bytes)=2588770304 Shuffle Errors BAD_ID=0 CONNECTION=0 IO_ERROR=0 WRONG_LENGTH=0 WRONG_MAP=0 WRONG_REDUCE=0 File Input Format Counters Bytes Read=8161 File Output Format Counters Bytes Written=86 [root@hadoop0 soft]# hdfs dfs -cat /video/dwd/part-r-00000 动画 10 影视 9 搞笑 14 游戏 13 生活 25 知识 16 美食 7 运动 3 音乐 3 [root@hadoop0 soft]# hadoop jar hadoop.jar com.lihaozhe.video.CategoryPercentWriteDB 2023-11-12 12:17:28,673 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm2 2023-11-12 12:17:28,899 WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this. 2023-11-12 12:17:28,953 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/root/.staging/job_1699762297160_0003 2023-11-12 12:17:29,393 INFO input.FileInputFormat: Total input files to process : 1 2023-11-12 12:17:29,572 INFO mapreduce.JobSubmitter: number of splits:1 2023-11-12 12:17:29,802 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1699762297160_0003 2023-11-12 12:17:29,803 INFO mapreduce.JobSubmitter: Executing with tokens: [] 2023-11-12 12:17:29,991 INFO conf.Configuration: resource-types.xml not found 2023-11-12 12:17:29,991 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'. 2023-11-12 12:17:30,085 INFO impl.YarnClientImpl: Submitted application application_1699762297160_0003 2023-11-12 12:17:30,151 INFO mapreduce.Job: The url to track the job: http://hadoop1:8088/proxy/application_1699762297160_0003/ 2023-11-12 12:17:30,152 INFO mapreduce.Job: Running job: job_1699762297160_0003 2023-11-12 12:17:38,342 INFO mapreduce.Job: Job job_1699762297160_0003 running in uber mode : false 2023-11-12 12:17:38,346 INFO mapreduce.Job: map 0% reduce 0% 2023-11-12 12:17:42,456 INFO mapreduce.Job: Task Id : attempt_1699762297160_0003_m_000000_0, Status : FAILED Error: java.io.IOException: com.mysql.cj.jdbc.Driver at org.apache.hadoop.mapreduce.lib.db.DBOutputFormat.getRecordWriter(DBOutputFormat.java:197) at org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.<init>(MapTask.java:660) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:780) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:348) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:178) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1899) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:172) 2023-11-12 12:17:48,617 INFO mapreduce.Job: Task Id : attempt_1699762297160_0003_m_000000_1, Status : FAILED Error: java.io.IOException: com.mysql.cj.jdbc.Driver at org.apache.hadoop.mapreduce.lib.db.DBOutputFormat.getRecordWriter(DBOutputFormat.java:197) at org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.<init>(MapTask.java:660) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:780) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:348) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:178) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1899) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:172) 2023-11-12 12:17:55,694 INFO mapreduce.Job: Task Id : attempt_1699762297160_0003_m_000000_2, Status : FAILED Error: java.io.IOException: com.mysql.cj.jdbc.Driver at org.apache.hadoop.mapreduce.lib.db.DBOutputFormat.getRecordWriter(DBOutputFormat.java:197) at org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.<init>(MapTask.java:660) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:780) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:348) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:178) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1899) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:172) ^C[root@hadoop0 soft]# scp $HADOOP_HOME/share/hadoop/common/lib/mysql-connector-j-0.33.jar root@hadoop1:$HADOOP_HOME/share/hadoop/common/lib/ mysql-connector-j-8.0.33.jar 100% 2423KB 38.1MB/s 00:00 [root@hadoop0 soft]# scp $HADOOP_HOME/share/hadoop/common/lib/mysql-connector-j-8.0.33.jar root@hadoop2:$HADOOP_HOME/share/hadoop/common/lib/ mysql-connector-j-8.0.33.jar 100% 2423KB 17.6MB/s 00:00 [root@hadoop0 soft]# stop-all.sh Stopping namenodes on [hadoop0 hadoop1] 上一次登录：日 11月 12 12:11:17 CST 2023pts/0 上 Stopping datanodes 上一次登录：日 11月 12 12:20:37 CST 2023pts/0 上 Stopping journal nodes [hadoop0 hadoop2 hadoop1] 上一次登录：日 11月 12 12:20:38 CST 2023pts/0 上 Stopping ZK Failover Controllers on NN hosts [hadoop0 hadoop1] 上一次登录：日 11月 12 12:20:43 CST 2023pts/0 上 Stopping nodemanagers 上一次登录：日 11月 12 12:20:46 CST 2023pts/0 上 Stopping resourcemanagers on [ hadoop0 hadoop1] 上一次登录：日 11月 12 12:20:47 CST 2023pts/0 上 hadoop0: WARNING: resourcemanager did not stop gracefully after 5 seconds: Trying to kill with kill -9 [root@hadoop0 soft]# start-all.sh Starting namenodes on [hadoop0 hadoop1] 上一次登录：日 11月 12 12:20:52 CST 2023pts/0 上 Starting datanodes 上一次登录：日 11月 12 12:21:10 CST 2023pts/0 上 Starting journal nodes [hadoop0 hadoop2 hadoop1] 上一次登录：日 11月 12 12:21:13 CST 2023pts/0 上 Starting ZK Failover Controllers on NN hosts [hadoop0 hadoop1] 上一次登录：日 11月 12 12:21:18 CST 2023pts/0 上 Starting resourcemanagers on [ hadoop0 hadoop1] 上一次登录：日 11月 12 12:21:22 CST 2023pts/0 上 Starting nodemanagers 上一次登录：日 11月 12 12:21:29 CST 2023pts/0 上 [root@hadoop0 soft]# hadoop jar hadoop.jar com.lihaozhe.video.CategoryPercentJob ^C[root@hadoop0 soft]# hadoop jar hadoop.jar com.lihaozhe.video.CategoryPercentJob 2023-11-12 12:22:14,496 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm2 2023-11-12 12:22:14,870 WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this. 2023-11-12 12:22:14,970 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/root/.staging/job_1699762900005_0001 2023-11-12 12:22:18,384 INFO input.FileInputFormat: Total input files to process : 1 2023-11-12 12:22:18,768 INFO mapreduce.JobSubmitter: number of splits:1 2023-11-12 12:22:19,387 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1699762900005_0001 2023-11-12 12:22:19,394 INFO mapreduce.JobSubmitter: Executing with tokens: [] 2023-11-12 12:22:19,826 INFO conf.Configuration: resource-types.xml not found 2023-11-12 12:22:19,827 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'. 2023-11-12 12:22:22,310 INFO impl.YarnClientImpl: Submitted application application_1699762900005_0001 2023-11-12 12:22:22,538 INFO mapreduce.Job: The url to track the job: http://hadoop1:8088/proxy/application_1699762900005_0001/ 2023-11-12 12:22:22,541 INFO mapreduce.Job: Running job: job_1699762900005_0001 2023-11-12 12:22:40,911 INFO mapreduce.Job: Job job_1699762900005_0001 running in uber mode : false 2023-11-12 12:22:40,918 INFO mapreduce.Job: map 0% reduce 0% 2023-11-12 12:22:48,048 INFO mapreduce.Job: map 100% reduce 0% 2023-11-12 12:22:59,175 INFO mapreduce.Job: map 100% reduce 100% 2023-11-12 12:23:00,201 INFO mapreduce.Job: Job job_1699762900005_0001 completed successfully 2023-11-12 12:23:00,362 INFO mapreduce.Job: Counters: 54 File System Counters FILE: Number of bytes read=123 FILE: Number of bytes written=559873 FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations=0 HDFS: Number of bytes read=8269 HDFS: Number of bytes written=86 HDFS: Number of read operations=8 HDFS: Number of large read operations=0 HDFS: Number of write operations=2 HDFS: Number of bytes read erasure-coded=0 Job Counters Launched map tasks=1 Launched reduce tasks=1 Data-local map tasks=1 Total time spent by all maps in occupied slots (ms)=4113 Total time spent by all reduces in occupied slots (ms)=8309 Total time spent by all map tasks (ms)=4113 Total time spent by all reduce tasks (ms)=8309 Total vcore-milliseconds taken by all map tasks=4113 Total vcore-milliseconds taken by all reduce tasks=8309 Total megabyte-milliseconds taken by all map tasks=4211712 Total megabyte-milliseconds taken by all reduce tasks=8508416 Map-Reduce Framework Map input records=100 Map output records=100 Map output bytes=1100 Map output materialized bytes=123 Input split bytes=108 Combine input records=100 Combine output records=9 Reduce input groups=9 Reduce shuffle bytes=123 Reduce input records=9 Reduce output records=9 Spilled Records=18 Shuffled Maps =1 Failed Shuffles=0 Merged Map outputs=1 GC time elapsed (ms)=135 CPU time spent (ms)=1890 Physical memory (bytes) snapshot=448860160 Virtual memory (bytes) snapshot=5168750592 Total committed heap usage (bytes)=359137280 Peak Map Physical memory (bytes)=289730560 Peak Map Virtual memory (bytes)=2581680128 Peak Reduce Physical memory (bytes)=159129600 Peak Reduce Virtual memory (bytes)=2587070464 Shuffle Errors BAD_ID=0 CONNECTION=0 IO_ERROR=0 WRONG_LENGTH=0 WRONG_MAP=0 WRONG_REDUCE=0 File Input Format Counters Bytes Read=8161 File Output Format Counters Bytes Written=86 [root@hadoop0 soft]# hdfs dfs -cat /video/dwd/part-r-00000 动画 10 影视 9 搞笑 14 游戏 13 生活 25 知识 16 美食 7 运动 3 音乐 3 [root@hadoop0 soft]# hadoop jar hadoop.jar com.lihaozhe.video.CategoryPercentWriteDB 2023-11-12 12:23:19,005 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm2 2023-11-12 12:23:19,225 WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this. 2023-11-12 12:23:19,272 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/root/.staging/job_1699762900005_0002 2023-11-12 12:23:19,667 INFO input.FileInputFormat: Total input files to process : 1 2023-11-12 12:23:19,845 INFO mapreduce.JobSubmitter: number of splits:1 2023-11-12 12:23:20,113 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1699762900005_0002 2023-11-12 12:23:20,115 INFO mapreduce.JobSubmitter: Executing with tokens: [] 2023-11-12 12:23:20,341 INFO conf.Configuration: resource-types.xml not found 2023-11-12 12:23:20,342 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'. 2023-11-12 12:23:20,435 INFO impl.YarnClientImpl: Submitted application application_1699762900005_0002 2023-11-12 12:23:20,483 INFO mapreduce.Job: The url to track the job: http://hadoop1:8088/proxy/application_1699762900005_0002/ 2023-11-12 12:23:20,484 INFO mapreduce.Job: Running job: job_1699762900005_0002 2023-11-12 12:23:36,840 INFO mapreduce.Job: Job job_1699762900005_0002 running in uber mode : false 2023-11-12 12:23:36,846 INFO mapreduce.Job: map 0% reduce 0% 2023-11-12 12:23:43,011 INFO mapreduce.Job: map 100% reduce 0% 2023-11-12 12:23:44,031 INFO mapreduce.Job: Job job_1699762900005_0002 completed successfully 2023-11-12 12:23:44,199 INFO mapreduce.Job: Counters: 33 File System Counters FILE: Number of bytes read=0 FILE: Number of bytes written=280015 FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations=0 HDFS: Number of bytes read=194 HDFS: Number of bytes written=0 HDFS: Number of read operations=2 HDFS: Number of large read operations=0 HDFS: Number of write operations=0 HDFS: Number of bytes read erasure-coded=0 Job Counters Launched map tasks=1 Data-local map tasks=1 Total time spent by all maps in occupied slots (ms)=3756 Total time spent by all reduces in occupied slots (ms)=0 Total time spent by all map tasks (ms)=3756 Total vcore-milliseconds taken by all map tasks=3756 Total megabyte-milliseconds taken by all map tasks=3846144 Map-Reduce Framework Map input records=9 Map output records=9 Input split bytes=108 Spilled Records=0 Failed Shuffles=0 Merged Map outputs=0 GC time elapsed (ms)=46 CPU time spent (ms)=800 Physical memory (bytes) snapshot=198701056 Virtual memory (bytes) snapshot=2584031232 Total committed heap usage (bytes)=136314880 Peak Map Physical memory (bytes)=198701056 Peak Map Virtual memory (bytes)=2584031232 File Input Format Counters Bytes Read=86 File Output Format Counters Bytes Written=0 [root@hadoop0 soft]# stop-all.sh Stopping namenodes on [hadoop0 hadoop1] 上一次登录：日 11月 12 12:21:31 CST 2023pts/0 上 Stopping datanodes 上一次登录：日 11月 12 12:25:18 CST 2023pts/0 上 Stopping journal nodes [hadoop0 hadoop2 hadoop1] 上一次登录：日 11月 12 12:25:20 CST 2023pts/0 上 Stopping ZK Failover Controllers on NN hosts [hadoop0 hadoop1] 上一次登录：日 11月 12 12:25:24 CST 2023pts/0 上 Stopping nodemanagers 上一次登录：日 11月 12 12:25:27 CST 2023pts/0 上 Stopping resourcemanagers on [ hadoop0 hadoop1] 上一次登录：日 11月 12 12:25:29 CST 2023pts/0 上 [root@hadoop0 soft]#
