MongoDB数据的导出导入及日志分析

本文涉及的产品
云数据库 MongoDB,通用型 2核4GB
简介: MongoDB数据的导出导入及日志分析

一、远程连接导出报错超时

mongodump -h 10.110.63.150:27017 -u'admin' -p'passwd!' --authenticationDatabase flowtest --db flowtest -o /home/mongod/bak > mongodump.log 2>&1 &
导出报错:
2022-11-17T18:58:54.663+0800    [###############.........]  flowtest.processDataObj  68313/106962  (63.9%)
2022-11-17T18:58:56.506+0800    Failed: error writing data for collection `flowtest.processDataObj` to disk: error reading collection: connection pool for 10.110.63.150:27017 was cleared because another operation failed with: connection(10.110.63.150:27017[-2]) incomplete read of message header: read tcp 7.7.7.11:53305->10.110.63.150:27017: i/o timeout

二、本地导出不会超时

bin/mongodump -u'admin' -p'passwd!' --authenticationDatabase flowtest --db flowtest -o /home/nrms/thirdparty/mongodb/mongodb-linux-x86_64-3.6.13/bak > mongodump.log 2>&1 &
同时可见,导出不压缩,数据量比源目录大了很多,大约2倍,侧面说明了mongodb库本身的内部数据压缩效果不错:
1.4G    data
2.8G    bak
233M    bak.tar

三、导入1个集合 history_task 到 testdb 库

mongorestore -h node1:20000 -u'liking' -p'passwd!2#' --authenticationDatabase testdb --db testdb --collection history_task /home/mongod/bak/flowtest/history_task.bson > mongorestore.log 2>&1 &

四、导入全部

mongorestore -h node1:20000 -u'admin' -p'passwd!2#' --authenticationDatabase admin --db flowtest /home/mongod/bak/flowtest > mongorestore.log 2>&1 &

五、导入日志分析

确定还原的集合列表,并读取各个集合的元数据:

2022-11-18T09:59:51.909+0800    The --db and --collection flags are deprecated for this use-case; please use --nsInclude instead, i.e. with --nsInclude=${DATABASE}.${COLLECTION}
2022-11-18T09:59:51.910+0800    building a list of collections to restore from /home/mongod/bak/flowtest dir
2022-11-18T09:59:51.911+0800    reading metadata for flowtest.history_task from metadata
2022-11-18T09:59:51.912+0800    reading metadata for flowtest.processConfiguration from metadata
2022-11-18T09:59:51.925+0800    reading metadata for flowtest.processDataObj from metadata
2022-11-18T09:59:51.930+0800    reading metadata for flowtest.processDataObjInit from metadata
2022-11-18T09:59:51.931+0800    reading metadata for flowtest.snapshotResource from metadata
2022-11-18T09:59:51.932+0800    reading metadata for flowtest.activityConfiguration from metadata
2022-11-18T09:59:51.945+0800    reading metadata for flowtest.processDataObjDraft from metadata
2022-11-18T09:59:51.945+0800    reading metadata for flowtest.processsDataObjInit from metadata
2022-11-18T09:59:51.946+0800    reading metadata for flowtest.role from metadata

同时开启4个并发 restoring:

2022-11-18T09:59:52.290+0800    restoring flowtest.snapshotResource
2022-11-18T09:59:52.335+0800    restoring flowtest.history_task
2022-11-18T09:59:52.369+0800    restoring flowtest.processDataObj
2022-11-18T09:59:52.396+0800    restoring flowtest.processDataObjInit

每 finished 1个,即再开始 restoring 另1个:

2022-11-18T09:59:52.824+0800    finished restoring flowtest.snapshotResource (790 documents, 0 failures)
2022-11-18T09:59:52.878+0800    restoring flowtest.processDataObjDraft
2022-11-18T09:59:53.096+0800    finished restoring flowtest.processDataObjDraft (123 documents, 0 failures)
2022-11-18T09:59:53.159+0800    restoring flowtest.activityConfiguration

每 finished 2个,即再开始 restoring 另2个:

2022-11-18T09:59:53.335+0800    finished restoring flowtest.activityConfiguration (1308 documents, 0 failures)
2022-11-18T09:59:53.382+0800    finished restoring flowtest.history_task (3629 documents, 0 failures)
2022-11-18T09:59:53.464+0800    restoring flowtest.processConfiguration
2022-11-18T09:59:53.544+0800    restoring flowtest.processsDataObjInit
2022-11-18T09:59:53.577+0800    finished restoring flowtest.processConfiguration (168 documents, 0 failures)
2022-11-18T09:59:53.622+0800    restoring flowtest.role from /home/mongod/bak/flowtest/role.bson
2022-11-18T09:59:55.108+0800    finished restoring flowtest.role (10 documents, 0 failures)
2022-11-18T09:59:55.108+0800    finished restoring flowtest.processsDataObjInit (4 documents, 0 failures)
2022-11-18T10:00:28.471+0800    finished restoring flowtest.processDataObjInit (96008 documents, 0 failures)
2022-11-18T10:00:45.768+0800    finished restoring flowtest.processDataObj (106968 documents, 0 failures)

开始恢复 indexes:

2022-11-18T10:00:45.781+0800    restoring indexes for collection flowtest.processDataObjInit from metadata
2022-11-18T10:00:45.799+0800    index: &idx.IndexDocument{Options:primitive.M{"name":"flowId_1_activityConfiguration.activityNameEn_1", "ns":"flowtest.processDataObjInit", "v":2}, Key:primitive.D{primitive.E{Key:"flowId", Value:1}, primitive.E{Key:"activityConfiguration.activityNameEn", Value:1}}, PartialFilterExpression:primitive.D(nil)}
2022-11-18T10:00:45.799+0800    no indexes to restore for collection flowtest.snapshotResource
2022-11-18T10:00:45.799+0800    no indexes to restore for collection flowtest.activityConfiguration
2022-11-18T10:00:45.799+0800    no indexes to restore for collection flowtest.history_task
2022-11-18T10:00:45.799+0800    restoring indexes for collection flowtest.processDataObj from metadata
2022-11-18T10:00:45.799+0800    index: &idx.IndexDocument{Options:primitive.M{"name":"flowId_1_activityConfiguration.activityNameEn_1", "ns":"flowtest.processDataObj", "v":2}, Key:primitive.D{primitive.E{Key:"flowId", Value:1}, primitive.E{Key:"activityConfiguration.activityNameEn", Value:1}}, PartialFilterExpression:primitive.D(nil)}
2022-11-18T10:00:45.799+0800    index: &idx.IndexDocument{Options:primitive.M{"name":"flowNo_1", "ns":"flowtest.processDataObj", "v":2}, Key:primitive.D{primitive.E{Key:"flowNo", Value:1}}, PartialFilterExpression:primitive.D(nil)}
2022-11-18T10:00:45.799+0800    no indexes to restore for collection flowtest.processsDataObjInit
2022-11-18T10:00:45.799+0800    no indexes to restore for collection flowtest.processConfiguration
2022-11-18T10:00:45.799+0800    no indexes to restore for collection flowtest.processDataObjDraft
2022-11-18T10:00:45.800+0800    index: &idx.IndexDocument{Options:primitive.M{"name":"flowNo_1", "ns":"flowtest.processDataObjInit", "v":2}, Key:primitive.D{primitive.E{Key:"flowNo", Value:1}}, PartialFilterExpression:primitive.D(nil)}
2022-11-18T10:00:45.800+0800    no indexes to restore for collection flowtest.role

最后的总结:恢复了 209008 document(s)

2022-11-18T10:00:48.031+0800    209008 document(s) restored successfully. 0 document(s) failed to restore.
相关实践学习
MongoDB数据库入门
MongoDB数据库入门实验。
快速掌握 MongoDB 数据库
本课程主要讲解MongoDB数据库的基本知识,包括MongoDB数据库的安装、配置、服务的启动、数据的CRUD操作函数使用、MongoDB索引的使用(唯一索引、地理索引、过期索引、全文索引等)、MapReduce操作实现、用户管理、Java对MongoDB的操作支持(基于2.x驱动与3.x驱动的完全讲解)。 通过学习此课程,读者将具备MongoDB数据库的开发能力,并且能够使用MongoDB进行项目开发。   相关的阿里云产品:云数据库 MongoDB版 云数据库MongoDB版支持ReplicaSet和Sharding两种部署架构,具备安全审计,时间点备份等多项企业能力。在互联网、物联网、游戏、金融等领域被广泛采用。 云数据库MongoDB版(ApsaraDB for MongoDB)完全兼容MongoDB协议,基于飞天分布式系统和高可靠存储引擎,提供多节点高可用架构、弹性扩容、容灾、备份回滚、性能优化等解决方案。 产品详情: https://www.aliyun.com/product/mongodb
目录
相关文章
|
11天前
|
监控 NoSQL MongoDB
mongoDB查看数据的插入日志
【5月更文挑战第9天】mongoDB查看数据的插入日志
312 4
|
11天前
|
监控 NoSQL MongoDB
mongoDB查看数据的插入日志
【5月更文挑战第2天】mongoDB查看数据的插入日志
310 0
|
18小时前
|
监控 NoSQL MongoDB
mongoDB查看数据的插入日志
【5月更文挑战第22天】mongoDB查看数据的插入日志
12 3
|
2天前
|
开发工具 iOS开发
如何导出iPhone手机中app共享文件夹的文件与闪退日志的收集
如何导出iPhone手机中app共享文件夹的文件与闪退日志的收集
7 1
|
11天前
|
存储 监控 NoSQL
【MongoDB 专栏】MongoDB 的日志管理与分析
【5月更文挑战第11天】MongoDB日志管理与分析至关重要,包括系统日志和操作日志,用于监控、故障排查和性能优化。合理配置日志详细程度、存储位置和保留策略,使用日志分析工具提升效率,发现性能瓶颈和安全性问题。日志分析有助于优化查询、调整配置,确保数据安全,并可与其他监控系统集成。面对日志量增长的挑战,需采用新技术如分布式存储和数据压缩来保障存储和传输。随着技术发展,不断进化日志管理与分析能力,以支持MongoDB的稳定高效运行。
【MongoDB 专栏】MongoDB 的日志管理与分析
|
11天前
|
NoSQL 数据管理 MongoDB
数据管理DMS产品使用合集之如何通过阿里云的数据管理服务(DMS)导出MongoDB数据
阿里云数据管理DMS提供了全面的数据管理、数据库运维、数据安全、数据迁移与同步等功能,助力企业高效、安全地进行数据库管理和运维工作。以下是DMS产品使用合集的详细介绍。
|
11天前
|
SQL 存储 关系型数据库
mysql bin-log日志导出
mysql bin-log日志导出
|
11天前
|
JSON NoSQL MongoDB
mongodb导出聚合查询的数据
mongodb导出聚合查询的数据
|
11天前
|
JSON NoSQL MongoDB
mongoDB数据的导出导入
mongoDB数据的导出导入
104 0
|
9天前
|
存储 SQL NoSQL
什么是 MongoDB,为什么它是当今最受欢迎的数据库之一?
什么是 MongoDB,为什么它是当今最受欢迎的数据库之一?

热门文章

最新文章