MongoDB数据的导出导入及日志分析

本文涉及的产品
云数据库 MongoDB,独享型 2核8GB
推荐场景:
构建全方位客户视图
日志服务 SLS,月写入数据量 50GB 1个月
简介: MongoDB数据的导出导入及日志分析

一、远程连接导出报错超时

mongodump -h 10.110.63.150:27017 -u'admin' -p'passwd!' --authenticationDatabase flowtest --db flowtest -o /home/mongod/bak > mongodump.log 2>&1 &
导出报错:
2022-11-17T18:58:54.663+0800    [###############.........]  flowtest.processDataObj  68313/106962  (63.9%)
2022-11-17T18:58:56.506+0800    Failed: error writing data for collection `flowtest.processDataObj` to disk: error reading collection: connection pool for 10.110.63.150:27017 was cleared because another operation failed with: connection(10.110.63.150:27017[-2]) incomplete read of message header: read tcp 7.7.7.11:53305->10.110.63.150:27017: i/o timeout

二、本地导出不会超时

bin/mongodump -u'admin' -p'passwd!' --authenticationDatabase flowtest --db flowtest -o /home/nrms/thirdparty/mongodb/mongodb-linux-x86_64-3.6.13/bak > mongodump.log 2>&1 &
同时可见,导出不压缩,数据量比源目录大了很多,大约2倍,侧面说明了mongodb库本身的内部数据压缩效果不错:
1.4G    data
2.8G    bak
233M    bak.tar

三、导入1个集合 history_task 到 testdb 库

mongorestore -h node1:20000 -u'liking' -p'passwd!2#' --authenticationDatabase testdb --db testdb --collection history_task /home/mongod/bak/flowtest/history_task.bson > mongorestore.log 2>&1 &

四、导入全部

mongorestore -h node1:20000 -u'admin' -p'passwd!2#' --authenticationDatabase admin --db flowtest /home/mongod/bak/flowtest > mongorestore.log 2>&1 &

五、导入日志分析

确定还原的集合列表,并读取各个集合的元数据:

2022-11-18T09:59:51.909+0800    The --db and --collection flags are deprecated for this use-case; please use --nsInclude instead, i.e. with --nsInclude=${DATABASE}.${COLLECTION}
2022-11-18T09:59:51.910+0800    building a list of collections to restore from /home/mongod/bak/flowtest dir
2022-11-18T09:59:51.911+0800    reading metadata for flowtest.history_task from metadata
2022-11-18T09:59:51.912+0800    reading metadata for flowtest.processConfiguration from metadata
2022-11-18T09:59:51.925+0800    reading metadata for flowtest.processDataObj from metadata
2022-11-18T09:59:51.930+0800    reading metadata for flowtest.processDataObjInit from metadata
2022-11-18T09:59:51.931+0800    reading metadata for flowtest.snapshotResource from metadata
2022-11-18T09:59:51.932+0800    reading metadata for flowtest.activityConfiguration from metadata
2022-11-18T09:59:51.945+0800    reading metadata for flowtest.processDataObjDraft from metadata
2022-11-18T09:59:51.945+0800    reading metadata for flowtest.processsDataObjInit from metadata
2022-11-18T09:59:51.946+0800    reading metadata for flowtest.role from metadata

同时开启4个并发 restoring:

2022-11-18T09:59:52.290+0800    restoring flowtest.snapshotResource
2022-11-18T09:59:52.335+0800    restoring flowtest.history_task
2022-11-18T09:59:52.369+0800    restoring flowtest.processDataObj
2022-11-18T09:59:52.396+0800    restoring flowtest.processDataObjInit

每 finished 1个,即再开始 restoring 另1个:

2022-11-18T09:59:52.824+0800    finished restoring flowtest.snapshotResource (790 documents, 0 failures)
2022-11-18T09:59:52.878+0800    restoring flowtest.processDataObjDraft
2022-11-18T09:59:53.096+0800    finished restoring flowtest.processDataObjDraft (123 documents, 0 failures)
2022-11-18T09:59:53.159+0800    restoring flowtest.activityConfiguration

每 finished 2个,即再开始 restoring 另2个:

2022-11-18T09:59:53.335+0800    finished restoring flowtest.activityConfiguration (1308 documents, 0 failures)
2022-11-18T09:59:53.382+0800    finished restoring flowtest.history_task (3629 documents, 0 failures)
2022-11-18T09:59:53.464+0800    restoring flowtest.processConfiguration
2022-11-18T09:59:53.544+0800    restoring flowtest.processsDataObjInit
2022-11-18T09:59:53.577+0800    finished restoring flowtest.processConfiguration (168 documents, 0 failures)
2022-11-18T09:59:53.622+0800    restoring flowtest.role from /home/mongod/bak/flowtest/role.bson
2022-11-18T09:59:55.108+0800    finished restoring flowtest.role (10 documents, 0 failures)
2022-11-18T09:59:55.108+0800    finished restoring flowtest.processsDataObjInit (4 documents, 0 failures)
2022-11-18T10:00:28.471+0800    finished restoring flowtest.processDataObjInit (96008 documents, 0 failures)
2022-11-18T10:00:45.768+0800    finished restoring flowtest.processDataObj (106968 documents, 0 failures)

开始恢复 indexes:

2022-11-18T10:00:45.781+0800    restoring indexes for collection flowtest.processDataObjInit from metadata
2022-11-18T10:00:45.799+0800    index: &idx.IndexDocument{Options:primitive.M{"name":"flowId_1_activityConfiguration.activityNameEn_1", "ns":"flowtest.processDataObjInit", "v":2}, Key:primitive.D{primitive.E{Key:"flowId", Value:1}, primitive.E{Key:"activityConfiguration.activityNameEn", Value:1}}, PartialFilterExpression:primitive.D(nil)}
2022-11-18T10:00:45.799+0800    no indexes to restore for collection flowtest.snapshotResource
2022-11-18T10:00:45.799+0800    no indexes to restore for collection flowtest.activityConfiguration
2022-11-18T10:00:45.799+0800    no indexes to restore for collection flowtest.history_task
2022-11-18T10:00:45.799+0800    restoring indexes for collection flowtest.processDataObj from metadata
2022-11-18T10:00:45.799+0800    index: &idx.IndexDocument{Options:primitive.M{"name":"flowId_1_activityConfiguration.activityNameEn_1", "ns":"flowtest.processDataObj", "v":2}, Key:primitive.D{primitive.E{Key:"flowId", Value:1}, primitive.E{Key:"activityConfiguration.activityNameEn", Value:1}}, PartialFilterExpression:primitive.D(nil)}
2022-11-18T10:00:45.799+0800    index: &idx.IndexDocument{Options:primitive.M{"name":"flowNo_1", "ns":"flowtest.processDataObj", "v":2}, Key:primitive.D{primitive.E{Key:"flowNo", Value:1}}, PartialFilterExpression:primitive.D(nil)}
2022-11-18T10:00:45.799+0800    no indexes to restore for collection flowtest.processsDataObjInit
2022-11-18T10:00:45.799+0800    no indexes to restore for collection flowtest.processConfiguration
2022-11-18T10:00:45.799+0800    no indexes to restore for collection flowtest.processDataObjDraft
2022-11-18T10:00:45.800+0800    index: &idx.IndexDocument{Options:primitive.M{"name":"flowNo_1", "ns":"flowtest.processDataObjInit", "v":2}, Key:primitive.D{primitive.E{Key:"flowNo", Value:1}}, PartialFilterExpression:primitive.D(nil)}
2022-11-18T10:00:45.800+0800    no indexes to restore for collection flowtest.role

最后的总结:恢复了 209008 document(s)

2022-11-18T10:00:48.031+0800    209008 document(s) restored successfully. 0 document(s) failed to restore.
目录
相关文章
|
4天前
|
数据采集 运维 监控
不重启、不重写、不停机:SLS 软删除如何实现真正的“无感数据急救”?
SLS 全新推出的「软删除」功能,以接近索引查询的性能,解决了数据应急删除与脏数据治理的痛点。2 分钟掌握这一数据管理神器。
|
1月前
|
存储 缓存 Apache
StarRocks+Paimon 落地阿里日志采集:万亿级实时数据秒级查询
A+流量分析平台是阿里集团统一的全域流量数据分析平台,致力于通过埋点、采集、计算构建流量数据闭环,助力业务提升流量转化。面对万亿级日志数据带来的写入与查询挑战,平台采用Flink+Paimon+StarRocks技术方案,实现高吞吐写入与秒级查询,优化存储成本与扩展性,提升日志分析效率。
245 1
|
4月前
|
存储 缓存 Apache
StarRocks+Paimon 落地阿里日志采集:万亿级实时数据秒级查询
本文介绍了阿里集团A+流量分析平台的日志查询优化方案,针对万亿级日志数据的写入与查询挑战,提出基于Flink、Paimon和StarRocks的技术架构。通过Paimon存储日志数据,结合StarRocks高效计算能力,实现秒级查询性能。具体包括分桶表设计、数据缓存优化及文件大小控制等措施,解决高并发、大数据量下的查询效率问题。最终,日志查询耗时从分钟级降至秒级,显著提升业务响应速度,并为未来更低存储成本、更高性能及更多业务场景覆盖奠定基础。
|
1月前
|
存储 关系型数据库 数据库
【赵渝强老师】PostgreSQL数据库的WAL日志与数据写入的过程
PostgreSQL中的WAL(预写日志)是保证数据完整性的关键技术。在数据修改前,系统会先将日志写入WAL,确保宕机时可通过日志恢复数据。它减少了磁盘I/O,提升了性能,并支持手动切换日志文件。WAL文件默认存储在pg_wal目录下,采用16进制命名规则。此外,PostgreSQL提供pg_waldump工具解析日志内容。
151 0
|
1月前
|
数据采集 运维 监控
|
3月前
|
存储 NoSQL MongoDB
Docker中安装MongoDB并配置数据、日志、配置文件持久化。
现在,你有了一个运行在Docker中的MongoDB,它拥有自己的小空间,对高楼大厦的崩塌视而不见(会话丢失和数据不持久化的问题)。这个MongoDB的数据、日志、配置文件都会妥妥地保存在你为它精心准备的地方,天旋地转,它也不会失去一丁点儿宝贵的记忆(即使在容器重启后)。
391 4
|
4月前
|
监控 容灾 算法
阿里云 SLS 多云日志接入最佳实践:链路、成本与高可用性优化
本文探讨了如何高效、经济且可靠地将海外应用与基础设施日志统一采集至阿里云日志服务(SLS),解决全球化业务扩展中的关键挑战。重点介绍了高性能日志采集Agent(iLogtail/LoongCollector)在海外场景的应用,推荐使用LoongCollector以获得更优的稳定性和网络容错能力。同时分析了多种网络接入方案,包括公网直连、全球加速优化、阿里云内网及专线/CEN/VPN接入等,并提供了成本优化策略和多目标发送配置指导,帮助企业构建稳定、低成本、高可用的全球日志系统。
615 55
|
9月前
|
监控 安全 Apache
什么是Apache日志?为什么Apache日志分析很重要?
Apache是全球广泛使用的Web服务器软件,支持超过30%的活跃网站。它通过接收和处理HTTP请求,与后端服务器通信,返回响应并记录日志,确保网页请求的快速准确处理。Apache日志分为访问日志和错误日志,对提升用户体验、保障安全及优化性能至关重要。EventLog Analyzer等工具可有效管理和分析这些日志,增强Web服务的安全性和可靠性。
254 9

推荐镜像

更多