[20170220]快速拷贝文件在linux磁盘之间.txt
--上个星期五要将1.3T的文件(每个都很大)从1个磁盘移到另外的磁盘,测试发现cp 根本无法忍受.几乎要8个小时问题感觉出在文件系统
--的cache上,google看了一些连接:
1.首先cp慢的主要原因我感觉应该出现在文件系统缓存上,这个时候使用缓存没有必要,因为仅仅拷贝一次,使用缓存有点多余.
如果通过dstat观察可以发现如下,开始一段很好,读写可以达到200M,写入更快到400M.但是一般15秒上下就停滞下来.等待4x秒,
有开始加快.
通过free观察free内存越来越少.最后基本保持不动.
2.我看了许多这方面的文档:
--//一个最简单的方法就是dd 在参数iflag=direct,oflag=direct参数,绕过缓存,但是这样的缺点就是要编程,实际上也不是很麻烦.
--//注有一些文档提供nocache参数:
http://www.gnu.org/software/coreutils/manual/html_node/dd-invocation.html
'nocache'
Request to discard the system data cache for a file. When count=0 all cached data for the file is specified, otherwise
the cache is dropped for the processed portion of the file. Also when count=0, failure to discard the cache is diagnosed
and reflected in the exit status.
Note data that is not already persisted to storage will not be discarded from cache, so note the use of the "sync"
options in the examples below, which are used to maximize the effectiveness of the 'nocache' flag.
Here are some usage examples:
# Advise to drop cache for whole file
dd if=ifile iflag=nocache count=0
# Ensure drop cache for the whole file
dd of=ofile oflag=nocache conv=notrunc,fdatasync count=0
# Drop cache for part of file
dd if=ifile iflag=nocache skip=10 count=10 of=/dev/null
# Stream data using just the read-ahead cache.
# See also the 'direct' flag.
dd if=ifile of=ofile iflag=nocache oflag=nocache,sync
--//不过我使用的dd版本没有nocache参数.
--//在google的过程中,发现这个链接https://github.com/Feh/nocache:
nocache - minimize filesystem caching effects
The nocache tool tries to minimize the effect an application has on the Linux file system cache. This is done by
intercepting the open and close system calls and calling posix_fadvise with the POSIX_FADV_DONTNEED parameter. Because
the library remembers which pages (ie., 4K-blocks of the file) were already in file system cache when the file was
opened, these will not be marked as "don't need", because other applications might need that, although they are not
actively used (think: hot standby).
Use case: backup processes that should not interfere with the present state of the cache.
--//决定在自己的测试环境先测试看看.而且debian已经集成了这个包,感觉应该很可靠.
3.下载安装很简单,就是一般安装的3步曲:
解压=>make=>make install ( 注有一些要先执行./configure,这里不需要.)
4.测试:
--首先清除该文件缓存
# cachedel datafile/system01.bdf
# cachedel /u01/system01.bdf
# time /bin/cp system01.bdf /u01/system01.bdf
real 0m51.536s
user 0m0.272s
sys 0m18.136s
# cachestats datafile/system01.bdf
pages in cache: 1024002/1024002 (100.0%) [filesize=4096008.0K, pagesize=4K]
--//可以发现拷贝的文件缓存100%,备份出来的文件也一样. 大约4096/52=78M/秒.
--完成后free的显示:
# free
total used free shared buffers cached
Mem: 132261196 131797060 464136 0 591348 123554608
-/+ buffers/cache: 7651104 124610092
Swap: 31455264 780220 30675044
--//可以发现free内存剩余很小.如果你再重复上面的命令/bin/cp system01.bdf /u01/system01.bdf,可以发现飞快,因为文件已经缓
--//存了.
5.测试使用nocache:
# cachedel datafile/system01.bdf
# cachedel /u01/system01.bdf
# time nocache /bin/cp system01.bdf /u01/system01.bdf
real 0m50.444s
user 0m0.217s
sys 0m18.698s
--//时间相差不大.对比上面,因为文件没有缓存.
--//在拷贝的过程中执行:
# cachestats datafile/system01.bdf
pages in cache: 172291/1024002 (16.8%) [filesize=4096008.0K, pagesize=4K]
# cachestats datafile/system01.bdf
pages in cache: 190499/1024002 (18.6%) [filesize=4096008.0K, pagesize=4K]
....
# cachestats datafile/system01.bdf
pages in cache: 340483/1024002 (33.3%) [filesize=4096008.0K, pagesize=4K]
# cachestats datafile/system01.bdf
pages in cache: 365411/1024002 (35.7%) [filesize=4096008.0K, pagesize=4K]
...
# cachestats datafile/system01.bdf
pages in cache: 24/1024002 (0.0%) [filesize=4096008.0K, pagesize=4K]
--//可以发现cache不断增加,到结束时清除cache.观察备份的文件也一样.
--//也可以看出nocache的作用实际上就是结束后清除缓存.
# cachestats /u01/system01.bdf
pages in cache: 6/1024002 (0.0%) [filesize=4096008.0K, pagesize=4K]
--完成后free的显示:
# free
total used free shared buffers cached
Mem: 132261196 121362884 10898312 0 597632 113115280
-/+ buffers/cache: 7649972 124611224
Swap: 31455264 780220 30675044
--//可以发现free依旧保持很多内存.
6.最好测试使用dd看看:
# cachedel datafile/system01.bdf
# cachedel /u01/system01.bdf
--//注有些访问提到iflag=nocache oflag=nocache参数,我这个版本不支持这个参数.感觉好像使用这个参数会快一些.
# time dd if=system01.bdf of=/u01/system01.bdf bs=1024M iflag=direct oflag=direct
3+1 records in
3+1 records out
4194312192 bytes (4.2 GB) copied, 80.2868 seconds, 52.2 MB/s
real 1m20.603s
user 0m0.000s
sys 1m2.941s
--//这个有一点慢.
# cachestats /u01/system01.bdf
pages in cache: 0/1024002 (0.0%) [filesize=4096008.0K, pagesize=4K]
# cachestats datafile/system01.bdf
pages in cache: 18/1024002 (0.0%) [filesize=4096008.0K, pagesize=4K]
--//可以发现没有缓存.
# free
total used free shared buffers cached
Mem: 132261196 120296788 11964408 0 586588 112113960
-/+ buffers/cache: 7596240 124664956
Swap: 31455264 786416 30668848
7.一些补充:
--//googel还可以发现rsync有参数,好像官方的版本不支持这个参数.
--drop-cache that works local
--remote-drop-cache that works on remote
--//网上也可以找到tar的方式,我的测试效果一样,文件也会缓存.
# tar cf - . | ( cd dest ; tar xvf - )
# tar cf - datafile | tar xvf - -C /mnt
8.再补充网络拷贝:
--我一般借助tar+ssh+pigz模式,例子:
#tar cf - oracle -I pigz | ssh oracle@ip_address tar xvf - -I pigz -C /u01/app/
--//如果不支持-I参数,使用--use-compress-program ,pigz要另外安装,另外有文章提高lz4压缩工具,有机会另行测试.
--//感觉这个使用nocache也可以获得好的效果,不再测试,有机会再测试吧^_^.
--//相关讨论:
http://intermediatesql.com/linux/scrap-the-scp-how-to-copy-data-fast-using-pigz-and-nc/
http://www.gnu.org/software/coreutils/manual/html_node/dd-invocation.html
后记:最后确定是存储设置有问题,以交给同事解决,具体细节以后有机会写出来.
