[20130109]使用bbcp传输文件.txt
今天看了一些blog，发现一些blog提到了bbcp。使用它来传输大文件据说很快。
相关站点：
http://www.slac.stanford.edu/~abh/bbcp/bin/
http://www.slac.stanford.edu/~abh/bbcp/
--不行，可以自己下载编译：
git clone http://www.slac.stanford.edu/~abh/bbcp/bbcp.git
cd bbcp
make
cp bin/amd64_linux26/bbcp /usr/local/bin/bbcp
1.开始测试：
一看文档，晕！这么多参数，使用简单一点看看:
http://www.slac.stanford.edu/~abh/bbcp/
bbcp -P 2 /data/orcl/users01.dbf oracle@remote_host:/tmp/users01.dbf
--提示，执行文件名明明在那里，为什么不行。直接进入/usr/local/bin目录执行，也一样。
bash: bbcp: command not found
bbcp: bbcp unexpectedly terminated on XXX.XXX.XXX.XXX
--仔细看了相关文档
bbcp assumes the remote system's non-interactive environment contains the path to the bbcp utility.
This can be determined by with the following command:
ssh remotesystem which bbcp
--也就是远端也需要这个一个执行文件。将bbcp拷贝过去。
# bbcp -P 2 /data/orcl/users01.dbf oracle@remote_host:/tmp/users01.dbf
oracle@xxX.xxx.xxx.xxx's password:
bbcp: Creating /tmp/users01.dbf
bbcp: 130109 16:50:04 5% done; 55.7 MB/s
bbcp: 130109 16:50:06 15% done; 63.2 MB/s
bbcp: 130109 16:50:08 25% done; 66.1 MB/s
bbcp: 130109 16:50:10 36% done; 71.3 MB/s
bbcp: 130109 16:50:12 47% done; 72.8 MB/s
bbcp: 130109 16:50:14 55% done; 70.0 MB/s
bbcp: 130109 16:50:16 65% done; 70.7 MB/s
bbcp: 130109 16:50:18 74% done; 70.4 MB/s
bbcp: 130109 16:50:20 84% done; 70.4 MB/s
bbcp: 130109 16:50:22 95% done; 71.6 MB/s
--文件大小1.4G,大约20秒完成。
# du -sm /tmp/users01.dbf
1467 /tmp/users01.dbf
2.使用scp做一个比较:
scp /data/orcl.20121016/users01.dbf oracle@192.168.101.209:/tmp/users01.dbf
users01.dbf 100% 1465MB 45.8MB/s 00:32
lftp也做了一个测试，稍微快一点，大约30秒。1536180224 bytes transferred in 30 seconds (48.95M/s)
3.另外bbcp有一个好处，如果拷贝文件已经存在，提示文件存在直接退出。
加入-f 参数
--force | -f
forces the copy by erasing the target prior to copying the source file. By default, if the target already exists for the
source file, the copy fails. See also the –K option modifier.
如果在拷贝文件时，在远程机器使用ps -elf | grep bbcp 看。
# ps -elf | grep bbcp | grep -v grep
0 S oracle 7398 7397 1 81 0 - 12251 wait 17:04 ? 00:00:00 bbcp SNK
1 S oracle 7427 7398 68 76 0 - 22495 184466 17:04 ? 00:00:13 bbcp SNK
--会有2个进程。
看了一些文档从调整性能讲重要的两个参数（我自己认为）,我估计如果机器忙，带宽不足，也许会影响这个网络。
1.2.1 Window Size (-w)
The first and most important option is –w. This determines the TCP window size as well as the default I/O size. The
default of 128Kis usually good enough for LAN and larger values are likely to hurt performance.
1.2.2 Streams (-s)
The -- s options is the second most important tuning parameter. This specifies the number of parallel TCP streams. A
na?ve explanation would say that streams can make up for not having a large enough window. The idea is that if you can’
t get enough packets moving in a single window, then create multiple windows and run them simultaneously. This is only
partially true. While multiple streams do provide multiple windows, multiple windows also parallelize traffic with
independent time-outs, re-transmissions, and greatly improved I/O overlapping. This is a cumulative effect that
dramatically increases the overall bandwidth utilization. But, too much of a good thing can also be bad, as you will see
below.
今天看了一些blog，发现一些blog提到了bbcp。使用它来传输大文件据说很快。
相关站点：
http://www.slac.stanford.edu/~abh/bbcp/bin/
http://www.slac.stanford.edu/~abh/bbcp/
--不行，可以自己下载编译：
git clone http://www.slac.stanford.edu/~abh/bbcp/bbcp.git
cd bbcp
make
cp bin/amd64_linux26/bbcp /usr/local/bin/bbcp
1.开始测试：
一看文档，晕！这么多参数，使用简单一点看看:
http://www.slac.stanford.edu/~abh/bbcp/
bbcp -P 2 /data/orcl/users01.dbf oracle@remote_host:/tmp/users01.dbf
--提示，执行文件名明明在那里，为什么不行。直接进入/usr/local/bin目录执行，也一样。
bash: bbcp: command not found
bbcp: bbcp unexpectedly terminated on XXX.XXX.XXX.XXX
--仔细看了相关文档
bbcp assumes the remote system's non-interactive environment contains the path to the bbcp utility.
This can be determined by with the following command:
ssh remotesystem which bbcp
--也就是远端也需要这个一个执行文件。将bbcp拷贝过去。
# bbcp -P 2 /data/orcl/users01.dbf oracle@remote_host:/tmp/users01.dbf
oracle@xxX.xxx.xxx.xxx's password:
bbcp: Creating /tmp/users01.dbf
bbcp: 130109 16:50:04 5% done; 55.7 MB/s
bbcp: 130109 16:50:06 15% done; 63.2 MB/s
bbcp: 130109 16:50:08 25% done; 66.1 MB/s
bbcp: 130109 16:50:10 36% done; 71.3 MB/s
bbcp: 130109 16:50:12 47% done; 72.8 MB/s
bbcp: 130109 16:50:14 55% done; 70.0 MB/s
bbcp: 130109 16:50:16 65% done; 70.7 MB/s
bbcp: 130109 16:50:18 74% done; 70.4 MB/s
bbcp: 130109 16:50:20 84% done; 70.4 MB/s
bbcp: 130109 16:50:22 95% done; 71.6 MB/s
--文件大小1.4G,大约20秒完成。
# du -sm /tmp/users01.dbf
1467 /tmp/users01.dbf
2.使用scp做一个比较:
scp /data/orcl.20121016/users01.dbf oracle@192.168.101.209:/tmp/users01.dbf
users01.dbf 100% 1465MB 45.8MB/s 00:32
lftp也做了一个测试，稍微快一点，大约30秒。1536180224 bytes transferred in 30 seconds (48.95M/s)
3.另外bbcp有一个好处，如果拷贝文件已经存在，提示文件存在直接退出。
加入-f 参数
--force | -f
forces the copy by erasing the target prior to copying the source file. By default, if the target already exists for the
source file, the copy fails. See also the –K option modifier.
如果在拷贝文件时，在远程机器使用ps -elf | grep bbcp 看。
# ps -elf | grep bbcp | grep -v grep
0 S oracle 7398 7397 1 81 0 - 12251 wait 17:04 ? 00:00:00 bbcp SNK
1 S oracle 7427 7398 68 76 0 - 22495 184466 17:04 ? 00:00:13 bbcp SNK
--会有2个进程。
看了一些文档从调整性能讲重要的两个参数（我自己认为）,我估计如果机器忙，带宽不足，也许会影响这个网络。
1.2.1 Window Size (-w)
The first and most important option is –w. This determines the TCP window size as well as the default I/O size. The
default of 128Kis usually good enough for LAN and larger values are likely to hurt performance.
1.2.2 Streams (-s)
The -- s options is the second most important tuning parameter. This specifies the number of parallel TCP streams. A
na?ve explanation would say that streams can make up for not having a large enough window. The idea is that if you can’
t get enough packets moving in a single window, then create multiple windows and run them simultaneously. This is only
partially true. While multiple streams do provide multiple windows, multiple windows also parallelize traffic with
independent time-outs, re-transmissions, and greatly improved I/O overlapping. This is a cumulative effect that
dramatically increases the overall bandwidth utilization. But, too much of a good thing can also be bad, as you will see
below.
