数据迁移上云优化之 - 带宽压缩加速-阿里云开发者社区

开发者社区> 阿里云数据库> 正文
登录阅读全文

数据迁移上云优化之 - 带宽压缩加速

简介:

现在云已经是一个趋势毋庸置疑,但是怎样在有限的时间内把数据弄上云端是个需要考虑的问题。
如果能增量复制,整个时间窗口可以不需要太关注,因为切换的时间窗口比较短。
但是如果是全量迁移,或者增量迁移前的全量迁移,则需要考虑时间窗口。
一般来说,从线下到线上,网络带宽都比较低,这个我在以前把香港机房的数据迁移到国内机房时深有同感,速度之慢难以忍受。
下面是一套加速方案:
1
2
3
4
5
6
下面是一个例子:
首先在本地服务器建立远程数据库服务器隧道 :

ssh -C -L 6666:127.0.0.1:1921 postgres@remote_ip  

在另外一个本地的Shell里面执行如下命令,备份远程数据库的数据到本地:

postgres@db-172-16-3-33-> date +%F%T;psql -h 127.0.0.1 -p 6666 -U postgres db_remote -c "copy schema.tbl_log to stdout" >./tbl_log.dmp;date +%F%T  
2011-06-2717:16:50  
2011-06-2717:17:08  

耗时18秒

不走隧道的话

postgres@db-172-16-3-33-> date +%F%T;psql -h remote_ip -p 1921 -U schema db_remote -c "copy schema.tbl_log to stdout" >./tbl_log2.dmp;date +%F%T  
2011-06-2717:20:06  
2011-06-2717:22:29  

耗时143秒

速度提升非常惊人.

参考:

man ssh
     -L [bind_address:]port:host:hostport  
             Specifies that the given port on the local (client) host is to be forwarded to the given host and port on  
             the remote side.  This works by allocating a socket to listen to port on the local side, optionally bound  
             to the specified bind_address.  Whenever a connection is made to this port, the connection is forwarded  
             over the secure channel, and a connection is made to host port hostport from the remote machine.  Port  
             forwardings can also be specified in the configuration file.  IPv6 addresses can be specified with an  
             alternative syntax: [bind_address/]port/host/hostport or by enclosing the address in square brackets.  
             Only the superuser can forward privileged ports.  By default, the local port is bound in accordance with  
             the GatewayPorts setting.  However, an explicit bind_address may be used to bind the connection to a spe-  
             cific address.  The bind_address of “localhost” indicates that the listening port be bound for local use  
             only, while an empty address or ‘*’ indicates that the port should be available from all interfaces.  
  
     -C      Requests compression of all data (including stdin, stdout, stderr, and data for forwarded X11 and TCP  
             connections).  The compression algorithm is the same used by gzip(1), and the “level” can be controlled  
             by the CompressionLevel option for protocol version 1.  Compression is desirable on modem lines and other  
             slow connections, but will only slow down things on fast networks.  The default value can be set on a  
             host-by-host basis in the configuration files; see the Compression option.  

版权声明:本文内容由阿里云实名注册用户自发贡献,版权归原作者所有,阿里云开发者社区不拥有其著作权,亦不承担相应法律责任。具体规则请查看《阿里云开发者社区用户服务协议》和《阿里云开发者社区知识产权保护指引》。如果您发现本社区中有涉嫌抄袭的内容,填写侵权投诉表单进行举报,一经查实,本社区将立刻删除涉嫌侵权内容。

分享:
阿里云数据库
使用钉钉扫一扫加入圈子
+ 订阅

帮用户承担一切数据库风险,给您何止是安心!

官方博客
链接