最近我这里docker单机平台正式上线使用，使用中有很多问题都一样解决，在给一个游戏项目做测试的时候，此项目由于8080端口对公网全部开放，并且安全策略没有做好（默认的tomcat模板没有删除），导致被人进行webshell，跑了很多的流量，为了解决此问题我针对tc与openvswitch本身的qos做了深入研究，最后选择openvswitch的qos作为容器的网络资源限制方法。
docker本身仅能对容器的cpu、内存做限制，而且必须是在容器运行前做，运行过程中未发现如何动态修改，并且不提供网络资源限制，所以只能使用其他软件做了。
我的docker网络没有使用默认bridge，使用none，然后绑定openvswitch的bridge，并使用pipework提供网络，所以我可以根据容器对于openvswitch的port来进行基于port的网络资源限制，好处是可以动态的修改，坏处是容器一重启还得重新做，但也可以通过其他方法来解决。
一、下面是我做测试的结果：
对于限速100m以下，其实硬盘的类型与读写速度没什么影响，但如何限速150m以上，或者无限速，那么硬盘肯定是ssd>sas>sata，所以建议进来使用sas的磁盘作为docker的挂载分区。
openvswitch默认官方文档提供的限速方法“rate limiting vm traffic using qos policing"，地址是http://openvswitch.org/support/config-cookbooks/qos-rate-limiting/，此限速仅能对上传做限速，下载没有办法，所以还得通过其他的qos来做限制。
二、下面是具体限速脚本
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
|
#!/bin/bash
#filename:modify_docker_container_network_limit.sh
#author:Deng Lei
#email:dl528888@gmail.com
op
=$1
container=$2
limit=$3
# Mbits/s
if
[ -z $1 ] || [ -z $2 ];
then
echo
"Usage: operation container_name limit(default:5m)"
echo
"Example1: I want limit 5m in the container:test"
echo
"The command is: bash `basename $0` limit test 5"
echo
"Example2: I want delete network limit in the container:test"
echo
"The command is: bash `basename $0` ulimit test"
exit
1
fi
if
[ -z $3 ];
then
limit=
'5m'
fi
if
[ `docker inspect --
format
"``.`State`.`Pid`"
$container &>>
/dev/null
&&
echo
0 ||
echo
1` -
eq
1 ];
then
echo
"no this container:$container"
exit
1
fi
ovs_prefix=
'veth1pl'
container_id=`docker inspect --
format
"``.`State`.`Pid`"
$container`
device_name=`
echo
${ovs_prefix}${container_id}`
if
[ $
op
==
'limit'
];
then
for
v
in
$device_name;
do
ovs-vsctl
set
interface $
v
ingress_policing_rate=$((limit*1000))
ovs-vsctl
set
interface $
v
ingress_policing_burst=$((limit*100))
ovs-vsctl
set
port $
v
qos=@newqos -- --
id
=@newqos create qos
type
=linux-htb queues=0=@q0 other-config:max-rate=$((limit*1000000)) -- --
id
=@q0 create queue other-config:min-rate=$((limit*1000000)) other-config:max-rate=$((limit*1000000)) &>>
/dev/null
&&
echo
'modify success!'
||
echo
'modify fail!'
done
elif
[ $
op
==
'ulimit'
];
then
for
v
in
$device_name;
do
ovs-vsctl
set
interface $
v
ingress_policing_rate=0
ovs-vsctl
set
interface $
v
ingress_policing_burst=0
ovs-vsctl
clear
Port $
v
qos &>>
/dev/null
&&
echo
'modify success!'
||
echo
'modify fail!'
done
fi
|
此脚本使用的话，限速可以直接针对下载与上传，并且限制是统一生效的，比如我限制一个容器带宽为5m，那么下载与上传的限速都是5m，单位是bit不是byte。
三、下面是使用方法：
|
1
2
3
4
5
6
|
[root@docker-test3 tmp]
# sh modify_docker_container_network_limit.sh
Usage: operation container_name limit(default:5m)
Example1: I want limit 5m
in
the container:
test
The
command
is:
bash
modify_docker_container_network_limit.sh limit
test
5
Example2: I want delete network limit
in
the container:
test
The
command
is:
bash
modify_docker_container_network_limit.sh
ulimit
test
|
四、下面是测试过程：
测试的方法是：
找另外一个主机172.16.1.126，然后dd生成个100m的文件/tmp/test_client.iso，在本机下载这个文件来测试下载速度，在本机dd生成100m的文件/tmp/test_server.iso把此文件上传到172.16.1.126里测试上传速度。
sata 7.5k
1、没有限制的情况
下载速度
|
1
2
3
4
5
6
7
8
|
14:12:18
# rsync -avz --progress 172.16.1.126:/tmp/test_client.iso/tmp/
root@172.16.1.126's password:
receiving incremental
file
list
test_client.iso
104857600 100% 56.82MB
/s
0:00:01 (xfer
#1, to-check=0/1)
sent 30 bytes received104892888 bytes 1226817.75 bytes
/sec
total size is 104857600 speedup is 1.00
|
上传速度
|
1
2
3
4
5
6
7
8
|
14:14:27
# rsync -avz --progress /tmp/test_server.iso172.16.1.126:/tmp/
root@172.16.1.126's password:
sending incremental
file
list
test_server.iso
104857600 100% 51.08MB
/s
0:00:01 (xfer
#1, to-check=0/1)
sent 104892883 bytes received31 bytes 29969404.00 bytes
/sec
total size is 104857600 speedup is 1.00
|
2、限速5m的
下载速度
|
1
2
3
4
5
6
7
8
|
14:15:27
# rsync -avz --progress 172.16.1.126:/tmp/test_client.iso/tmp/
root@172.16.1.126's password:
receiving incremental
file
list
test_client.iso
104857600 100% 580.46kB
/s
0:02:56 (xfer
#1, to-check=0/1)
sent 30 bytes received104892888 bytes 590946.02 bytes
/sec
total size is 104857600 speedup is 1.00
|
上传速度
|
1
2
3
4
5
6
7
8
|
14:22:10
# rsync -avz --progress /tmp/test_server.iso172.16.1.126:/tmp/
root@172.16.1.126's password:
sending incremental
file
list
test_server.iso
104857600 100% 616.19kB
/s
0:02:46 (xfer
#1, to-check=0/1)
sent 104892883 bytes received31 bytes 571623.51 bytes
/sec
total size is 104857600 speedup is 1.00
|
3、限速为10m
下载速度
|
1
2
3
4
5
6
7
8
9
|
14:28:55
# rsync -avz --progress 172.16.1.126:/tmp/test_client.iso/tmp/
root@172.16.1.126's password:
receiving incremental
file
list
test_client.iso
104857600 100% 1.13MB
/s
0:01:28 (xfer
#1, to-check=0/1)
sent 30 bytes received 104892888bytes 1109977.97 bytes
/sec
total size is 104857600 speedup is 1.00
root@fdc81b0d2508:
/tmp
|
上传速度
|
1
2
3
4
5
6
7
8
|
14:30:33
# rsync -avz --progress /tmp/test_server.iso172.16.1.126:/tmp/
root@172.16.1.126's password:
sending incremental
file
list
test_server.iso
104857600 100% 1.21MB
/s
0:01:22 (xfer
#1, to-check=0/1)
sent 104892883 bytes received31 bytes 1133977.45 bytes
/sec
total size is 104857600 speedup is 1.00
|
4、限速为20m
下载速度
|
1
2
3
4
5
6
7
8
|
14:32:57
# rsync -avz --progress 172.16.1.126:/tmp/test_client.iso/tmp/
root@172.16.1.126's password:
receiving incremental
file
list
test_client.iso
104857600 100% 2.27MB
/s
0:00:44 (xfer
#1, to-check=0/1)
sent 30 bytes received104892888 bytes 2305338.86 bytes
/sec
total size is 104857600 speedup is 1.00
|
上传速度
|
1
2
3
4
5
6
7
8
|
14:33:59
# rsync -avz --progress /tmp/test_server.iso172.16.1.126:/tmp/
root@172.16.1.126's password:
sending incremental
file
list
test_server.iso
104857600 100% 2.45MB
/s
0:00:40 (xfer
#1, to-check=0/1)
sent 104892883 bytes received31 bytes 2305338.77 bytes
/sec
total size is 104857600 speedup is 1.00
|
5、限速为50m
下载速度
|
1
2
3
4
5
6
7
8
|
14:35:20
# rsync -avz --progress 172.16.1.126:/tmp/test_client.iso/tmp/
root@172.16.1.126's password:
receiving incremental
file
list
test_client.iso
104857600 100% 5.67MB
/s
0:00:17 (xfer
#1, to-check=0/1)
sent 30 bytes received 104892888bytes 5379124.00 bytes
/sec
total size is 104857600 speedup is 1.00
|
上传速度
|
1
2
3
4
5
6
7
8
|
14:35:54
# rsync -avz --progress /tmp/test_server.iso172.16.1.126:/tmp/
root@172.16.1.126's password:
sending incremental
file
list
test_server.iso
104857600 100% 6.33MB
/s
0:00:15 (xfer
#1, to-check=0/1)
sent 104892883 bytes received31 bytes 5116727.51 bytes
/sec
total size is 104857600 speedup is 1.00
|
6、限速100m
下载速度
|
1
2
3
4
5
6
7
8
|
14:37:18
# rsync -avz --progress 172.16.1.126:/tmp/test_client.iso/tmp/
root@172.16.1.126's password:
receiving incremental
file
list
test_client.iso
104857600 100% 11.35MB
/s
0:00:08 (xfer
#1, to-check=0/1)
sent 30 bytes received104892888 bytes 8391433.44 bytes
/sec
total size is 104857600 speedup is 1.00
|
上传速度
|
1
2
3
4
5
6
7
8
|
14:37:39
# rsync -avz --progress /tmp/test_server.iso172.16.1.126:/tmp/
root@172.16.1.126's password:
sending incremental
file
list
test_server.iso
104857600 100% 13.19MB
/s
0:00:07 (xfer
#1, to-check=0/1)
sent 104892883 bytes received31 bytes 11041359.37 bytes
/sec
total size is 104857600 speedup is 1.00
|
7、限速150m
下载速度
|
1
2
3
4
5
6
7
8
|
14:38:39
# rsync -avz --progress 172.16.1.126:/tmp/test_client.iso/tmp/
root@172.16.1.126's password:
receiving incremental
file
list
test_client.iso
104857600 100% 11.35MB
/s
0:00:08 (xfer
#1, to-check=0/1)
sent 30 bytes received 104892888bytes 11041359.79 bytes
/sec
total size is 104857600 speedup is 1.00
|
上传速度
|
1
2
3
4
5
6
7
8
|
14:38:56
# rsync -avz --progress /tmp/test_server.iso172.16.1.126:/tmp/
root@172.16.1.126's password:
sending incremental
file
list
test_server.iso
104857600 100% 13.14MB
/s
0:00:07 (xfer
#1, to-check=0/1)
sent 104892883 bytes received31 bytes 11041359.37 bytes
/sec
total size is 104857600 speedup is 1.00
|
下面是使用sas 7.5k硬盘的测试结果
8、无限速的
下载速度
|
1
2
3
4
5
6
7
8
|
17:11:07
# rsync -avz --progress172.16.1.53:/tmp/test_client.iso /tmp/
root@172.16.1.53's password:
receiving incremental
file
list
test_client.iso
2147483648 100% 106.42MB
/s
0:00:19 (xfer
#1, to-check=0/1)
sent 30 bytes received 2148204633 bytes 95475762.80 bytes
/sec
total size is 2147483648 speedup is 1.00
|
上传速度
|
1
2
3
4
5
6
7
8
|
17:20:06
# rsync -avz --progress/tmp/test_server.iso 172.16.1.53:/tmp/
root@172.16.1.53's password:
sending incremental
file
list
test_server.iso
2147483648 100% 112.51MB
/s
0:00:18 (xfer
#1, to-check=0/1)
sent 2148204628 bytes received 31 bytes 95475762.62 bytes
/sec
total size is 2147483648 speedup is 1.00
|
9、5m限速
下载速度
|
1
2
3
4
5
6
7
8
|
17:30:30
# rsync -avz --progress172.16.1.53:/tmp/test_client.iso /tmp/
root@172.16.1.53's password:
receiving incremental
file
list
test_client.iso
104857600 100% 591.44kB
/s
0:02:53 (xfer
#1, to-check=0/1)
sent 30 bytes received 104892888 bytes 587635.39 bytes
/sec
total size is 104857600 speedup is 1.00
|
上传速度
|
1
2
3
4
5
6
7
8
|
17:38:57
# rsync -avz --progress/tmp/test_server.iso 172.16.1.53:/tmp/
root@172.16.1.53's password:
sending incremental
file
list
test_server.iso
104857600 100% 590.35kB
/s
0:02:53 (xfer
#1, to-check=0/1)
sent 104892883 bytes received 31 bytes 574755.69 bytes
/sec
total size is 104857600 speedup is 1.00
|
10、限速10m
下载速度
|
1
2
3
4
5
6
7
8
|
17:42:54
# rsync -avz --progress172.16.1.53:/tmp/test_client.iso /tmp/
root@172.16.1.53's password:
receiving incremental
file
list
test_client.iso
104857600 100% 1.15MB
/s
0:01:26 (xfer
#1, to-check=0/1)
sent 30 bytes received 104892888 bytes 1146370.69 bytes
/sec
total size is 104857600 speedup is 1.00
|
上传速度
|
1
2
3
4
5
6
7
8
|
17:44:31
# rsync -avz --progress/tmp/test_server.iso 172.16.1.53:/tmp/
root@172.16.1.53's password:
sending incremental
file
list
test_server.iso
104857600 100% 1.16MB
/s
0:01:26 (xfer
#1, to-check=0/1)
sent 104892883 bytes received 31 bytes 1146370.64 bytes
/sec
total size is 104857600 speedup is 1.00
|
11、限速20m
下载速度
|
1
2
3
4
5
6
7
8
|
17:47:02
# rsync -avz --progress172.16.1.53:/tmp/test_client.iso /tmp/
root@172.16.1.53's password:
receiving incremental
file
list
test_client.iso
104857600 100% 2.32MB
/s
0:00:43 (xfer
#1, to-check=0/1)
sent 30 bytes received 104892888 bytes 2162740.58 bytes
/sec
total size is 104857600 speedup is 1.00
|
上传速度
|
1
2
3
4
5
6
7
8
|
17:48:06
# rsync -avz --progress/tmp/test_server.iso 172.16.1.53:/tmp/
root@172.16.1.53's password:
sending incremental
file
list
test_server.iso
104857600 100% 2.38MB
/s
0:00:42 (xfer
#1, to-check=0/1)
sent 104892883 bytes received 31 bytes 2255761.59 bytes
/sec
total size is 104857600 speedup is 1.00
|
12、限速50m
下载速度
|
1
2
3
4
5
6
7
8
|
17:52:52
# rsync -avz --progress172.16.1.53:/tmp/test_client.iso /tmp/
root@172.16.1.53's password:
receiving incremental
file
list
test_client.iso
104857600 100% 5.84MB
/s
0:00:17 (xfer
#1, to-check=0/1)
sent 30 bytes received 104892888 bytes 5116727.71 bytes
/sec
total size is 104857600 speedup is 1.00
|
上传速度
|
1
2
3
4
5
6
7
8
|
17:53:15
# rsync -avz --progress/tmp/test_server.iso 172.16.1.53:/tmp/
root@172.16.1.53's password:
sending incremental
file
list
test_server.iso
104857600 100% 6.10MB
/s
0:00:16 (xfer
#1, to-check=0/1)
sent 104892883 bytes received 31 bytes 5993880.80 bytes
/sec
total size is 104857600 speedup is 1.00
|
13、限速为100m
下载速度
|
1
2
3
4
5
6
7
8
|
17:55:16
# rsync -avz --progress172.16.1.53:/tmp/test_client.iso /tmp/
root@172.16.1.53's password:
receiving incremental
file
list
test_client.iso
104857600 100% 11.75MB
/s
0:00:08 (xfer
#1, to-check=0/1)
sent 30 bytes received 104892888 bytes 9989801.71 bytes
/sec
total size is 104857600 speedup is 1.00
|
上传速度
|
1
2
3
4
5
6
7
8
|
17:55:39
# rsync -avz --progress/tmp/test_server.iso 172.16.1.53:/tmp/
root@172.16.1.53's password:
sending incremental
file
list
test_server.iso
104857600 100% 12.72MB
/s
0:00:07 (xfer
#1, to-check=0/1)
sent 104892883 bytes received 31 bytes 11041359.37 bytes
/sec
total size is 104857600 speedup is 1.00
|
14、限速150m
下载速度
|
1
2
3
4
5
6
7
8
|
17:56:58
# rsync -avz --progress 172.16.1.53:/tmp/test_client.iso/tmp/
root@172.16.1.53's password:
receiving incremental
file
list
test_client.iso
104857600 100% 17.69MB
/s
0:00:05 (xfer
#1, to-check=0/1)
sent 30 bytes received 104892888 bytes 13985722.40 bytes
/sec
total size is 104857600 speedup is 1.00
|
上传速度
|
1
2
3
4
5
6
7
8
|
17:57:18
# rsync -avz --progress/tmp/test_server.iso 172.16.1.53:/tmp/
root@172.16.1.53's password:
sending incremental
file
list
test_server.iso
104857600 100% 20.44MB
/s
0:00:04 (xfer
#1, to-check=0/1)
sent 104892883 bytes received 31 bytes 16137371.38 bytes
/sec
total size is 104857600 speedup is 1.00
|
如果大家有问题可以留言给我，我会及时回复。
本文转自 reinxu 51CTO博客，原文链接：http://blog.51cto.com/dl528888/1641569，如需转载请自行联系原作者
