docker network case with customized bridge

[root@db-172-16-3-221 ~]# brctl 
Usage: brctl [commands]
commands:
        addbr           <bridge>                add bridge
        delbr           <bridge>                delete bridge
        addif           <bridge> <device>       add interface to bridge
        delif           <bridge> <device>       delete interface from bridge
        setageing       <bridge> <time>         set ageing time
        setbridgeprio   <bridge> <prio>         set bridge priority
        setfd           <bridge> <time>         set bridge forward delay
        sethello        <bridge> <time>         set hello time
        setmaxage       <bridge> <time>         set max message age
        sethashel       <bridge> <int>          set hash elasticity
        sethashmax      <bridge> <int>          set hash max
        setmclmc        <bridge> <int>          set multicast last member count
        setmcrouter     <bridge> <int>          set multicast router
        setmcsnoop      <bridge> <int>          set multicast snooping
        setmcsqc        <bridge> <int>          set multicast startup query count
        setmclmi        <bridge> <time>         set multicast last member interval
        setmcmi         <bridge> <time>         set multicast membership interval
        setmcqpi        <bridge> <time>         set multicast querier interval
        setmcqi         <bridge> <time>         set multicast query interval
        setmcqri        <bridge> <time>         set multicast query response interval
        setmcqri        <bridge> <time>         set multicast startup query interval
        setpathcost     <bridge> <port> <cost>  set path cost
        setportprio     <bridge> <port> <prio>  set port priority
        setportmcrouter <bridge> <port> <int>   set port multicast router
        show            [ <bridge> ]            show a list of bridges
        showmacs        <bridge>                show a list of mac addrs
        showstp         <bridge>                show bridge stp info
        stp             <bridge> {on|off}       turn stp on/off

# vi network.sh
#!/bin/bash

brctl addbr br0
ip link set dev br0 up
ip addr del 172.16.3.221/24 dev eth0
ip addr add 172.16.3.221/24 dev br0
brctl addif br0 eth0
ip route add default via 172.16.3.1 dev br0
ip link set dev docker0 down
brctl delbr docker0

# . ./network

# vi /etc/resolv.conf
nameserver 202.101.172.35

除此之外, 还需要修改docker服务文件.

# vi /etc/init.d/docker
例如 : 
        $exec -d -g /data01/docker &>> $logfile &
改成
        $exec -d -b="br0" -g /data01/docker &>> $logfile &

改完后, 最好把network服务关掉, 配置文件也重命名, 用脚本来启动网络.

将命令加入/etc/rc.local

# chkconfig network off
# chkconfig NetworkManager off
# chkconfig docker off

# vi /etc/rc.local
brctl addbr br0
ip link set dev eth0 up
ip link set dev br0 up
ip addr add 172.16.3.221/24 dev br0
brctl addif br0 eth0
ip route add default via 172.16.3.1 dev br0
service docker start

[root@db-172-16-3-221 ~]# cd /etc/sysconfig/network-scripts/
[root@db-172-16-3-221 network-scripts]# mv ifcfg-eth0 ifcfg-eth0.bak

重启docker 服务, container 将以172.16.3.0/24网段来分配IP.

接下来问题来了, 因为docker container的网络地址是自动分配的, 如果有多台宿主机的话, 使用以上架构, 非常有可能出现IP冲突的情况(如果docker没有判断的话), 建议宿主机使用不同网段, 但是管理就太麻烦了, 所以docker 默认是DNAT通讯的, 省去了网络管理的繁杂, 如果是openstack的话, container的网络是由openstack集中管理的, 而不是docker自动分配, 所以也不会出现冲突的问题.

那么需要使用netns来对docker container进行网络设置.

参考 http://blog.163.com/digoal@126/blog/static/16387704020141062106481/

比较靠谱的方法 : 

在主机上创建多个网桥, 一个由openvswitch管理, (目前docker还未直接支持openvswitch接口, 所以需要手工来管理)

将物理网卡添加到ovs管理的网桥, 并在物理交换机相连的接口使用trunck接口(以接收多个VLAN的包).

另一个网桥由bridge-utils管理, (docker目前仅支持bridge接口)

将物理网卡和bridge link加入ovs管理的网桥. 并分配不同的VLAN号. (隔离container和物理网段).

container和物理网段通信或对外通信交给路由器负责.

[参考]

1. man brctl

2. man ip