use iperf test network performance (tcp window & performance)

简介:
用iperf可以测试客户端和服务端之间的网络性能。
例如两台服务器是在同一个交换机下面的,可以用这种方法测试交换机和网卡的转发能力。当然测试结果还和tcp窗口配置,还有包的大小有一定关系(这种情况下往往不是二层的瓶颈)。
测试例子:
启动服务端
# iperf -f M -i 5 -m -N -s

启动客户端,单线程测试,TCP窗口大小1000字节。(非常小,所以性能不能达到网卡瓶颈1Gbit)
# iperf -f M -i 5 -m -w 1000 -N -t 100 -c $server_ip
WARNING: TCP window size set to 1000 bytes. A small window size
will give poor performance. See the Iperf documentation.
------------------------------------------------------------
Client connecting to $server_ip, TCP port 5001
TCP window size: 0.00 MByte (WARNING: requested 0.00 MByte)
------------------------------------------------------------
[  3] local  port 38549 connected with $server_ip port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0- 5.0 sec   184 MBytes  36.8 MBytes/sec
[  3]  5.0-10.0 sec   184 MBytes  36.8 MBytes/sec

sar -n DEV 1 1000
03:29:43 PM     IFACE   rxpck/s   txpck/s    rxkB/s    txkB/s   rxcmp/s   txcmp/s  rxmcst/s
03:29:44 PM        lo     17.35     17.35      6.07      6.07      0.00      0.00      0.00
03:29:44 PM      eth0   5421.43  27036.73    338.84  39982.72      0.00      0.00      0.00


启动客户端,单线程测试,TCP窗口大小16K。速度有所提升
# iperf -f M -i 5 -m -w 16K -N -t 100 -c $server_ip
------------------------------------------------------------
Client connecting to $server_ip, TCP port 5001
TCP window size: 0.03 MByte (WARNING: requested 0.02 MByte)
------------------------------------------------------------
[  3] local  port 38551 connected with $server_ip port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0- 5.0 sec   494 MBytes  98.7 MBytes/sec
[  3]  5.0-10.0 sec   493 MBytes  98.6 MBytes/sec
[  3] 10.0-15.0 sec   493 MBytes  98.7 MBytes/sec
[  3] 15.0-20.0 sec   493 MBytes  98.7 MBytes/sec
[  3] 20.0-25.0 sec   494 MBytes  98.7 MBytes/sec
[  3] 25.0-30.0 sec   493 MBytes  98.7 MBytes/sec
[  3] 30.0-35.0 sec   493 MBytes  98.5 MBytes/sec

03:31:22 PM     IFACE   rxpck/s   txpck/s    rxkB/s    txkB/s   rxcmp/s   txcmp/s  rxmcst/s
03:31:23 PM        lo     22.00     22.00      7.88      7.88      0.00      0.00      0.00
03:31:23 PM      eth0   7138.00  71027.00    446.21 105035.24      0.00      0.00      0.00


窗口调整为32K时,基本达到了1Gbit的极限。
# iperf -f M -i 5 -m -w 32K -N -t 100 -c $server_ip
------------------------------------------------------------
Client connecting to $server_ip, TCP port 5001
TCP window size: 0.06 MByte (WARNING: requested 0.03 MByte)
------------------------------------------------------------
[  3] local  port 38552 connected with $server_ip port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0- 5.0 sec   566 MBytes   113 MBytes/sec
[  3]  5.0-10.0 sec   566 MBytes   113 MBytes/sec
[  3] 10.0-15.0 sec   566 MBytes   113 MBytes/sec

03:35:04 PM     IFACE   rxpck/s   txpck/s    rxkB/s    txkB/s   rxcmp/s   txcmp/s  rxmcst/s
03:35:05 PM        lo     21.74     21.74      8.40      8.40      0.00      0.00      0.00
03:35:05 PM      eth0   8379.35  88493.48    523.71 130868.47      0.00      0.00      0.00


最后测试一下多线程,使用1500的窗口,同样可以把网卡极限跑满:
# iperf -f M -i 5 -m -w 1500 -N -t 100 -c $server_ip -P 10
WARNING: TCP window size set to 1500 bytes. A small window size
will give poor performance. See the Iperf documentation.
------------------------------------------------------------
Client connecting to $server_ip, TCP port 5001
TCP window size: 0.00 MByte (WARNING: requested 0.00 MByte)
------------------------------------------------------------
[ 12] local  port 38564 connected with $server_ip port 5001
[  3] local  port 38554 connected with $server_ip port 5001
[  4] local  port 38555 connected with $server_ip port 5001
[  5] local  port 38557 connected with $server_ip port 5001
[  7] local  port 38558 connected with $server_ip port 5001
[  6] local  port 38559 connected with $server_ip port 5001
[  8] local  port 38560 connected with $server_ip port 5001
[  9] local  port 38561 connected with $server_ip port 5001
[ 10] local  port 38562 connected with $server_ip port 5001
[ 11] local  port 38563 connected with $server_ip port 5001
[ ID] Interval       Transfer     Bandwidth
[ 12]  0.0- 5.0 sec  47.2 MBytes  9.45 MBytes/sec
[  5]  0.0- 5.0 sec  56.4 MBytes  11.3 MBytes/sec
[  6]  0.0- 5.0 sec  50.6 MBytes  10.1 MBytes/sec
[  8]  0.0- 5.0 sec  43.6 MBytes  8.72 MBytes/sec
[  4]  0.0- 5.0 sec  52.4 MBytes  10.5 MBytes/sec
[ 10]  0.0- 5.0 sec  53.5 MBytes  10.7 MBytes/sec
[ 11]  0.0- 5.0 sec  50.0 MBytes  10.0 MBytes/sec
[  7]  0.0- 5.0 sec  53.0 MBytes  10.6 MBytes/sec
[  3]  0.0- 5.0 sec  55.4 MBytes  11.1 MBytes/sec
[  9]  0.0- 5.0 sec  56.0 MBytes  11.2 MBytes/sec
[SUM]  0.0- 5.0 sec   518 MBytes   104 MBytes/sec


所以如果我们使用单线程传输,tcp窗口的大小很关键。
Linux可以通过设置几个内核参数来设置默认的发送和接收窗口的大小。
net.ipv4.tcp_mem = 5794944      7726592 11589888
net.ipv4.tcp_wmem = 4096        16384   4194304
net.ipv4.tcp_rmem = 4096        87380   4194304
net.ipv4.tcp_window_scaling = 1

man tcp
       tcp_rmem (since Linux 2.4)
              This  is  a  vector  of  3  integers: [min, default, max].  These parameters are used by TCP to regulate
              receive buffer sizes.  TCP dynamically adjusts the size of the receive buffer from the  defaults  listed
              below, in the range of these values, depending on memory available in the system.

              min       minimum  size  of the receive buffer used by each TCP socket.  The default value is the system
                        page size.  (On Linux 2.4, the default value is 4K, lowered to PAGE_SIZE bytes  in  low-memory
                        systems.)   This  value is used to ensure that in memory pressure mode, allocations below this
                        size will still succeed.  This is not used to bound the size of the  receive  buffer  declared
                        using SO_RCVBUF on a socket.

              default   the  default  size  of the receive buffer for a TCP socket.  This value overwrites the initial
                        default buffer size from the generic global net.core.rmem_default defined for  all  protocols.
                        The  default value is 87380 bytes.  (On Linux 2.4, this will be lowered to 43689 in low-memory
                        systems.)  If larger receive buffer sizes are desired, this  value  should  be  increased  (to
                        affect  all  sockets).   To  employ large TCP windows, the net.ipv4.tcp_window_scaling must be
                        enabled (default).

              max       the maximum size of the receive buffer used by each TCP socket.  This value does not  override
                        the  global  net.core.rmem_max.   This  is  not  used  to limit the size of the receive buffer
                        declared using SO_RCVBUF on a socket.  The default value is calculated using the formula

                            max(87380, min(4MB, tcp_mem[1]*PAGE_SIZE/128))

                        (On Linux 2.4, the default is 87380*2 bytes, lowered to 87380 in low-memory systems).

       tcp_window_scaling (Boolean; default: enabled; since Linux 2.2)
              Enable RFC 1323 TCP window scaling.  This feature allows the use of a large window (> 64K) on a TCP con-
              nection,  should  the  other end support it.  Normally, the 16 bit window length field in the TCP header
              limits the window size to less than 64K bytes.  If larger windows are desired, applications can increase
              the  size of their socket buffers and the window scaling option will be employed.  If tcp_window_scaling
              is disabled, TCP will not negotiate the use of window scaling  with  the  other  end  during  connection
              setup.

       tcp_wmem (since Linux 2.4)
              This  is a vector of 3 integers: [min, default, max].  These parameters are used by TCP to regulate send
              buffer sizes.  TCP dynamically adjusts the size of the send buffer from the default values listed below,
              in the range of these values, depending on memory available.

              min       Minimum size of the send buffer used by each TCP socket.  The default value is the system page
                        size.  (On Linux 2.4, the default value is 4K bytes.)  This value is used to  ensure  that  in
                        memory  pressure  mode,  allocations  below this size will still succeed.  This is not used to
                        bound the size of the send buffer declared using SO_SNDBUF on a socket.

              default   The default size of the send buffer for a TCP  socket.   This  value  overwrites  the  initial
                        default  buffer  size from the generic global net.core.wmem_default defined for all protocols.
                        The default value is 16K bytes.  If larger send buffer sizes are desired, this value should be
                        increased    (to    affect    all    sockets).     To    employ   large   TCP   windows,   the
                        /proc/sys/net/ipv4/tcp_window_scaling must be set to a non-zero value (default).

              max       The maximum size of the send buffer used by each TCP socket.  This value does not override the
                        value  in  /proc/sys/net/core/wmem_max.  This is not used to limit the size of the send buffer
                        declared using SO_SNDBUF on a socket.  The default value is calculated using the formula

                            max(65536, min(4MB, tcp_mem[1]*PAGE_SIZE/128))

                        (On Linux 2.4, the default value is 128K bytes, lowered 64K depending on low-memory  systems.)


[参考]
1. man iperf
IPERF(1)                         User Manuals                         IPERF(1)

NAME
       iperf - perform network throughput tests

SYNOPSIS
       iperf -s [ options ]

       iperf -c server [ options ]

       iperf -u -s [ options ]

       iperf -u -c server [ options ]

DESCRIPTION
       iperf  is a tool for performing network throughput measurements.  It can test either TCP or UDP throughput.  To
       perform an iperf test the user must establish both a server (to discard traffic)  and  a  client  (to  generate
       traffic).

GENERAL OPTIONS
       -f, --format
              [kmKM]   format to report: Kbits, Mbits, KBytes, MBytes

       -h, --help
              print a help synopsis

       -i, --interval n
              pause n seconds between periodic bandwidth reports

       -l, --len n[KM]
              set length read/write buffer to n (default 8 KB)

       -m, --print_mss
              print TCP maximum segment size (MTU - TCP/IP header)

       -o, --output <filename>
              output the report or error message to this specified file
       -p, --port n
              set server port to listen on/connect to to n (default 5001)

       -u, --udp
              use UDP rather than TCP

       -w, --window n[KM]
              TCP window size (socket buffer size)

       -B, --bind <host>
              bind to <host>, an interface or multicast address

       -C, --compatibility
              for use with older versions does not sent extra msgs

       -M, --mss n
              set TCP maximum segment size (MTU - 40 bytes)

       -N, --nodelay
              set TCP no delay, disabling Nagle’s Algorithm

       -v, --version
              print version information and quit

       -V, --IPv6Version
              Set the domain to IPv6

       -x, --reportexclude
              [CDMSV]   exclude C(connection) D(data) M(multicast) S(settings) V(server) reports

       -y, --reportstyle C|c
              if set to C or c report results as CSV (comma separated values)

SERVER SPECIFIC OPTIONS
       -s, --server
              run in server mode

       -U, --single_udp
              run in single threaded UDP mode

       -D, --daemon
              run the server as a daemon

CLIENT SPECIFIC OPTIONS
       -b, --bandwidth n[KM]
              set target bandwidth to n bits/sec (default 1 Mbit/sec).  This setting requires UDP (-u).

       -c, --client <host>
              run in client mode, connecting to <host>

       -d, --dualtest
              Do a bidirectional test simultaneously

       -n, --num n[KM]
              number of bytes to transmit (instead of -t)

       -r, --tradeoff
              Do a bidirectional test individually

       -t, --time n
              time in seconds to transmit for (default 10 secs)

       -F, --fileinput <name>
              input the data to be transmitted from a file

       -I, --stdin
              input the data to be transmitted from stdin

       -L, --listenport n
              port to recieve bidirectional tests back on

       -P, --parallel n
              number of parallel client threads to run

       -T, --ttl n
              time-to-live, for multicast (default 1)
       -Z, --linux-congestion <algo>
              set TCP congestion control algorithm (Linux only)

ENVIRONMENT
       TCP_WINDOW_SIZE
              Controls the size of TCP buffers.

DIAGNOSTICS
       This section needs to be filled in.

BUGS
       Exit statuses are inconsistent.  The threading implementation is rather heinous.

AUTHORS
       Iperf was originally written by Mark Gates and Alex Warshavsky.  Man page and maintence by Jon Dugan <jdugan at
       x1024 dot net>.  Other contributions from Ajay Tirumala, Jim Ferguson, Feng Qin, Kevin  Gibbs,  John  Estabrook
       <jestabro  at  ncsa.uiuc.edu>, Andrew Gallatin <gallatin at gmail.com>, Stephen Hemminger <shemminger at linux-
       foundation.org>

SEE ALSO
       http://iperf.sourceforge.net/

NLANR/DAST                        APRIL 2008                          IPERF(1)

目录
相关文章
下一篇
无影云桌面