用iperf可以测试客户端和服务端之间的网络性能。
例如两台服务器是在同一个交换机下面的,可以用这种方法测试交换机和网卡的转发能力。当然测试结果还和tcp窗口配置,还有包的大小有一定关系(这种情况下往往不是二层的瓶颈)。
测试例子:
启动服务端
启动客户端,单线程测试,TCP窗口大小1000字节。(非常小,所以性能不能达到网卡瓶颈1Gbit)
启动客户端,单线程测试,TCP窗口大小16K。速度有所提升
窗口调整为32K时,基本达到了1Gbit的极限。
最后测试一下多线程,使用1500的窗口,同样可以把网卡极限跑满:
所以如果我们使用单线程传输,tcp窗口的大小很关键。
# iperf -f M -i 5 -m -N -s
启动客户端,单线程测试,TCP窗口大小1000字节。(非常小,所以性能不能达到网卡瓶颈1Gbit)
# iperf -f M -i 5 -m -w 1000 -N -t 100 -c $server_ip
WARNING: TCP window size set to 1000 bytes. A small window size
will give poor performance. See the Iperf documentation.
------------------------------------------------------------
Client connecting to $server_ip, TCP port 5001
TCP window size: 0.00 MByte (WARNING: requested 0.00 MByte)
------------------------------------------------------------
[ 3] local port 38549 connected with $server_ip port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0- 5.0 sec 184 MBytes 36.8 MBytes/sec
[ 3] 5.0-10.0 sec 184 MBytes 36.8 MBytes/sec
sar -n DEV 1 1000
03:29:43 PM IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s
03:29:44 PM lo 17.35 17.35 6.07 6.07 0.00 0.00 0.00
03:29:44 PM eth0 5421.43 27036.73 338.84 39982.72 0.00 0.00 0.00
启动客户端,单线程测试,TCP窗口大小16K。速度有所提升
# iperf -f M -i 5 -m -w 16K -N -t 100 -c $server_ip
------------------------------------------------------------
Client connecting to $server_ip, TCP port 5001
TCP window size: 0.03 MByte (WARNING: requested 0.02 MByte)
------------------------------------------------------------
[ 3] local port 38551 connected with $server_ip port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0- 5.0 sec 494 MBytes 98.7 MBytes/sec
[ 3] 5.0-10.0 sec 493 MBytes 98.6 MBytes/sec
[ 3] 10.0-15.0 sec 493 MBytes 98.7 MBytes/sec
[ 3] 15.0-20.0 sec 493 MBytes 98.7 MBytes/sec
[ 3] 20.0-25.0 sec 494 MBytes 98.7 MBytes/sec
[ 3] 25.0-30.0 sec 493 MBytes 98.7 MBytes/sec
[ 3] 30.0-35.0 sec 493 MBytes 98.5 MBytes/sec
03:31:22 PM IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s
03:31:23 PM lo 22.00 22.00 7.88 7.88 0.00 0.00 0.00
03:31:23 PM eth0 7138.00 71027.00 446.21 105035.24 0.00 0.00 0.00
窗口调整为32K时,基本达到了1Gbit的极限。
# iperf -f M -i 5 -m -w 32K -N -t 100 -c $server_ip
------------------------------------------------------------
Client connecting to $server_ip, TCP port 5001
TCP window size: 0.06 MByte (WARNING: requested 0.03 MByte)
------------------------------------------------------------
[ 3] local port 38552 connected with $server_ip port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0- 5.0 sec 566 MBytes 113 MBytes/sec
[ 3] 5.0-10.0 sec 566 MBytes 113 MBytes/sec
[ 3] 10.0-15.0 sec 566 MBytes 113 MBytes/sec
03:35:04 PM IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s
03:35:05 PM lo 21.74 21.74 8.40 8.40 0.00 0.00 0.00
03:35:05 PM eth0 8379.35 88493.48 523.71 130868.47 0.00 0.00 0.00
最后测试一下多线程,使用1500的窗口,同样可以把网卡极限跑满:
# iperf -f M -i 5 -m -w 1500 -N -t 100 -c $server_ip -P 10
WARNING: TCP window size set to 1500 bytes. A small window size
will give poor performance. See the Iperf documentation.
------------------------------------------------------------
Client connecting to $server_ip, TCP port 5001
TCP window size: 0.00 MByte (WARNING: requested 0.00 MByte)
------------------------------------------------------------
[ 12] local port 38564 connected with $server_ip port 5001
[ 3] local port 38554 connected with $server_ip port 5001
[ 4] local port 38555 connected with $server_ip port 5001
[ 5] local port 38557 connected with $server_ip port 5001
[ 7] local port 38558 connected with $server_ip port 5001
[ 6] local port 38559 connected with $server_ip port 5001
[ 8] local port 38560 connected with $server_ip port 5001
[ 9] local port 38561 connected with $server_ip port 5001
[ 10] local port 38562 connected with $server_ip port 5001
[ 11] local port 38563 connected with $server_ip port 5001
[ ID] Interval Transfer Bandwidth
[ 12] 0.0- 5.0 sec 47.2 MBytes 9.45 MBytes/sec
[ 5] 0.0- 5.0 sec 56.4 MBytes 11.3 MBytes/sec
[ 6] 0.0- 5.0 sec 50.6 MBytes 10.1 MBytes/sec
[ 8] 0.0- 5.0 sec 43.6 MBytes 8.72 MBytes/sec
[ 4] 0.0- 5.0 sec 52.4 MBytes 10.5 MBytes/sec
[ 10] 0.0- 5.0 sec 53.5 MBytes 10.7 MBytes/sec
[ 11] 0.0- 5.0 sec 50.0 MBytes 10.0 MBytes/sec
[ 7] 0.0- 5.0 sec 53.0 MBytes 10.6 MBytes/sec
[ 3] 0.0- 5.0 sec 55.4 MBytes 11.1 MBytes/sec
[ 9] 0.0- 5.0 sec 56.0 MBytes 11.2 MBytes/sec
[SUM] 0.0- 5.0 sec 518 MBytes 104 MBytes/sec
所以如果我们使用单线程传输,tcp窗口的大小很关键。
Linux可以通过设置几个内核参数来设置默认的发送和接收窗口的大小。
man tcp
[参考]
net.ipv4.tcp_mem = 5794944 7726592 11589888
net.ipv4.tcp_wmem = 4096 16384 4194304
net.ipv4.tcp_rmem = 4096 87380 4194304
net.ipv4.tcp_window_scaling = 1
man tcp
tcp_rmem (since Linux 2.4)
This is a vector of 3 integers: [min, default, max]. These parameters are used by TCP to regulate
receive buffer sizes. TCP dynamically adjusts the size of the receive buffer from the defaults listed
below, in the range of these values, depending on memory available in the system.
min minimum size of the receive buffer used by each TCP socket. The default value is the system
page size. (On Linux 2.4, the default value is 4K, lowered to PAGE_SIZE bytes in low-memory
systems.) This value is used to ensure that in memory pressure mode, allocations below this
size will still succeed. This is not used to bound the size of the receive buffer declared
using SO_RCVBUF on a socket.
default the default size of the receive buffer for a TCP socket. This value overwrites the initial
default buffer size from the generic global net.core.rmem_default defined for all protocols.
The default value is 87380 bytes. (On Linux 2.4, this will be lowered to 43689 in low-memory
systems.) If larger receive buffer sizes are desired, this value should be increased (to
affect all sockets). To employ large TCP windows, the net.ipv4.tcp_window_scaling must be
enabled (default).
max the maximum size of the receive buffer used by each TCP socket. This value does not override
the global net.core.rmem_max. This is not used to limit the size of the receive buffer
declared using SO_RCVBUF on a socket. The default value is calculated using the formula
max(87380, min(4MB, tcp_mem[1]*PAGE_SIZE/128))
(On Linux 2.4, the default is 87380*2 bytes, lowered to 87380 in low-memory systems).
tcp_window_scaling (Boolean; default: enabled; since Linux 2.2)
Enable RFC 1323 TCP window scaling. This feature allows the use of a large window (> 64K) on a TCP con-
nection, should the other end support it. Normally, the 16 bit window length field in the TCP header
limits the window size to less than 64K bytes. If larger windows are desired, applications can increase
the size of their socket buffers and the window scaling option will be employed. If tcp_window_scaling
is disabled, TCP will not negotiate the use of window scaling with the other end during connection
setup.
tcp_wmem (since Linux 2.4)
This is a vector of 3 integers: [min, default, max]. These parameters are used by TCP to regulate send
buffer sizes. TCP dynamically adjusts the size of the send buffer from the default values listed below,
in the range of these values, depending on memory available.
min Minimum size of the send buffer used by each TCP socket. The default value is the system page
size. (On Linux 2.4, the default value is 4K bytes.) This value is used to ensure that in
memory pressure mode, allocations below this size will still succeed. This is not used to
bound the size of the send buffer declared using SO_SNDBUF on a socket.
default The default size of the send buffer for a TCP socket. This value overwrites the initial
default buffer size from the generic global net.core.wmem_default defined for all protocols.
The default value is 16K bytes. If larger send buffer sizes are desired, this value should be
increased (to affect all sockets). To employ large TCP windows, the
/proc/sys/net/ipv4/tcp_window_scaling must be set to a non-zero value (default).
max The maximum size of the send buffer used by each TCP socket. This value does not override the
value in /proc/sys/net/core/wmem_max. This is not used to limit the size of the send buffer
declared using SO_SNDBUF on a socket. The default value is calculated using the formula
max(65536, min(4MB, tcp_mem[1]*PAGE_SIZE/128))
(On Linux 2.4, the default value is 128K bytes, lowered 64K depending on low-memory systems.)
[参考]
1. man iperf
IPERF(1) User Manuals IPERF(1)
NAME
iperf - perform network throughput tests
SYNOPSIS
iperf -s [ options ]
iperf -c server [ options ]
iperf -u -s [ options ]
iperf -u -c server [ options ]
DESCRIPTION
iperf is a tool for performing network throughput measurements. It can test either TCP or UDP throughput. To
perform an iperf test the user must establish both a server (to discard traffic) and a client (to generate
traffic).
GENERAL OPTIONS
-f, --format
[kmKM] format to report: Kbits, Mbits, KBytes, MBytes
-h, --help
print a help synopsis
-i, --interval n
pause n seconds between periodic bandwidth reports
-l, --len n[KM]
set length read/write buffer to n (default 8 KB)
-m, --print_mss
print TCP maximum segment size (MTU - TCP/IP header)
-o, --output <filename>
output the report or error message to this specified file
-p, --port n
set server port to listen on/connect to to n (default 5001)
-u, --udp
use UDP rather than TCP
-w, --window n[KM]
TCP window size (socket buffer size)
-B, --bind <host>
bind to <host>, an interface or multicast address
-C, --compatibility
for use with older versions does not sent extra msgs
-M, --mss n
set TCP maximum segment size (MTU - 40 bytes)
-N, --nodelay
set TCP no delay, disabling Nagle’s Algorithm
-v, --version
print version information and quit
-V, --IPv6Version
Set the domain to IPv6
-x, --reportexclude
[CDMSV] exclude C(connection) D(data) M(multicast) S(settings) V(server) reports
-y, --reportstyle C|c
if set to C or c report results as CSV (comma separated values)
SERVER SPECIFIC OPTIONS
-s, --server
run in server mode
-U, --single_udp
run in single threaded UDP mode
-D, --daemon
run the server as a daemon
CLIENT SPECIFIC OPTIONS
-b, --bandwidth n[KM]
set target bandwidth to n bits/sec (default 1 Mbit/sec). This setting requires UDP (-u).
-c, --client <host>
run in client mode, connecting to <host>
-d, --dualtest
Do a bidirectional test simultaneously
-n, --num n[KM]
number of bytes to transmit (instead of -t)
-r, --tradeoff
Do a bidirectional test individually
-t, --time n
time in seconds to transmit for (default 10 secs)
-F, --fileinput <name>
input the data to be transmitted from a file
-I, --stdin
input the data to be transmitted from stdin
-L, --listenport n
port to recieve bidirectional tests back on
-P, --parallel n
number of parallel client threads to run
-T, --ttl n
time-to-live, for multicast (default 1)
-Z, --linux-congestion <algo>
set TCP congestion control algorithm (Linux only)
ENVIRONMENT
TCP_WINDOW_SIZE
Controls the size of TCP buffers.
DIAGNOSTICS
This section needs to be filled in.
BUGS
Exit statuses are inconsistent. The threading implementation is rather heinous.
AUTHORS
Iperf was originally written by Mark Gates and Alex Warshavsky. Man page and maintence by Jon Dugan <jdugan at
x1024 dot net>. Other contributions from Ajay Tirumala, Jim Ferguson, Feng Qin, Kevin Gibbs, John Estabrook
<jestabro at ncsa.uiuc.edu>, Andrew Gallatin <gallatin at gmail.com>, Stephen Hemminger <shemminger at linux-
foundation.org>
SEE ALSO
http://iperf.sourceforge.net/
NLANR/DAST APRIL 2008 IPERF(1)