【原创】线上环境 SYN flooding 问题排查

...
Apr 20 18:21:48 localhost kernel: possible SYN flooding on port 80. Sending cookies.
Apr 20 18:24:37 localhost kernel: possible SYN flooding on port 80. Sending cookies.
Apr 20 18:25:50 localhost kernel: possible SYN flooding on port 80. Sending cookies.
Apr 20 18:27:02 localhost kernel: possible SYN flooding on port 80. Sending cookies.
Apr 20 18:29:01 localhost kernel: possible SYN flooding on port 80. Sending cookies.
Apr 20 18:30:14 localhost kernel: possible SYN flooding on port 80. Sending cookies.
Apr 20 18:31:28 localhost kernel: possible SYN flooding on port 80. Sending cookies.
Apr 20 18:32:44 localhost kernel: possible SYN flooding on port 80. Sending cookies.
Apr 20 18:35:33 localhost kernel: possible SYN flooding on port 80. Sending cookies.
Apr 20 18:37:06 localhost kernel: possible SYN flooding on port 80. Sending cookies.
Apr 20 18:37:52 localhost ntpd_intres[1732]: host name not found: 0.centos.pool.ntp.org
Apr 20 18:38:12 localhost ntpd_intres[1732]: host name not found: 1.centos.pool.ntp.org
Apr 20 18:38:20 localhost kernel: possible SYN flooding on port 80. Sending cookies.
Apr 20 18:38:32 localhost ntpd_intres[1732]: host name not found: 2.centos.pool.ntp.org
Apr 20 18:38:52 localhost ntpd_intres[1732]: host name not found: 3.centos.pool.ntp.org
Apr 20 18:39:29 localhost kernel: possible SYN flooding on port 80. Sending cookies.
Apr 20 18:40:43 localhost kernel: possible SYN flooding on port 80. Sending cookies.
...

• 大量的 HTTP 请求均来自属于公司终端设备的地址；
• 从抓包中可以看到，问题终端大约每秒会发送 1000 个 TCP SYN 包（实际发现存在不止一个问题终端）；
• nginx 未进行 backlog 配置，即使用的是 nginx 默认值 NGX_LISTEN_BACKLOG ，即 511；
• 后台检测脚本基于 nginx 的 /status 页面进行 HTTP 状态检测，会得到 500 状态码；
• 后台检测脚本在 3 次检测异常后，主动触发虚地址切换，在切换过程中通过虚拟 IP 地址进行访问的服务无法正常使用（这个是正常情况）；

• 问题终端会在一定场景下不断发起 TCP SYN 请求，导致看似发生了 DOS 攻击行为；
• 系统内核参数以及 nginx 配置参数均未进行过优化调整，存在性能瓶颈；

• net.core.somaxconn = 128 
• net.ipv4.tcp_max_syn_backlog = 2048
• net.ipv4.tcp_syncookies = 1

平台开启了 SYN 攻击检测，故平台会认为 80 端口受到 SYN flood 攻击，而在这种情况下连正常的 keepalive 心跳检测也会受到影响。昨天的问题，是因为 SYN 攻击检测导致的，不是 TCP SYN 缓存队列占满的原因。

• 针对 keepalive 来说，正如其他内核参数没有调整过一样，net.ipv4.tcp_keepalive_time 也一样没有进行过调整，而且一般也不建议做调整，可以查看默认值为 7200 秒；而这个时间长度在面临类似 SYN flood 攻击行为时，肯定已经可以不考虑 keepalive 问题了；
• 针对 TCP SYN 队列是否占满的问题，从相关资料或源码中可以看到“只有在 SYN 队列已满的情况下才会触发 SYN cookies 机制”，所以上面的说法其实是错误的；

• nginx 在转发终端请求时，存在转发到错误地址的情况（这个问题的原因未知），进而导致终端获取 token 值失效。
• 同时，根据终端逻辑，其会不断重新建链进行 token 获取，所以在抓包中才看到终端一直在持续发送大量的 tcp 包（其实终端的重连逻辑中还存在更大的问题，此处就呵呵吧）。

...
       #include <sys/types.h>          /* See NOTES */
       #include <sys/socket.h>

       int listen(int sockfd, int backlog);

...
       The behavior of the backlog argument on TCP sockets changed with Linux 2.2. 
  Now it specifies the queue length for completely established sockets waiting to be accepted, instead of the number of incomplete connection requests. 
  The maximum length of the queue for incomplete sockets can be set using /proc/sys/net/ipv4/tcp_max_syn_backlog. 
  When syncookies are enabled there is no logical maximum length and this setting is ignored. 
  See tcp(7) for more information.

       If the backlog argument is greater than the value in /proc/sys/net/core/somaxconn, then it is silently truncated to that value; 
  the default value in this file is 128. In kernels before 2.4.25, this limit was a hard coded value, SOMAXCONN, with the value 128.

tcp_abort_on_overflow (Boolean; default: disabled; since Linux 2.4)
   Enable resetting connections if the listening service is too slow and unable to keep up and accept them. 
   It means that if overflow occurred due to a burst, the connection will recover. 
   Enable this option only if you are really sure that the listening daemon cannot be tuned to accept connections faster. 
   Enabling this option can harm the clients of your server.
...
tcp_max_syn_backlog (integer; default: see below; since Linux 2.2)
   The maximum number of queued connection requests which have still not received an acknowledgement from the connecting client. 
   If this number is exceeded, the kernel will begin dropping requests. 
   The default value of 256 is increased to 1024 when the memory present in the system is adequate or greater (>= 128Mb), 
   and reduced to 128 for those systems with very low memory (<= 32Mb). 
   It is recommended that if this needs to be increased above 1024, TCP_SYNQ_HSIZE in include/net/tcp.h be modified to keep TCP_SYNQ_HSIZE*16<=tcp_max_syn_backlog, and the kernel be recompiled.
...
tcp_synack_retries (integer; default: 5; since Linux 2.2)
   The maximum number of times a SYN/ACK segment for a passive TCP connection will be retransmitted. 
   This number should not be higher than 255.

tcp_syncookies (Boolean; since Linux 2.2)
   Enable TCP syncookies. 
   The kernel must be compiled with CONFIG_SYN_COOKIES. 
   Send out syncookies when the syn backlog queue of a socket overflows. 
   The syncookies feature attempts to protect a socket from a SYN flood attack.
   This should be used as a last resort, if at all. 
   This is a violation of the TCP protocol, and conflicts with other areas of TCP such as TCP extensions. 
   It can cause problems for clients and relays. 
   It is not recommended as a tuning mechanism for heavily loaded servers to help with overloaded or misconfigured conditions. 
   For recommended alternatives see tcp_max_syn_backlog, tcp_synack_retries, and tcp_abort_on_overflow.
...