故障现象:
客户反映园区仓库访问NC系统速度很慢,从园区ping NC服务器地址时有严重的丢包现象。如下图所示:
处理过程:
登录用户两个园区的核心发现,两个园区的portchannel连接正常,但是其中一个园区portchannel成员端口下存在严重的CRC校验错误,如下图所示:
此时,关闭端口ten1/5/1,数据转发从另一个端口走,关闭后再次PING NC服务器,数据包正常
为进一步确认该故障,再次对ten1/5/1进行开启操作,但打开后发现,问题立即出现
SW#ping 192.168.x.x repeat 1000 source 192.168.x.x
Type escape sequence to abort.
Sending 1000, 100-byte ICMP Echos to 192.168.x.x, timeout is 2 seconds:
Packet sent with a source address of 192.168.x.x
!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!.!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!.!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!.!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!
!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!.!!!!!!!!!!!!!!!!!..!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!
!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!
Success rate is 97 percent (978/1000), round-trip min/avg/max = 1/2/20 ms
SW#sh int ten1/5/1
TenGigabitEthernet1/5/1 is up, line protocol is up (connected)
Hardware is Ten Gigabit Ethernet Port, address is 78da.6e07.7b7a (bia 78da.6e07.7b7a)
Description: connection to laishanchangqu
MTU 1500 bytes, BW 10000000 Kbit/sec, DLY 10 usec,
reliability 245/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 10Gb/s, link type is auto, media type is 10GBase-ER
input flow-control is on, output flow-control is on
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:03, output never, output hang never
Last clearing of "show interface" counters never
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 3209000 bits/sec, 358 packets/sec
5 minute output rate 926000 bits/sec, 685 packets/sec
4935914010 packets input, 5278619447870 bytes, 0 no buffer
Received 87068 broadcasts (87050 multicasts)
0 runts, 0 giants, 0 throttles
277020288 input errors, 277020288 CRC, 0 frame, 0 overrun, 0 ignored
0 input packets with dribble condition detected
9064745454 packets output, 3287137727436 bytes, 0 underruns
0 output errors, 0 collisions, 4 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier
0 output buffer failures, 0 output buffers swapped out
SW#conf t
Enter configuration commands, one per line. End with CNTL/Z.
SW(config)#int ten1/5/1
SW(config-if)#shut
SW(config-if)#do sh run int ten1/5/1
Building configuration...
Current configuration : 171 bytes
!
interface TenGigabitEthernet1/5/1
switchport access vlan 133
switchport mode access
shutdown
channel-group 10 mode on
end
SW(config-if)#exit
SW(config)#exit
SW#wr
Building configuration...
Compressed configuration from 21267 bytes to 7568 bytes[OK]
SW#ping 192.168.x.x repeat 1000 source 192.168.x.x
Type escape sequence to abort.
Sending 1000, 100-byte ICMP Echos to 192.168.x.x timeout is 2 seconds:
Packet sent with a source address of 192.168.x.x
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!
Success rate is 100 percent (1000/1000), round-trip min/avg/max = 1/2/20 ms
经最终确认,NC访问问题主要是由于该端口的CRC校验错误引起。
解决建议:
1、检查两厂区之间的链路
2、检查模块
3、检查尾纤
*********************
最终:
第二天测试两个园区之间的链路,发现一根光纤的衰减在-21左右,而该接口上的万兆模块可接受的衰减最小值是-15.8,后来运营商对线路进行调优检测,最终恢复正常,测试,网络访问正常
*********************