在db_rac1 alert*
Mon May 4 07:46:20 2015
ospid 24809: network interface with IP address 10.10.10.10 no longer running (check cable)
Mon May 4 07:47:20 2015
ospid 24809: network interface with IP address 10.10.10.10 is now running
在db_rac2 alert*这段时间的日志
Mon May 4 07:39:53 2015
Thread 2 advanced to log sequence 16878 (LGWR switch)
Current log# 10 seq# 16878 mem# 0: +RAC_DISK/dbrac/onlinelog/redo10_01.log
Current log# 10 seq# 16878 mem# 1: +RAC_DISK/dbrac/onlinelog/redo10_02.log
Mon May 4 07:46:57 2015
Thread 2 advanced to log sequence 16879 (LGWR switch)
Current log# 8 seq# 16879 mem# 0: +RAC_DISK/dbrac/onlinelog/redo08_01.log
Current log# 8 seq# 16879 mem# 1: +RAC_DISK/dbrac/onlinelog/redo08_02.log
Mon May 4 07:59:50 2015
Thread 2 advanced to log sequence 16880 (LGWR switch)
Current log# 9 seq# 16880 mem# 0: +RAC_DISK/dbrac/onlinelog/redo09_01.log
Current log# 9 seq# 16880 mem# 1: +RAC_DISK/dbrac/onlinelog/redo09_02.log
好像没有重启记录或者说没有重启过
在节点一中messages中发现问题
May 4 07:43:57 db_RAC1 kernel: igb: eth1 NIC Link is Down
May 4 07:44:00 db_RAC1 kernel: igb: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
May 4 07:46:12 db_RAC1 kernel: igb: eth1 NIC Link is Down
May 4 07:46:14 db_RAC1 kernel: igb: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
May 4 07:46:17 db_RAC1 kernel: igb: eth1 NIC Link is Down
May 4 07:46:20 db_RAC1 kernel: igb: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
May 4 07:47:41 db_RAC1 kernel: igb: eth1 NIC Link is Down
May 4 07:47:44 db_RAC1 kernel: igb: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
查看节点ip设置
ifconfig
eth1:10.10.10.10
eth1 NIC Link断掉
eth1 NIC Link链接
在节点二中查看messages
并没有发现MAY 4 7点左右的数据
查看节点一crs日志alertdbrac1
2015-04-26 23:51:29.519
[cssd(24361)]CRS-1611:node db_rac2 (2) at 75% heartbeat fatal, eviction in 14.144 seconds
2015-05-04 07:44:26.766
[cssd(24361)]CRS-1612:node db_rac2 (2) at 50% heartbeat fatal, eviction in 29.204 seconds
2015-05-04 07:44:41.769
[cssd(24361)]CRS-1611:node db_rac2 (2) at 75% heartbeat fatal, eviction in 14.204 seconds
2015-05-04 07:46:41.783
[cssd(24361)]CRS-1612:node db_rac2 (2) at 50% heartbeat fatal, eviction in 29.204 seconds
2015-05-04 07:46:56.793
[cssd(24361)]CRS-1611:node db_rac2 (2) at 75% heartbeat fatal, eviction in 14.194 seconds
2015-05-04 07:48:10.796
[cssd(24361)]CRS-1612:node db_rac2 (2) at 50% heartbeat fatal, eviction in 29.204 seconds
2015-05-04 07:48:25.808
[cssd(24361)]CRS-1611:node db_rac2 (2) at 75% heartbeat fatal, eviction in 14.194 seconds
查看节点二crs日志alertdbrac2
2015-05-04 07:44:26.214
[cssd(26097)]CRS-1612:node db_rac1 (1) at 50% heartbeat fatal, eviction in 29.048 seconds
2015-05-04 07:44:27.216
[cssd(26097)]CRS-1612:node db_rac1 (1) at 50% heartbeat fatal, eviction in 28.048 seconds
2015-05-04 07:44:41.217
[cssd(26097)]CRS-1611:node db_rac1 (1) at 75% heartbeat fatal, eviction in 14.048 seconds
2015-05-04 07:44:42.219
[cssd(26097)]CRS-1611:node db_rac1 (1) at 75% heartbeat fatal, eviction in 13.048 seconds
2015-05-04 07:46:41.235
[cssd(26097)]CRS-1612:node db_rac1 (1) at 50% heartbeat fatal, eviction in 29.048 seconds
2015-05-04 07:46:42.237
[cssd(26097)]CRS-1612:node db_rac1 (1) at 50% heartbeat fatal, eviction in 28.048 seconds
2015-05-04 07:46:56.236
[cssd(26097)]CRS-1611:node db_rac1 (1) at 75% heartbeat fatal, eviction in 14.048 seconds
2015-05-04 07:46:57.237
[cssd(26097)]CRS-1611:node db_rac1 (1) at 75% heartbeat fatal, eviction in 13.048 seconds
2015-05-04 07:48:10.248
[cssd(26097)]CRS-1612:node db_rac1 (1) at 50% heartbeat fatal, eviction in 29.048 seconds
2015-05-04 07:48:11.250
[cssd(26097)]CRS-1612:node db_rac1 (1) at 50% heartbeat fatal, eviction in 28.048 seconds
2015-05-04 07:48:25.252
[cssd(26097)]CRS-1611:node db_rac1 (1) at 75% heartbeat fatal, eviction in 14.048 seconds
2015-05-04 07:48:26.254
[cssd(26097)]CRS-1611:node db_rac1 (1) at 75% heartbeat fatal, eviction in 13.038 seconds
在节点一中ocssd.log
[ CSSD]2015-05-04 07:44:26.766 [1210603840] >TRACE: clssnmPollingThread: node db_rac2 (2) is impending reconfig, flag 1037, misstime 30540
[ CSSD]2015-05-04 07:44:26.766 [1210603840] >TRACE: clssnmPollingThread: diskTimeout set to (57000)ms impending reconfig status(1)
[ CSSD]2015-05-04 07:44:41.769 [1210603840] >WARNING: clssnmPollingThread: node db_rac2 (2) at 75 1.482197e-323artbeat fatal, eviction in 14.460 seconds
[ CSSD]2015-05-04 07:44:49.775 [1210603840] >TRACE: clssnmPollingThread: diskTimeout set to (200000)ms impending reconfig status(0)
[ CSSD]2015-05-04 07:46:41.783 [1210603840] >WARNING: clssnmPollingThread: node db_rac2 (2) at 50 1.482197e-323artbeat fatal, eviction in 29.460 seconds
[ CSSD]2015-05-04 07:46:41.783 [1210603840] >TRACE: clssnmPollingThread: node db_rac2 (2) is impending reconfig, flag 1037, misstime 30540
[ CSSD]2015-05-04 07:46:41.783 [1210603840] >TRACE: clssnmPollingThread: diskTimeout set to (57000)ms impending reconfig status(1)
[ CSSD]2015-05-04 07:46:56.793 [1210603840] >WARNING: clssnmPollingThread: node db_rac2 (2) at 75 1.482197e-323artbeat fatal, eviction in 14.450 seconds
[ CSSD]2015-05-04 07:47:04.791 [1210603840] >TRACE: clssnmPollingThread: diskTimeout set to (200000)ms impending reconfig status(0)
[ CSSD]2015-05-04 07:48:10.796 [1210603840] >WARNING: clssnmPollingThread: node db_rac2 (2) at 50 1.482197e-323artbeat fatal, eviction in 29.460 seconds
[ CSSD]2015-05-04 07:48:10.796 [1210603840] >TRACE: clssnmPollingThread: node db_rac2 (2) is impending reconfig, flag 1037, misstime 30540
[ CSSD]2015-05-04 07:48:10.796 [1210603840] >TRACE: clssnmPollingThread: diskTimeout set to (57000)ms impending reconfig status(1)
[ CSSD]2015-05-04 07:48:25.808 [1210603840] >WARNING: clssnmPollingThread: node db_rac2 (2) at 75 1.482197e-323artbeat fatal, eviction in 14.450 seconds
[ CSSD]2015-05-04 07:48:33.804 [1210603840] >TRACE: clssnmPollingThread: diskTimeout set to (200000)ms impending reconfig status(0)
在节点二中ocssd.log日志中
[ CSSD]2015-05-04 07:44:26.214 [1211832640] >WARNING: clssnmPollingThread: node db_rac1 (1) at 50 1.482197e-323artbeat fatal, eviction in 29.560 seconds
[ CSSD]2015-05-04 07:44:26.215 [1211832640] >TRACE: clssnmPollingThread: node db_rac1 (1) is impending reconfig, flag 1039, misstime 30440
[ CSSD]2015-05-04 07:44:26.215 [1211832640] >TRACE: clssnmPollingThread: diskTimeout set to (57000)ms impending reconfig status(1)
[ CSSD]2015-05-04 07:44:27.216 [1211832640] >WARNING: clssnmPollingThread: node db_rac1 (1) at 50 1.482197e-323artbeat fatal, eviction in 28.560 seconds
[ CSSD]2015-05-04 07:44:41.216 [1211832640] >WARNING: clssnmPollingThread: node db_rac1 (1) at 75 1.482197e-323artbeat fatal, eviction in 14.560 seconds
[ CSSD]2015-05-04 07:44:42.219 [1211832640] >WARNING: clssnmPollingThread: node db_rac1 (1) at 75 1.482197e-323artbeat fatal, eviction in 13.560 seconds
[ CSSD]2015-05-04 07:44:48.221 [1211832640] >TRACE: clssnmPollingThread: diskTimeout set to (200000)ms impending reconfig status(0)
[ CSSD]2015-05-04 07:46:41.235 [1211832640] >WARNING: clssnmPollingThread: node db_rac1 (1) at 50 1.482197e-323artbeat fatal, eviction in 29.560 seconds
[ CSSD]2015-05-04 07:46:41.235 [1211832640] >TRACE: clssnmPollingThread: node db_rac1 (1) is impending reconfig, flag 1039, misstime 30440
[ CSSD]2015-05-04 07:46:41.235 [1211832640] >TRACE: clssnmPollingThread: diskTimeout set to (57000)ms impending reconfig status(1)
[ CSSD]2015-05-04 07:46:42.236 [1211832640] >WARNING: clssnmPollingThread: node db_rac1 (1) at 50 1.482197e-323artbeat fatal, eviction in 28.560 seconds
[ CSSD]2015-05-04 07:46:56.235 [1211832640] >WARNING: clssnmPollingThread: node db_rac1 (1) at 75 1.482197e-323artbeat fatal, eviction in 14.560 seconds
[ CSSD]2015-05-04 07:46:57.237 [1211832640] >WARNING: clssnmPollingThread: node db_rac1 (1) at 75 1.482197e-323artbeat fatal, eviction in 13.560 seconds
[ CSSD]2015-05-04 07:47:03.240 [1211832640] >TRACE: clssnmPollingThread: diskTimeout set to (200000)ms impending reconfig status(0)
[ CSSD]2015-05-04 07:48:10.248 [1211832640] >WARNING: clssnmPollingThread: node db_rac1 (1) at 50 1.482197e-323artbeat fatal, eviction in 29.560 seconds
[ CSSD]2015-05-04 07:48:10.248 [1211832640] >TRACE: clssnmPollingThread: node db_rac1 (1) is impending reconfig, flag 1039, misstime 30440
[ CSSD]2015-05-04 07:48:10.248 [1211832640] >TRACE: clssnmPollingThread: diskTimeout set to (57000)ms impending reconfig status(1)
[ CSSD]2015-05-04 07:48:11.250 [1211832640] >WARNING: clssnmPollingThread: node db_rac1 (1) at 50 1.482197e-323artbeat fatal, eviction in 28.560 seconds
[ CSSD]2015-05-04 07:48:25.251 [1211832640] >WARNING: clssnmPollingThread: node db_rac1 (1) at 75 1.482197e-323artbeat fatal, eviction in 14.560 seconds
[ CSSD]2015-05-04 07:48:26.254 [1211832640] >WARNING: clssnmPollingThread: node db_rac1 (1) at 75 1.482197e-323artbeat fatal, eviction in 13.550 seconds
[ CSSD]2015-05-04 07:48:32.247 [1211832640] >TRACE: clssnmPollingThread: diskTimeout set to (200000)ms impending reconfig status(0)
如果只是ospid 24809:应该没有太大问题，但是要注意有可能几天后数据库可能会重启，注意查看是否时心跳线的问题。
