客户的10.2.0.4 RAC for Hp-un环境碰到了这个错误。
错误信息为：
Wed Feb 29 19:42:05 2012
Errors in file /opt/app/oracle/admin/orcl/udump/orcl1_ora_11261.trc:
ORA-00600: internal error code, arguments: [qersqCloseRem-2], [Invalid Handle], [], [], [], [], [], []
ORA-02068: following severe error from WEBDB.COM TB
ORA-03113: end-of-file on communication channel
Wed Feb 29 19:42:05 2012
Errors in file /opt/app/oracle/admin/orcl/udump/orcl1_ora_32036.trc:
ORA-00600: internal error code, arguments: [qersqCloseRem-2], [Invalid Handle], [], [], [], [], [], []
ORA-02068: following severe error from WEBDB.COM
ORA-03113: end-of-file on communication channel
Wed Feb 29 19:42:05 2012
Errors in file /opt/app/oracle/admin/orcl/udump/orcl1_ora_5935.trc:
ORA-00600: internal error code, arguments: [qersqCloseRem-2], [Invalid Handle], [], [], [], [], [], []
ORA-02068: following severe error from WEBDB.COM
ORA-03113: end-of-file on communication channel
Wed Feb 29 19:42:05 2012
Errors in file /opt/app/oracle/admin/orcl/udump/orcl1_ora_5026.trc:
ORA-00600: internal error code, arguments: [qersqCloseRem-2], [Invalid Handle], [], [], [], [], [], []
ORA-02068: following severe error from WEBDB.COM
ORA-03113: end-of-file on communication channel
Wed Feb 29 19:42:05 2012
Errors in file /opt/app/oracle/admin/orcl/udump/orcl1_ora_7620.trc:
ORA-00600: internal error code, arguments: [qersqCloseRem-2], [Invalid Handle], [], [], [], [], [], []
ORA-02068: following severe error from WEBDB.COM
ORA-03113: end-of-file on communication channel
Wed Feb 29 19:42:08 2012
Trace dumping is performing id=[cdmp_20120229194207]
Wed Feb 29 19:42:17 2012
Trace dumping is performing id=[cdmp_20120229194217]
这个ORA-600[qersqCloseRem-2]错误非常罕见， TB MOS上居然没有任何记载。不过从错误信息进行进一步的分析，这个错误发生在远端数据库的访问异常。
检查进一步的详细信息：
*** 2012-02-29 19:42:05.564
ksedmp: internal or fatal error
ORA-00600: internal error code, arguments: [qersqCloseRem-2], [Invalid Handle], [], [], [], [], [], []
ORA-02068: following severe error from WEBDB.COM
ORA-03113: end-of-file on communication channel
Current SQL statement for this session:
SELECT ACCESS_LOG_SEQUENCE.NEXTVAL@WEBDB.COM FROM DUAL
----- PL/SQL Call Stack -----
object line object
handle number name
0x39b5c3720 5 ECOMMERCE.P_USER_AT
----- Call Stack Trace -----
calling call entry argument values in hex
location type point (? means dubious value)
-------------------- -------- -------------------- ----------------------------
ksedst()+31 call ksedst1() 000000000 ? 000000001 ?
7FBFFF4370 ? 7FBFFF43D0 ?
7FBFFF4310 ? 000000000 ?
ksedmp()+610 call ksedst() 000000000 ? 000000001 ?
7FBFFF4370 ? 7FBFFF43D0 ?
7FBFFF4310 ? 000000000 ?
ksfdmp()+21 call ksedmp() 000000003 ? 000000001 ?
7FBFFF4370 ? 7FBFFF43D0 ?
7FBFFF4310 ? 000000000 ?
.
.
.
0059DF200 ? 683F6E400000001 ?
main()+116 call opimai_real() 000000002 ? 7FBFFFF4E0 ?
000000004 ? 7FBFFFF478 ?
0059DF200 ? 683F6E400000001 ?
__libc_start_main() call main() 000000002 ? 7FBFFFF4E0 ?
+219 000000004 ? 7FBFFFF478 ?
0059DF200 ? 683F6E400000001 ?
_start()+42 call __libc_start_main() 0007139F8 ? 000000002 ?
7FBFFFF628 ? 0052B4BD0 ?
000000000 ? 000000002 ?
--------------------- Binary Stack Dump ---------------------
从详细TRACE分析，在问题发生时刻，正在通过数据库链读取远端序列的值。而此时出现的ORA-3113通信错误，多半与远端数据库状态异常有关。
检查远端数据库的告警日志，果然发现在问题出现时刻，数据库状态异常并最终导致了实例重启：
Wed Feb 29 19:39:29 2012
WARNING: inbound connection timed out (ORA-3136)
Wed Feb 29 19:39:30 2012
WARNING: inbound connection timed out (ORA-3136)
Wed Feb 29 19:40:01 2012
WARNING: inbound connection timed out (ORA-3136)
Wed Feb 29 19:40:01 2012
WARNING: inbound connection timed out (ORA-3136)
Wed Feb 29 19:40:01 2012
WARNING: inbound connection timed out (ORA-3136)
Wed Feb 29 19:40:01 2012
WARNING: inbound connection timed out (ORA-3136)
Wed Feb 29 19:40:01 2012
WARNING: inbound connection timed out (ORA-3136)
Wed Feb 29 19:40:01 2012
WARNING: inbound connection timed out (ORA-3136)
.
.
.
Wed Feb 29 19:43:28 2012
WARNING: inbound connection timed out (ORA-3136)
Wed Feb 29 19:43:28 2012
WARNING: inbound connection timed out (ORA-3136)
Wed Feb 29 19:43:28 2012
WARNING: inbound connection timed out (ORA-3136)
Wed Feb 29 19:43:28 2012
WARNING: inbound connection timed out (ORA-3136)
Wed Feb 29 19:43:29 2012
WARNING: inbound connection timed out (ORA-3136)
Wed Feb 29 19:43:29 2012
WARNING: inbound connection timed out (ORA-3136)
Wed Feb 29 19:43:29 2012
WARNING: inbound connection timed out (ORA-3136)
Wed Feb 29 19:43:29 2012
WARNING: inbound connection timed out (ORA-3136)
Wed Feb 29 19:43:30 2012
WARNING: inbound connection timed out (ORA-3136)
Wed Feb 29 19:45:26 2012
PMON failed to acquire latch, see PMON dump
Wed Feb 29 19:46:32 2012
WARNING: inbound connection timed out (ORA-3136)
Wed Feb 29 19:46:33 2012
WARNING: inbound connection timed out (ORA-3136)
Wed Feb 29 19:46:34 2012
PMON failed to acquire latch, see PMON dump
Wed Feb 29 19:46:40 2012
WARNING: inbound connection timed out (ORA-3136)
Wed Feb 29 19:46:43 2012
WARNING: inbound connection timed out (ORA-3136)
Wed Feb 29 19:46:44 2012
Errors in file /opt/app/oracle/admin/orcl/bdump/orcl1_asmb_14614.trc:
ORA-15064: communication failure with ASM instance
ORA-03113: end-of-file on communication channel
Wed Feb 29 19:46:44 2012
ASMB: terminating instance due to error 15064
Wed Feb 29 19:46:44 2012
System state dump is made for local instance
System State dumped to trace file /opt/app/oracle/admin/orcl/bdump/orcl1_diag_14555.trc
Wed Feb 29 19:46:47 2012
Shutting down instance (abort)
License high water mark = 1623
Wed Feb 29 19:46:49 2012
Instance terminated by ASMB, pid = 14614
Wed Feb 29 19:46:52 2012
Instance terminated by USER, pid = 3684
显然远端数据库状态异常是这个ORA-600错误的直接原因。