客户的一套11.2.0.1 RAC数据库部署了 Enterprise Manager 11g以方便管理,但系统运行一段时间后EM就变得无法正常使用了,页面显示"the database status is currently unavailable.......",但实际上数据库和监听都处于正常可用的状态。
作为乙方来说对Enterprise Manager并不太熟悉,虽然EM是一个十分强大的图形化监控工具,但是我们很少有机会在生产环境中用到它,这一点可能和甲方的DBA不同。 针对该EM不可用的问题首先想到了尝试重启dbconsole,以验证该问题是否可重现。
emctl stop dbconsole emctl start dbconsole重启dbconsole后问题依旧,查MOS发现引发该EM不可用问题的可能性有多种,通过$ORACLE_HOME/_/sysman/log/emoms.log的EM相关日志可以进一步定位该问题:
tail -500 emoms.log at com.evermind.server.http.HttpRequestHandler.doProcessRequest(HttpRequestHandler.java:871) at com.evermind.server.http.HttpRequestHandler.processRequest(HttpRequestHandler.java:453) at com.evermind.server.http.HttpRequestHandler.serveOneRequest(HttpRequestHandler.java:221) at com.evermind.server.http.HttpRequestHandler.run(HttpRequestHandler.java:122) at com.evermind.server.http.HttpRequestHandler.run(HttpRequestHandler.java:111) at oracle.oc4j.network.ServerSocketReadHandler$SafeRunnable.run(ServerSocketReadHandler.java:260) at oracle.oc4j.network.ServerSocketAcceptHandler.procClientSocket(ServerSocketAcceptHandler.java:234) at oracle.oc4j.network.ServerSocketAcceptHandler.access$700(ServerSocketAcceptHandler.java:29) at oracle.oc4j.network.ServerSocketAcceptHandler$AcceptHandlerHorse.run(ServerSocketAcceptHandler.java:879) at com.evermind.util.ReleasableResourcePooledExecutor$MyWorker.run(ReleasableResourcePooledExecutor.java:303) at java.lang.Thread.run(Thread.java:595) 2011-08-02 14:39:46,131 [EMUI_14_39_46_/console/aboutApplication] ERROR em.console doGet.363 - java.lang.IllegalStateException: Response has already been committed, be sure not to write to the OutputStream or to trigger a commit due to any other action before calling this method. java.lang.IllegalStateException: Response has already been committed, be sure not to write to the OutputStream or to trigger a commit due to any other action before calling this method. at com.evermind.server.http.EvermindHttpServletResponse.sendRedirect(EvermindHttpServletResponse.java:1371) at oracle.sysman.emSDK.svlt.EMServlet.myDoGet(EMServlet.java:812) at oracle.sysman.emSDK.svlt.EMServlet.doGet(EMServlet.java:343) at oracle.sysman.eml.app.Console.doGet(Console.java:157) at javax.servlet.http.HttpServlet.service(HttpServlet.java:743) at javax.servlet.http.HttpServlet.service(HttpServlet.java:856) at com.evermind.server.http.ResourceFilterChain.doFilter(ResourceFilterChain.java:64) at oracle.sysman.eml.app.EMRepLoginFilter.doFilter(EMRepLoginFilter.java:107) at com.evermind.server.http.EvermindFilterChain.doFilter(EvermindFilterChain.java:15) at oracle.sysman.db.adm.inst.HandleRepDownFilter.doFilter(HandleRepDownFilter.java:146) at com.evermind.server.http.EvermindFilterChain.doFilter(EvermindFilterChain.java:17) at oracle.sysman.eml.app.BrowserVersionFilter.doFilter(BrowserVersionFilter.java:122) at com.evermind.server.http.EvermindFilterChain.doFilter(EvermindFilterChain.java:17) at oracle.sysman.emSDK.svlt.EMRedirectFilter.doFilter(EMRedirectFilter.java:102) at com.evermind.server.http.EvermindFilterChain.doFilter(EvermindFilterChain.java:17) at oracle.sysman.eml.app.ContextInitFilter.doFilter(ContextInitFilter.java:327) at com.evermind.server.http.ServletRequestDispatcher.invoke(ServletRequestDispatcher.java:621) at com.evermind.server.http.ServletRequestDispatcher.forwardInternal(ServletRequestDispatcher.java:370) at com.evermind.server.http.HttpRequestHandler.doProcessRequest(HttpRequestHandler.java:871) at com.evermind.server.http.HttpRequestHandler.processRequest(HttpRequestHandler.java:453) at com.evermind.server.http.HttpRequestHandler.serveOneRequest(HttpRequestHandler.java:221) at com.evermind.server.http.HttpRequestHandler.run(HttpRequestHandler.java:122) at com.evermind.server.http.HttpRequestHandler.run(HttpRequestHandler.java:111) at oracle.oc4j.network.ServerSocketReadHandler$SafeRunnable.run(ServerSocketReadHandler.java:260) at oracle.oc4j.network.ServerSocketAcceptHandler.procClientSocket(ServerSocketAcceptHandler.java:234) at oracle.oc4j.network.ServerSocketAcceptHandler.access$700(ServerSocketAcceptHandler.java:29) at oracle.oc4j.network.ServerSocketAcceptHandler$AcceptHandlerHorse.run(ServerSocketAcceptHandler.java:879) at com.evermind.util.ReleasableResourcePooledExecutor$MyWorker.run(ReleasableResourcePooledExecutor.java:303) at java.lang.Thread.run(Thread.java:595) 2011-08-02 14:39:46,150 [HTTPThreadGroup-4] ERROR conn.ConnectionService verifyRepositoryEx.891 - Invalid Connection Pool. ERROR = ORA-28000: the account is locked 2011-08-02 14:39:46,157 [HTTPThreadGroup-6] ERROR eml.OMSHandshake processFailure.806 - OMSHandshake failed.(AGENT URL = https://maclean01.oracledatabase12g.com:3938/emd/main)(ERROR = INTERNAL_ERROR)(CAUSE =java.sql.SQLException: ORA-28000: the account is locked ) 2011-08-02 14:39:46,182 [HTTPThreadGroup-4] ERROR eml.OMSHandshake processFailure.806 - OMSHandshake failed.(AGENT URL = https://maclean04.oracledatabase12g.com:3938/emd/main)(ERROR = INTERNAL_ERROR)(CAUSE =java.sql.SQLException: ORA-28000: the account is locked ) 2011-08-02 14:39:51,352 [HTTPThreadGroup-4] ERROR conn.ConnectionService verifyRepositoryEx.891 - Invalid Connection Pool. ERROR = ORA-28000: the account is locked 2011-08-02 14:39:51,394 [HTTPThreadGroup-4] ERROR eml.OMSHandshake processFailure.806 - OMSHandshake failed.(AGENT URL = https://maclean03.oracledatabase12g.com:3938/emd/main)(ERROR = INTERNAL_ERROR)(CAUSE =java.sql.SQLException: ORA-28000: the account is locked ) 2011-08-02 14:39:51,509 [HTTPThreadGroup-4] ERROR conn.ConnectionService verifyRepositoryEx.891 - Invalid Connection Pool. ERROR = ORA-28000: the account is locked 2011-08-02 14:39:51,537 [HTTPThreadGroup-4] ERROR eml.OMSHandshake processFailure.806 - OMSHandshake failed.(AGENT URL = https://maclean03.oracledatabase12g.com:3938/emd/main)(ERROR = INTERNAL_ERROR)(CAUSE =java.sql.SQLException: ORA-28000: the account is locked ) 2011-08-02 14:39:56,757 [HTTPThreadGroup-4] ERROR conn.ConnectionService verifyRepositoryEx.891 - Invalid Connection Pool. ERROR = ORA-28000: the account is locked 2011-08-02 14:40:03,574 [HTTPThreadGroup-4] ERROR conn.ConnectionService verifyRepositoryEx.891 - Invalid Connection Pool. ERROR = ORA-28000: the account is locked 2011-08-02 14:40:03,603 [HTTPThreadGroup-4] ERROR eml.OMSHandshake processFailure.806 - OMSHandshake failed.(AGENT URL = https://maclean02.oracledatabase12g.com:3938/emd/main)(ERROR = INTERNAL_ERROR)(CAUSE =java.sql.SQLException: ORA-28000: the account is locked ) 2011-08-02 14:40:03,729 [HTTPThreadGroup-4] ERROR conn.ConnectionService verifyRepositoryEx.891 - Invalid Connection Pool. ERROR = ORA-28000: the account is locked 2011-08-02 14:40:03,800 [HTTPThreadGroup-4] ERROR eml.OMSHandshake processFailure.806 - OMSHandshake failed.(AGENT URL = https://maclean02.oracledatabase12g.com:3938/emd/main)(ERROR = INTERNAL_ERROR)(CAUSE =java.sql.SQLException: ORA-28000: the account is locked ) 2011-08-02 14:40:09,328 [HTTPThreadGroup-4] ERROR conn.ConnectionService verifyRepositoryEx.891 - Invalid Connection Pool. ERROR = ORA-28000: the account is locked 2011-08-02 14:40:09,361 [HTTPThreadGroup-4] ERROR eml.OMSHandshake processFailure.806 - OMSHandshake failed.(AGENT URL = https://maclean03.oracledatabase12g.com:3938/emd/main)(ERROR = INTERNAL_ERROR)(CAUSE =java.sql.SQLException: ORA-28000: the account is locked ) 2011-08-02 14:40:11,946 [HTTPThreadGroup-4] ERROR conn.ConnectionService verifyRepositoryEx.891 - Invalid Connection Pool. ERROR = ORA-28000: the account is locked 2011-08-02 14:40:16,285 [HTTPThreadGroup-4] ERROR conn.ConnectionService verifyRepositoryEx.891 - Invalid Connection Pool. ERROR = ORA-28000: the account is locked 2011-08-02 14:40:16,307 [HTTPThreadGroup-6] ERROR conn.ConnectionService verifyRepositoryEx.891 - Invalid Connection Pool. ERROR = ORA-28000: the account is locked 2011-08-02 14:40:16,321 [HTTPThreadGroup-4] ERROR eml.OMSHandshake processFailure.806 - OMSHandshake failed.(AGENT URL = https://maclean01.oracledatabase12g.com:3938/emd/main)(ERROR = INTERNAL_ERROR)(CAUSE =java.sql.SQLException: ORA-28000: the account is locked ) 2011-08-02 14:40:16,368 [HTTPThreadGroup-6] ERROR eml.OMSHandshake processFailure.806 - OMSHandshake failed.(AGENT URL = https://maclean04.oracledatabase12g.com:3938/emd/main)(ERROR = INTERNAL_ERROR)(CAUSE =java.sql.SQLException: ORA-28000: the account is locked ) 2011-08-02 14:40:16,437 [HTTPThreadGroup-4] ERROR conn.ConnectionService verifyRepositoryEx.891 - Invalid Connection Pool. ERROR = ORA-28000: the account is locked 2011-08-02 14:40:16,472 [HTTPThreadGroup-4] ERROR eml.OMSHandshake processFailure.806 - OMSHandshake failed.(AGENT URL = https://maclean01.oracledatabase12g.com:3938/emd/main)(ERROR = INTERNAL_ERROR)(CAUSE =java.sql.SQLException: ORA-28000: the account is locked ) 2011-08-02 14:40:16,507 [HTTPThreadGroup-6] ERROR conn.ConnectionService verifyRepositoryEx.891 - Invalid Connection Pool. ERROR = ORA-28000: the account is locked 2011-08-02 14:40:16,536 [HTTPThreadGroup-6] ERROR eml.OMSHandshake processFailure.806 - OMSHandshake failed.(AGENT URL = https://maclean04.oracledatabase12g.com:3938/emd/main)(ERROR = INTERNAL_ERROR)(CAUSE =java.sql.SQLException: ORA-28000: the account is locked ) 2011-08-02 14:40:17,187 [HTTPThreadGroup-6] ERROR conn.ConnectionService verifyRepositoryEx.891 - Invalid Connection Pool. ERROR = ORA-28000: the account is locked 2011-08-02 14:40:17,189 [EMUI_14_40_17_/console/aboutApplication] ERROR svlt.PageHandler handleRequest.639 - java.lang.IllegalStateException: Response has already been committed java.lang.IllegalStateException: Response has already been committed at com.evermind.server.http.EvermindHttpServletResponse.resetBuffer(EvermindHttpServletResponse.java:1892) at com.evermind.server.http.ServletRequestDispatcher.unprivileged_forward(ServletRequestDispatcher.java:249) at com.evermind.server.http.ServletRequestDispatcher.access$100(ServletRequestDispatcher.java:51) at com.evermind.server.http.ServletRequestDispatcher$2.oc4jRun(ServletRequestDispatcher.java:193) at oracle.oc4j.security.OC4JSecurity.doPrivileged(OC4JSecurity.java:284) at com.evermind.server.http.ServletRequestDispatcher.forward(ServletRequestDispatcher.java:198) at oracle.sysman.emSDK.svlt.PageHandler.render(PageHandler.java:1113) at oracle.sysman.emSDK.svlt.PageHandler.myRender(PageHandler.java:1173) at oracle.sysman.emSDK.svlt.PageHandler.handleRequest(PageHandler.java:597) at oracle.sysman.emSDK.svlt.EMServlet.myDoGet(EMServlet.java:787) at oracle.sysman.emSDK.svlt.EMServlet.doGet(EMServlet.java:343) at oracle.sysman.eml.app.Console.doGet(Console.java:157) at javax.servlet.http.HttpServlet.service(HttpServlet.java:743) at javax.servlet.http.HttpServlet.service(HttpServlet.java:856) at com.evermind.server.http.ResourceFilterChain.doFilter(ResourceFilterChain.java:64) at oracle.sysman.eml.app.EMRepLoginFilter.doFilter(EMRepLoginFilter.java:107) at com.evermind.server.http.EvermindFilterChain.doFilter(EvermindFilterChain.java:15) at oracle.sysman.db.adm.inst.HandleRepDownFilter.doFilter(HandleRepDownFilter.java:146) at com.evermind.server.http.EvermindFilterChain.doFilter(EvermindFilterChain.java:17) at oracle.sysman.eml.app.BrowserVersionFilter.doFilter(BrowserVersionFilter.java:122) at com.evermind.server.http.EvermindFilterChain.doFilter(EvermindFilterChain.java:17) at oracle.sysman.emSDK.svlt.EMRedirectFilter.doFilter(EMRedirectFilter.java:102) at com.evermind.server.http.EvermindFilterChain.doFilter(EvermindFilterChain.java:17) at oracle.sysman.eml.app.ContextInitFilter.doFilter(ContextInitFilter.java:327) at com.evermind.server.http.ServletRequestDispatcher.invoke(ServletRequestDispatcher.java:621) at com.evermind.server.http.ServletRequestDispatcher.forwardInternal(ServletRequestDispatcher.java:370) at com.evermind.server.http.HttpRequestHandler.doProcessRequest(HttpRequestHandler.java:871) at com.evermind.server.http.HttpRequestHandler.processRequest(HttpRequestHandler.java:453) at com.evermind.server.http.HttpRequestHandler.serveOneRequest(HttpRequestHandler.java:221) at com.evermind.server.http.HttpRequestHandler.run(HttpRequestHandler.java:122) at com.evermind.server.http.HttpRequestHandler.run(HttpRequestHandler.java:111) at oracle.oc4j.network.ServerSocketReadHandler$SafeRunnable.run(ServerSocketReadHandler.java:260) at oracle.oc4j.network.ServerSocketAcceptHandler.procClientSocket(ServerSocketAcceptHandler.java:234) at oracle.oc4j.network.ServerSocketAcceptHandler.access$700(ServerSocketAcceptHandler.java:29) at oracle.oc4j.network.ServerSocketAcceptHandler$AcceptHandlerHorse.run(ServerSocketAcceptHandler.java:879) at com.evermind.util.ReleasableResourcePooledExecutor$MyWorker.run(ReleasableResourcePooledExecutor.java:303) at java.lang.Thread.run(Thread.java:595) 2011-08-02 14:40:17,189 [EMUI_14_40_17_/console/aboutApplication] ERROR em.console doGet.363 - java.lang.IllegalStateException: Response has already been committed, be sure not to write to the OutputStream or to trigger a commit due to any other action before calling this method. java.lang.IllegalStateException: Response has already been committed, be sure not to write to the OutputStream or to trigger a commit due to any other action before calling this method. at com.evermind.server.http.EvermindHttpServletResponse.sendRedirect(EvermindHttpServletResponse.java:1371) at oracle.sysman.emSDK.svlt.EMServlet.myDoGet(EMServlet.java:812) at oracle.sysman.emSDK.svlt.EMServlet.doGet(EMServlet.java:343) at oracle.sysman.eml.app.Console.doGet(Console.java:157) at javax.servlet.http.HttpServlet.service(HttpServlet.java:743) at javax.servlet.http.HttpServlet.service(HttpServlet.java:856) at com.evermind.server.http.ResourceFilterChain.doFilter(ResourceFilterChain.java:64) at oracle.sysman.eml.app.EMRepLoginFilter.doFilter(EMRepLoginFilter.java:107) at com.evermind.server.http.EvermindFilterChain.doFilter(EvermindFilterChain.java:15) at oracle.sysman.db.adm.inst.HandleRepDownFilter.doFilter(HandleRepDownFilter.java:146) at com.evermind.server.http.EvermindFilterChain.doFilter(EvermindFilterChain.java:17) at oracle.sysman.eml.app.BrowserVersionFilter.doFilter(BrowserVersionFilter.java:122) at com.evermind.server.http.EvermindFilterChain.doFilter(EvermindFilterChain.java:17) at oracle.sysman.emSDK.svlt.EMRedirectFilter.doFilter(EMRedirectFilter.java:102) at com.evermind.server.http.EvermindFilterChain.doFilter(EvermindFilterChain.java:17) at oracle.sysman.eml.app.ContextInitFilter.doFilter(ContextInitFilter.java:327) at com.evermind.server.http.ServletRequestDispatcher.invoke(ServletRequestDispatcher.java:621) at com.evermind.server.http.ServletRequestDispatcher.forwardInternal(ServletRequestDispatcher.java:370) at com.evermind.server.http.HttpRequestHandler.doProcessRequest(HttpRequestHandler.java:871) at com.evermind.server.http.HttpRequestHandler.processRequest(HttpRequestHandler.java:453) at com.evermind.server.http.HttpRequestHandler.serveOneRequest(HttpRequestHandler.java:221) at com.evermind.server.http.HttpRequestHandler.run(HttpRequestHandler.java:122) at com.evermind.server.http.HttpRequestHandler.run(HttpRequestHandler.java:111) at oracle.oc4j.network.ServerSocketReadHandler$SafeRunnable.run(ServerSocketReadHandler.java:260) at oracle.oc4j.network.ServerSocketAcceptHandler.procClientSocket(ServerSocketAcceptHandler.java:234) at oracle.oc4j.network.ServerSocketAcceptHandler.access$700(ServerSocketAcceptHandler.java:29) at oracle.oc4j.network.ServerSocketAcceptHandler$AcceptHandlerHorse.run(ServerSocketAcceptHandler.java:879) at com.evermind.util.ReleasableResourcePooledExecutor$MyWorker.run(ReleasableResourcePooledExecutor.java:303) at java.lang.Thread.run(Thread.java:595) 2011-08-02 14:40:21,661 [HTTPThreadGroup-6] ERROR conn.ConnectionService verifyRepositoryEx.891 - Invalid Connection Pool. ERROR = ORA-28000: the account is locked 2011-08-02 14:40:21,691 [HTTPThreadGroup-6] ERROR eml.OMSHandshake processFailure.806 - OMSHandshake failed.(AGENT URL = https://maclean03.oracledatabase12g.com:3938/emd/main)(ERROR = INTERNAL_ERROR)(CAUSE =java.sql.SQLException: ORA-28000: the account is locked ) 2011-08-02 14:40:21,822 [HTTPThreadGroup-6] ERROR conn.ConnectionService verifyRepositoryEx.891 - Invalid Connection Pool. ERROR = ORA-28000: the account is locked 2011-08-02 14:40:21,886 [HTTPThreadGroup-6] ERROR eml.OMSHandshake processFailure.806 - OMSHandshake failed.(AGENT URL = https://maclean03.oracledatabase12g.com:3938/emd/main)(ERROR = INTERNAL_ERROR)(CAUSE =java.sql.SQLException: ORA-28000: the account is locked ) 2011-08-02 14:40:27,141 [HTTPThreadGroup-6] ERROR conn.ConnectionService verifyRepositoryEx.891 - Invalid Connection Pool. ERROR = ORA-28000: the account is locked 2011-08-02 14:40:33,941 [HTTPThreadGroup-6] ERROR conn.ConnectionService verifyRepositoryEx.891 - Invalid Connection Pool. ERROR = ORA-28000: the account is locked 2011-08-02 14:40:33,975 [HTTPThreadGroup-6] ERROR eml.OMSHandshake processFailure.806 - OMSHandshake failed.(AGENT URL = https://maclean02.oracledatabase12g.com:3938/emd/main)(ERROR = INTERNAL_ERROR)(CAUSE =java.sql.SQLException: ORA-28000: the account is locked ) 2011-08-02 14:40:34,092 [HTTPThreadGroup-6] ERROR conn.ConnectionService verifyRepositoryEx.891 - Invalid Connection Pool. ERROR = ORA-28000: the account is locked 2011-08-02 14:40:34,124 [HTTPThreadGroup-6] ERROR eml.OMSHandshake processFailure.806 - OMSHandshake failed.(AGENT URL = https://maclean02.oracledatabase12g.com:3938/emd/main)(ERROR = INTERNAL_ERROR)(CAUSE =java.sql.SQLException: ORA-28000: the account is locked ) 2011-08-02 14:40:42,088 [PAFStartupThread] ERROR conn.ConnectionService verifyRepositoryEx.891 - Invalid Connection Pool. ERROR = ORA-28000: the account is locked 2011-08-02 14:40:42,332 [HTTPThreadGroup-6] ERROR conn.ConnectionService verifyRepositoryEx.891 - Invalid Connection Pool. ERROR = ORA-28000: the account is locked 2011-08-02 14:40:42,584 [HTTPThreadGroup-6] ERROR conn.ConnectionService verifyRepositoryEx.891 - Invalid Connection Pool. ERROR = ORA-28000: the account is locked 2011-08-02 14:40:42,611 [HTTPThreadGroup-6] ERROR eml.OMSHandshake processFailure.806 - OMSHandshake failed.(AGENT URL = https://maclean01.oracledatabase12g.com:3938/emd/main)(ERROR = INTERNAL_ERROR)(CAUSE =java.sql.SQLException: ORA-28000: the account is locked ) 2011-08-02 14:40:46,594 [HTTPThreadGroup-6] ERROR conn.ConnectionService verifyRepositoryEx.891 - Invalid Connection Pool. ERROR = ORA-28000: the account is locked 2011-08-02 14:40:46,622 [HTTPThreadGroup-6] ERROR eml.OMSHandshake processFailure.806 - OMSHandshake failed.(AGENT URL = https://maclean01.oracledatabase12g.com:3938/emd/main)(ERROR = INTERNAL_ERROR)(CAUSE =java.sql.SQLException: ORA-28000: the account is locked ) 2011-08-02 14:40:46,653 [HTTPThreadGroup-4] ERROR conn.ConnectionService verifyRepositoryEx.891 - Invalid Connection Pool. ERROR = ORA-28000: the account is locked 2011-08-02 14:40:46,683 [HTTPThreadGroup-4] ERROR eml.OMSHandshake processFailure.806 - OMSHandshake failed.(AGENT URL = https://maclean04.oracledatabase12g.com:3938/emd/main)(ERROR = INTERNAL_ERROR)(CAUSE =java.sql.SQLException: ORA-28000: the account is locked ) 2011-08-02 14:40:46,738 [HTTPThreadGroup-6] ERROR conn.ConnectionService verifyRepositoryEx.891 - Invalid Connection Pool. ERROR = ORA-28000: the account is locked 2011-08-02 14:40:46,766 [HTTPThreadGroup-6] ERROR eml.OMSHandshake processFailure.806 - OMSHandshake failed.(AGENT URL = https://maclean01.oracledatabase12g.com:3938/emd/main)(ERROR = INTERNAL_ERROR)(CAUSE =java.sql.SQLException: ORA-28000: the account is locked ) 2011-08-02 14:40:46,793 [HTTPThreadGroup-4] ERROR conn.ConnectionService verifyRepositoryEx.891 - Invalid Connection Pool. ERROR = ORA-28000: the account is locked 2011-08-02 14:40:46,822 [HTTPThreadGroup-4] ERROR eml.OMSHandshake processFailure.806 - OMSHandshake failed.(AGENT URL = https://maclean04.oracledatabase12g.com:3938/emd/main)(ERROR = INTERNAL_ERROR)(CAUSE =java.sql.SQLException: ORA-28000: the account is locked ) 2011-08-02 14:40:48,244 [HTTPThreadGroup-4] ERROR conn.ConnectionService verifyRepositoryEx.891 - Invalid Connection Pool. ERROR = ORA-28000: the account is locked从以上em日志中可以发现最显著的问题为ORA-28000,显然是EM相关的数据库用户被lock住了。10g中与EM相关的数据库用户包括SYSMAN和DBSNMP,分别查询该2个用户的状态:
SQL> select account_status,lock_date,username from dba_users where username in ('SYSMAN','DBSNMP'); ACCOUNT_STATUS LOCK_DATE USERNAME -------------------------------- --------------- ------------------------------ OPEN DBSNMP LOCKED(TIMED) 02-JUL-11 SYSMAN发现SYSMAN用户在7月份被锁定了,而且是LOCKED(TIMED),说明是因为登陆密码不正确多次登陆后被锁的。LOCKED(TIMED)的相关信息以下Note:
DBA_USERS.ACCOUNT_STATUS shows LOCKED(TIMED) instead of LOCKED even if PASSWORD_LOCK_TIME set to unlimited [ID 284344.1] Modified 18-FEB-2011 Type PROBLEM Status PUBLISHED In this Document Symptoms Cause Solution References Applies to: Oracle Server - Enterprise Edition - Version: 9.2.0.1 to 11.2.0.1 - Release: 9.2 to 11.2 Information in this document applies to any platform. Checked for relevance on 21-Sep-2010 Symptoms DBA_USERS.ACCOUNT_STATUS shows LOCKED(TIMED) instead of LOCKED for a user with a profile with a PASSWORD_LOCK_TIME set to unlimited and number of failed logins > FAILED_LOGIN_ATTEMPTS For eg. create profile my_profile limit failed_login_attempts 3 password_lock_time unlimited ; SQL> create user my_user identified by my_user default tablespace users temporary tablespace temp profile my_profile ; User created. SQL> grant connect to my_user ; (after 3 failed login attempts) USERNAME PROFILE ACCOUNT_STATUS ------------------------------ ------------------------------ -------------- MY_USER MY_PROFILE LOCKED(TIMED) Cause Up until Oracle RDBMS version 11.1.0.7, after a number of failed logins > FAILED_LOGIN_ATTEMPTS, the account status of the user will always be LOCKED(TIMED). One would only see LOCKED if the account was explicitly locked during create or alter user/role (ie after using CREATE USER or ALTER USER statements) eg. alter user my_lock account lock ; USERNAME PROFILE ACCOUNT_STATUS ------------------------------- MY_USER MY_PROFILE LOCKED Solution Expected behaviour is 1. Oracle release is FAILED_LOGIN_ATTEMPTS 2. Oracle release is >= 11.2 and PASSWORD_LOCK_TIME = unlimited: DBA_USERS.ACCOUNT_STATUS = LOCKED whenever the number of failed login attempts is > FAILED_LOGIN_ATTEMPTS 3. Oracle release is >= 11.2 and PASSWORD_LOCK_TIME = DBA_USERS.ACCOUNT_STATUS = LOCKED(TIMED) whenever the number of failed login attempts is > FAILED_LOGIN_ATTEMPTS于是我尝试将SYSMAN用户解锁,并重启EM并再次尝试登陆:
SQL> alter user sysman account unlock; User altered. 重启EM后重新登陆,发现问题仍未解决,再次检查账户信息 SQL> select account_status,lock_date,username from dba_users where username in ('SYSMAN','DBSNMP'); ACCOUNT_STATUS LOCK_DATE USERNAME -------------------------------- --------------- ------------------------------ OPEN DBSNMP LOCKED(TIMED) 02-AUG-11 SYSMAN发现SYSMAN用户又被LOCK住了,显然是有人在数据库内修改了SYSMAN的密码后,没有重新配置EM中的SYSMAN密码,导致每次EM启动时都会尝试用错误的密码登陆数据库,最终造成SYSMAN用户被LOCKED(TIMED)住,并出现"ORA-28000: the account is locked"错误。 找出根本原因后问题就显得十分简单了,我们只需要关闭EM,并同步数据库和EM中的SYSMAN密码,再重启EM就可以了:
1. 首先关闭EM export ORACLE_SID= export ORACLE_UNQNAME= cd $ORACLE_HOME/bin ./emctl stop dbconsole 2. 修改DB中的SYSMAN密码并解锁用户,并验证登陆正常 alter user sysman identified by ; alter user sysman account unlock; conn sysman/password 3. 更新EM中的SYSMAN密码 export ORACLE_SID= export ORACLE_UNQNAME= cd $ORACLE_HOME/bin ./emctl setpasswd dbconsole Oracle Enterprise Manager 11g Database Control Release 11.2.0.1.0 Copyright (c) 1996, 2009 Oracle Corporation. All rights reserved. https://maclean.oracledatabase12g.com:1158/em/console/aboutApplication Please enter new repository password: Repository password successfully updated. 4. 启动EM export ORACLE_SID= export ORACLE_UNQNAME= cd $ORACLE_HOME/bin./emctl start dbconsole
本文转自maclean_007 51CTO博客,原文链接:
http://blog.51cto.com/maclean/1277858