Weblogic和Oracle：Stuck ExecuteThread

Question

我們在集羣Weblogic環境中運行帶有Hibernate的SpringMVC。我們的應用程序運行良好，直到遇到數據庫空間問題。因此，我們的DB提供者擴展了數據庫存儲。

之後，更改Weblogic系統狀態是抱怨線程池「線程池已經干擾線程」。

的堆棧跟蹤日誌文件如下所示：

<Error> <WebLogicServer> <BEA-000337> <[STUCK] ExecuteThread: '12' for queue: 'weblogic.ke 
rnel.Default (self-tuning)' has been busy for "641" seconds working on the request "weblogic.servlet.internal.ServletReq 
uestImpl@17782ff3[ 
GET /default/default/applicationPreloader/load.do?_=1331887527968 HTTP/1.1 
User-Agent: Mozilla/5.0 (Windows NT 5.1; rv:10.0.2) Gecko/20100101 Firefox/10.0.2 
Accept: */* 
Accept-Language: en-gb,en;q=0.5 
Accept-Encoding: gzip, deflate 
x-requested-with: XMLHttpRequest 
Referer: http://novintegweb01.eu.acme.net:9136/default/default/applicationPreloader.do 
Cookie: JSESSIONID1=NhBdPv9Hx8Wj5lnr5vnGhLSbR2GGM1DPhqT2YGzWCGFvhN3WJWVH!700169023!1391220923 
Connection: Keep-Alive 
Proxy-Client-IP: 192.168.1.1 
X-Forwarded-For: 192.168.1.1 
X-WebLogic-KeepAliveSecs: 30 
X-WebLogic-Request-ClusterInfo: true 
x-weblogic-cluster-hash: G4kGT1kfuDBbtj5w//SYNt6hlVo 

]", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace: 
     java.net.SocketInputStream.socketRead0(Native Method) 
     java.net.SocketInputStream.read(SocketInputStream.java:129) 
     oracle.net.ns.Packet.receive(Packet.java:293) 
     oracle.net.ns.DataPacket.receive(DataPacket.java:104) 
     oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:315) 
     oracle.net.ns.NetInputStream.read(NetInputStream.java:260) 
     oracle.net.ns.NetInputStream.read(NetInputStream.java:185) 
     oracle.net.ns.NetInputStream.read(NetInputStream.java:102) 
     oracle.jdbc.driver.T4CSocketInputStreamWrapper.readNextPacket(T4CSocketInputStreamWrapper.java:124) 
     oracle.jdbc.driver.T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:80) 
     oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1136) 
     oracle.jdbc.driver.T4CMAREngine.unmarshalSB1(T4CMAREngine.java:1113) 
     oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:288) 
     oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:191) 
     oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:523) 
     oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:207) 
     oracle.jdbc.driver.T4CPreparedStatement.executeForRows(T4CPreparedStatement.java:1009) 
     oracle.jdbc.driver.OraclePreparedStatement.executeForRowsWithTimeout(OraclePreparedStatement.java:10117) 
     oracle.jdbc.driver.OraclePreparedStatement.executeBatch(OraclePreparedStatement.java:10221) 
     oracle.jdbc.driver.OracleStatementWrapper.executeBatch(OracleStatementWrapper.java:216) 
     weblogic.jdbc.wrapper.PreparedStatement.executeBatch(PreparedStatement.java:210) 
     org.hibernate.jdbc.BatchingBatcher.doExecuteBatch(BatchingBatcher.java:48) 
     org.hibernate.jdbc.AbstractBatcher.executeBatch(AbstractBatcher.java:247) 
     org.hibernate.jdbc.AbstractBatcher.prepareStatement(AbstractBatcher.java:93) 
     org.hibernate.jdbc.AbstractBatcher.prepareStatement(AbstractBatcher.java:88) 
     org.hibernate.jdbc.AbstractBatcher.prepareBatchStatement(AbstractBatcher.java:223) 
     org.hibernate.persister.collection.AbstractCollectionPersister.recreate(AbstractCollectionPersister.java:1119) 
     org.hibernate.action.CollectionRecreateAction.execute(CollectionRecreateAction.java:39) 
     org.hibernate.engine.ActionQueue.execute(ActionQueue.java:279) 
     org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:263) 
     org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:171) 
     org.hibernate.event.def.AbstractFlushingEventListener.performExecutions(AbstractFlushingEventListener.java:298) 
     org.hibernate.event.def.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:27) 
     org.hibernate.impl.SessionImpl.flush(SessionImpl.java:1001) 
     org.hibernate.impl.SessionImpl.managedFlush(SessionImpl.java:339) 
     org.hibernate.transaction.JDBCTransaction.commit(JDBCTransaction.java:106) 
     org.springframework.orm.hibernate3.HibernateTransactionManager.doCommit(HibernateTransactionManager.java:656) 
     org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:754) 
     org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:723) 
     org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:393) 
     org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:120) 
     org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) 
     org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:89) 
     org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) 
     org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor.intercept(Cglib2AopProxy.java:621) 
     com.acme.superproject.dao.CertificationDaoImpl$$EnhancerByCGLIB$$bf3135a5.getById(<generated>) 
     com.acme.superproject.integration.UserProfileSabaServiceImpl.determineTargetDateFromWs(UserProfileSabaServiceImpl.java:436) 
     com.acme.superproject.integration.UserProfileSabaServiceImpl.determineTargetDate(UserProfileSabaServiceImpl.java:401) 
     com.acme.superproject.integration.UserProfileSabaServiceImpl.createCertificationRelationObject(UserProfileSabaServiceImpl.java:373) 
     com.acme.superproject.integration.UserProfileSabaServiceImpl.getRequiredLearning(UserProfileSabaServiceImpl.java:334) 
     com.acme.superproject.integration.UserProfileSabaServiceImpl.extractUserProfile(UserProfileSabaServiceImpl.java:203) 
     com.acme.superproject.integration.UserProfileSabaServiceImpl.getUserProfile(UserProfileSabaServiceImpl.java:73) 
     com.acme.superproject.integration.UserProfileSabaServiceImpl.getUserProfile(UserProfileSabaServiceImpl.java:114) 
     com.acme.superproject.integration.UserProfileSabaServiceImpl$$FastClassByCGLIB$$e48c61c9.invoke(<generated>) 
     net.sf.cglib.proxy.MethodProxy.invoke(MethodProxy.java:149) 
     org.springframework.aop.framework.Cglib2AopProxy$CglibMethodInvocation.invokeJoinpoint(Cglib2AopProxy.java:688) 
     org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) 
     org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:80) 
     com.acme.superproject.techarch.TracingAspect.trace(TracingAspect.java:31) 
     sun.reflect.GeneratedMethodAccessor5382.invoke(Unknown Source) 
     sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) 
     java.lang.reflect.Method.invoke(Method.java:597) 
     org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621) 
     org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610) 
     org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:65) 
     org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) 
     org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:89) 
     org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) 
     org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor.intercept(Cglib2AopProxy.java:621) 
     com.acme.superproject.integration.UserProfileSabaServiceImpl$$EnhancerByCGLIB$$5d7b85df.getUserProfile(<generated>) 
     com.acme.superproject.service.UserProfileServiceImpl.getUserProfile(UserProfileServiceImpl.java:26) 
     com.acme.superproject.service.UserProfileServiceImpl$$FastClassByCGLIB$$70cfe815.invoke(<generated>) 
     net.sf.cglib.proxy.MethodProxy.invoke(MethodProxy.java:149) 
     org.springframework.aop.framework.Cglib2AopProxy$CglibMethodInvocation.invokeJoinpoint(Cglib2AopProxy.java:688) 
     org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) 
     org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:80) 
     com.acme.superproject.techarch.TracingAspect.trace(TracingAspect.java:31) 
     sun.reflect.GeneratedMethodAccessor5382.invoke(Unknown Source) 
     sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) 
     java.lang.reflect.Method.invoke(Method.java:597) 
     org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621) 
     org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610) 
     org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:65) 
     org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) 
     org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:89) 
     org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) 
     org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor.intercept(Cglib2AopProxy.java:621) 
     com.acme.superproject.service.UserProfileServiceImpl$$EnhancerByCGLIB$$5bcb212b.getUserProfile(<generated>) 
     com.acme.superproject.web.session.SessionDataProvider.fetchUserProfile(SessionDataProvider.java:97) 
     com.acme.superproject.web.session.SessionDataProvider.getUserProfile(SessionDataProvider.java:78) 
     com.acme.superproject.web.session.SessionDataProvider.fetchUserRegistrationCollection(SessionDataProvider.java:155) 
     com.acme.superproject.web.session.SessionDataProvider.getUserRegistrationCollection(SessionDataProvider.java:141) 
     com.acme.superproject.web.ApplicationPreloaderController.loadAction(ApplicationPreloaderController.java:84) 
     sun.reflect.GeneratedMethodAccessor6506.invoke(Unknown Source) 
     sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) 
     java.lang.reflect.Method.invoke(Method.java:597) 
     org.springframework.web.bind.annotation.support.HandlerMethodInvoker.invokeHandlerMethod(HandlerMethodInvoker.java:176) 
     org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.invokeHandlerMethod(AnnotationMethodHandlerAdapter.java:426) 
     org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.handle(AnnotationMethodHandlerAdapter.java:414) 
     org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:790) 
     org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:719) 
     org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:644) 
     org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:549) 
     javax.servlet.http.HttpServlet.service(HttpServlet.java:707) 
     javax.servlet.http.HttpServlet.service(HttpServlet.java:820) 
     weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:227) 
     weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:125) 
     weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:300) 
     weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26) 
     weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56) 
     org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:368) 
     org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:109) 
     org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:83) 
     org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:380) 
     org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:97) 
     org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:380) 
     org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:105) 
     org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:380) 
     org.springframework.security.web.authentication.preauth.AbstractPreAuthenticatedProcessingFilter.doFilter(AbstractPreAuthenticatedProcessingFilter.java:89) 
     org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:380) 
     org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:79) 
     org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:380) 
     org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:169) 
     org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237) 
     org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167) 
     weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56) 
     org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:88) 
     org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76) 
     weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56) 
     weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3715) 
     weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3681) 
     weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321) 
     weblogic.security.service.SecurityManager.runAs(SecurityManager.java:120) 
     weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2277) 
     weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2183) 
     weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1454) 
     weblogic.work.ExecuteThread.execute(ExecuteThread.java:207) 
     weblogic.work.ExecuteThread.run(ExecuteThread.java:176) 
> 

我們已經重新啓動的Weblogic，數據庫服務器，並提出了新的部署。問題總是一樣的：當我們嘗試訪問應用程序的登錄頁面時，它會掛起 - 就像我在嘗試執行數據庫事務時看到的一樣。除此之外，當我嘗試使用「Oracle SQL Developer」連接到我們的數據庫時，我們也得到錯誤「ORA-00257：歸檔錯誤。僅連接內部，直到被釋放。」 不幸的是，我們沒有SYSDBA權限（因爲由外部提供商管理）。

我的問題是：

• 你們當中有誰有過類似的問題？
• 你認爲這種問題的根本原因更多的是在應用方面，還是在數據庫方面更多？
• 我怎樣才能至少擺脫困擾線程？

Answer 1

很可能數據庫存儲已滿，或者歸檔程序無法移動或創建新的歸檔日誌。這幾乎肯定是數據庫級別的問題，而不是應用程序服務器級別。由於您收到此消息，這意味着您已經連接到數據庫，因此它不在等待連接。

WebLogic中的阻塞線程並不總是需要「固定」，並不總是一個問題。卡住線程機制的目的是在線程執行某些操作時（例如訪問數據庫）異常地通知您。卡住的線程通知就是這樣 - 通知某事可能不正確，在這種情況下它就是爲此目的服務的。

如果您有線程通常需要20分鐘才能完成某些操作，但如果需要25分鐘以上的時間纔會收到通知，那麼必須將Stuck Thread Limit增加到25分鐘，並在該時間點收到通知，然後您必須將此限制從600秒到1500秒（25 * 60）。

您可以編輯在WebLogic管理BASE_DOMAIN阻塞線程限制>環境>服務器>管理服務器>配置/調整。

參考以下鏈接： https://docs.oracle.com/cd/E19316-01/820-3745/ghudn/index.html

Answer 2

這似乎是數據庫的問題。線程卡在數據庫上。你的表現似乎已經退化了。我會建議把這個案子交給你的DBA進一步調查。另外，請嘗試在被訪問的表上重新創建索引。

Answer 3

我們曾面臨這個問題時，Oracle數據庫服務器是在一個VirtualBox，在開發環境。指出這個問題有點困難。我們注意到這一點，當ThreadPool沒有任何更多的線程來處理請求，並請求被拒絕。