JVM消耗所有的CPU，大部分線程被阻塞。 JVM的錯誤？

昨晚，在VMWare上的Linux VM中運行的服務器（JBoss 5.1GA，Java（TM）SE運行時環境（版本1.6.0_20-b02）， Java HotSpot TM 64位服務器VM）突然啓動使用100％的CPU。該應用程序是一個非常典型的運行Seam的J2EE商業應用程序，沒有什麼特別的。當時的負載非常低。JVM消耗所有的CPU，大部分線程被阻塞。 JVM的錯誤？

我設法在進程死亡並重新啓動之前獲得線程轉儲（應用程序必須可用）。 JStack在死鎖檢測中有一個例外;除了大多數線程處於阻塞狀態之外。

這是一個JVM的錯誤？我以前沒有見過。重新啓動後，一切正常，就像自該應用程序在該服務器上使用幾個月後（定期更新）一樣。

感謝您的任何建議。

稍微編輯的線程轉儲（以適應SO的郵政大小限制）如下。我從JBoss/Seam中的各個線程池中刪除了一些相同的線程，並且只保留了棧跟蹤中的頂部線。

編輯：全線程轉儲在http://pastie.org/1083984

[~]$ jstack -F 13553 
Attaching to process ID 13553, please wait... 
Debugger attached successfully. 
Server compiler detected. 
JVM version is 11.3-b02 
Deadlock Detection: 

java.lang.NullPointerException 
     at sun.jvm.hotspot.oops.InstanceKlass.computeSubtypeOf(InstanceKlass.java:426) 
     at sun.jvm.hotspot.oops.Klass.isSubtypeOf(Klass.java:137) 
     at sun.jvm.hotspot.oops.Oop.isA(Oop.java:92) 
     at sun.jvm.hotspot.runtime.DeadlockDetector.print(DeadlockDetector.java:93) 
     at sun.jvm.hotspot.runtime.DeadlockDetector.print(DeadlockDetector.java:39) 
     at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:52) 
     at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45) 
     at sun.jvm.hotspot.tools.JStack.run(JStack.java:60) 
     at sun.jvm.hotspot.tools.Tool.start(Tool.java:221) 
     at sun.jvm.hotspot.tools.JStack.main(JStack.java:86) 
     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) 
     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) 
     at java.lang.reflect.Method.invoke(Method.java:597) 
     at sun.tools.jstack.JStack.runJStackTool(JStack.java:118) 
     at sun.tools.jstack.JStack.main(JStack.java:84) 
Can't print deadlocks:null 
Thread 26208: (state = BLOCKED) 


Thread 25250: (state = BLOCKED) 
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame) 
- java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=158 (Interpreted frame) 

Thread 25249: (state = BLOCKED) 
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame) 
- java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=158 (Interpreted frame) 

Thread 21240: (state = BLOCKED) 
- java.lang.Object.wait(long) @bci=0 (Interpreted frame) 
- java.lang.Object.wait() @bci=2, line=485 (Interpreted frame) 
- org.apache.tomcat.util.net.JIoEndpoint$Worker.await() @bci=8, line=416 (Interpreted frame) 

Thread 21140: (state = BLOCKED) 
- java.lang.Object.wait(long) @bci=0 (Interpreted frame) 
- java.lang.Object.wait() @bci=2, line=485 (Interpreted frame) 
- org.apache.tomcat.util.net.JIoEndpoint$Worker.await() @bci=8, line=416 (Interpreted frame) 

Thread 21139: (state = BLOCKED) 
- java.lang.Object.wait(long) @bci=0 (Interpreted frame) 
- java.lang.Object.wait() @bci=2, line=485 (Interpreted frame) 
- org.apache.tomcat.util.net.JIoEndpoint$Worker.await() @bci=8, line=416 (Interpreted frame) 

Thread 21138: (state = BLOCKED) 
- java.lang.Object.wait(long) @bci=0 (Interpreted frame) 
- java.lang.Object.wait() @bci=2, line=485 (Interpreted frame) 
- org.apache.tomcat.util.net.JIoEndpoint$Worker.await() @bci=8, line=416 (Interpreted frame) 

Thread 19380: (state = BLOCKED) 
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame) 
- java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=158 (Interpreted frame) 

Thread 19377: (state = BLOCKED) 
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame) 
- java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=158 (Interpreted frame) 

Thread 19361: (state = BLOCKED) 
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame) 

Thread 19343: (state = BLOCKED) 
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame) 
- java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=158 (Interpreted frame) 

Thread 19317: (state = BLOCKED) 
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame) 
- java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=158 (Interpreted frame) 

Thread 18995: (state = BLOCKED) 
- java.lang.Object.wait(long) @bci=0 (Interpreted frame) 
- java.lang.Object.wait() @bci=2, line=485 (Interpreted frame) 

Thread 18986: (state = BLOCKED) 
- java.lang.Object.wait(long) @bci=0 (Interpreted frame) 
- java.lang.Object.wait() @bci=2, line=485 (Interpreted frame) 

Thread 17659: (state = BLOCKED) 
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame) 
- java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=158 (Interpreted frame) 

Thread 17658: (state = BLOCKED) 
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame) 
- java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=158 (Interpreted frame) 

Thread 17653: (state = BLOCKED) 
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame) 
- java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=158 (Interpreted frame) 

Thread 17631: (state = BLOCKED) 
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame) 
- java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=158 (Interpreted frame) 

Thread 14977: (state = BLOCKED) 
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame) 
- java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=158 (Interpreted frame) 

Thread 14796: (state = BLOCKED) 
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame) 
- java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=158 (Interpreted frame) 

Thread 14609: (state = BLOCKED) 
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame) 
- java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=158 (Interpreted frame) 

Thread 14437: (state = BLOCKED) 
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame) 
- java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=158 (Interpreted frame) 

Thread 14248: (state = BLOCKED) 
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame) 
- java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=158 (Interpreted frame) 

Thread 14232: (state = BLOCKED) 
- java.util.Arrays.copyOf(char[], int) @bci=1, line=2882 (Compiled frame) 
- java.lang.AbstractStringBuilder.expandCapacity(int) @bci=33, line=100 (Compiled frame) 

Thread 14040: (state = IN_NATIVE) 
- java.net.PlainSocketImpl.socketAccept(java.net.SocketImpl) @bci=0 (Interpreted frame) 
- java.net.PlainSocketImpl.accept(java.net.SocketImpl) @bci=7, line=384 (Interpreted frame) 

Thread 13858: (state = BLOCKED) 
- java.lang.Object.wait(long) @bci=0 (Interpreted frame) 
- java.lang.ref.ReferenceQueue.remove(long) @bci=44, line=116 (Compiled frame) 

Thread 13815: (state = BLOCKED) 
- java.lang.Object.wait(long) @bci=0 (Interpreted frame) 
- java.lang.Object.wait() @bci=2, line=485 (Interpreted frame) 
- org.apache.tomcat.util.net.JIoEndpoint$Worker.await() @bci=8, line=416 (Interpreted frame) 

Thread 13814: (state = IN_NATIVE) 
- java.net.PlainSocketImpl.socketAccept(java.net.SocketImpl) @bci=0 (Interpreted frame) 

Thread 13813: (state = BLOCKED) 
- java.lang.Object.wait(long) @bci=0 (Interpreted frame) 
- java.lang.Object.wait() @bci=2, line=485 (Interpreted frame) 
- org.apache.tomcat.util.net.JIoEndpoint$Worker.await() @bci=8, line=416 (Interpreted frame) 


Thread 13812: (state = BLOCKED) 
Error occurred during stack walking: 
sun.jvm.hotspot.utilities.AssertionFailure: range check 
     at sun.jvm.hotspot.utilities.Assert.that(Assert.java:32) 
     at sun.jvm.hotspot.runtime.RegisterMap.<init>(RegisterMap.java:109) 
     at sun.jvm.hotspot.runtime.x86.X86RegisterMap.<init>(X86RegisterMap.java:39) 
     at sun.jvm.hotspot.runtime.x86.X86RegisterMap.clone(X86RegisterMap.java:43) 
     at sun.jvm.hotspot.runtime.VFrame.<init>(VFrame.java:37) 
     at sun.jvm.hotspot.runtime.JavaVFrame.<init>(JavaVFrame.java:45) 
     at sun.jvm.hotspot.runtime.CompiledVFrame.<init>(CompiledVFrame.java:43) 
     at sun.jvm.hotspot.runtime.VFrame.newVFrame(VFrame.java:77) 
     at sun.jvm.hotspot.runtime.VFrame.sender(VFrame.java:129) 
     at sun.jvm.hotspot.runtime.VFrame.javaSender(VFrame.java:146) 
     at sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:231) 
     at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76) 
     at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45) 
     at sun.jvm.hotspot.tools.JStack.run(JStack.java:60) 
     at sun.jvm.hotspot.tools.Tool.start(Tool.java:221) 
     at sun.jvm.hotspot.tools.JStack.main(JStack.java:86) 
     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) 
     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) 
     at java.lang.reflect.Method.invoke(Method.java:597) 
     at sun.tools.jstack.JStack.runJStackTool(JStack.java:118) 
     at sun.tools.jstack.JStack.main(JStack.java:84) 


Thread 13811: (state = IN_NATIVE) 
- java.net.PlainSocketImpl.socketAccept(java.net.SocketImpl) @bci=0 (Interpreted frame) 

Thread 13810: (state = IN_NATIVE) 
- java.net.PlainSocketImpl.socketAccept(java.net.SocketImpl) @bci=0 (Interpreted frame) 

Thread 13809: (state = BLOCKED) 
- java.lang.Object.wait(long) @bci=0 (Interpreted frame) 
- java.lang.Object.wait() @bci=2, line=485 (Interpreted frame) 
- org.jboss.jms.client.container.ClientConsumer.getMessage(long) @bci=71, line=866 (Interpreted frame) 

Thread 13808: (state = BLOCKED) 
- java.lang.Object.wait(long) @bci=0 (Interpreted frame) 
- java.lang.Object.wait() @bci=2, line=485 (Interpreted frame) 
- org.jboss.jms.client.container.ClientConsumer.getMessage(long) @bci=71, line=866 (Interpreted frame) 

Thread 13807: (state = BLOCKED) 
- java.lang.Object.wait(long) @bci=0 (Interpreted frame) 
- java.lang.Object.wait() @bci=2, line=485 (Interpreted frame) 
- org.jboss.bootstrap.AbstractServerImpl$LifeThread.run() @bci=11, line=866 (Interpreted frame) 

Thread 13806: (state = BLOCKED) 
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame) 
- java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=158 (Interpreted frame) 

Thread 13805: (state = BLOCKED) 
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame) 
- java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=158 (Interpreted frame) 

Thread 13804: (state = BLOCKED) 
- java.lang.Object.wait(long) @bci=0 (Interpreted frame) 
- java.util.TimerThread.mainLoop() @bci=201, line=509 (Interpreted frame) 

Thread 13803: (state = BLOCKED) 
- java.lang.Object.wait(long) @bci=0 (Interpreted frame) 
- java.util.TimerThread.mainLoop() @bci=201, line=509 (Interpreted frame) 

Thread 13802: (state = BLOCKED) 
- java.util.Arrays.copyOfRange(char[], int, int) @bci=40, line=3209 (Compiled frame) 
- java.lang.String.<init>(char[], int, int) @bci=65, line=216 (Compiled frame) 
- java.lang.StringBuffer.toString() @bci=13, line=585 (Compiled frame) 
- org.apache.log4j.PatternLayout.format(org.apache.log4j.spi.LoggingEvent) @bci=68, line=506 (Compiled frame) 

Thread 13784: (state = BLOCKED) 
- java.lang.Object.wait(long) @bci=0 (Interpreted frame) 
- java.lang.ref.ReferenceQueue.remove(long) @bci=44, line=116 (Compiled frame) 
- org.jboss.el.util.ReferenceCache$ReferenceQueueRunner.run() @bci=1, line=159 (Interpreted frame) 

Thread 13753: (state = BLOCKED) 
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise) 
- org.jboss.ejb3.cache.simple.SimpleStatefulCache$SessionTimeoutTask.block() @bci=11, line=210 (Interpreted frame) 

Thread 13752: (state = BLOCKED) 
- java.lang.Object.wait(long) @bci=0 (Interpreted frame) 
- org.jboss.resource.connectionmanager.IdleRemover$IdleRemoverRunnable.run() @bci=31, line=167 (Interpreted frame) 

Thread 13749: (state = BLOCKED) 
- java.lang.Object.wait(long) @bci=0 (Interpreted frame) 
- java.lang.Object.wait() @bci=2, line=485 (Interpreted frame) 
- org.jboss.remoting.transport.socket.ServerThread.run() @bci=209, line=284 (Interpreted frame) 

Thread 13734: (state = BLOCKED) 
- java.lang.Object.wait(long) @bci=0 (Interpreted frame) 
- java.util.TimerThread.mainLoop() @bci=201, line=509 (Interpreted frame) 

Thread 13701: (state = BLOCKED) 
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame) 
- java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=158 (Interpreted frame) 

Thread 13700: (state = BLOCKED) 
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise) 
- org.jboss.console.plugins.AOPLister$RefreshPoller.run() @bci=41, line=898 (Interpreted frame) 

Thread 13699: (state = BLOCKED) 
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise) 
- org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run() @bci=22, line=1590 (Interpreted frame) 
- java.lang.Thread.run() @bci=11, line=619 (Interpreted frame) 


Thread 13698: (state = IN_NATIVE) 
- java.net.PlainSocketImpl.socketAccept(java.net.SocketImpl) @bci=0 (Interpreted frame) 

Thread 13678: (state = BLOCKED) 
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame) 
- java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20, line=198 (Interpreted frame) 

Thread 13554: (state = BLOCKED) 

Thread 13560: (state = BLOCKED) 

Thread 13559: (state = BLOCKED) 
- java.lang.Object.wait(long) @bci=0 (Interpreted frame) 
- java.lang.ref.ReferenceQueue.remove(long) @bci=44, line=116 (Compiled frame) 
- java.lang.ref.ReferenceQueue.remove() @bci=2, line=132 (Compiled frame) 
- java.lang.ref.Finalizer$FinalizerThread.run() @bci=3, line=159 (Compiled frame) 


Thread 13558: (state = BLOCKED) 
- java.lang.Object.wait(long) @bci=0 (Interpreted frame) 
- java.lang.Object.wait() @bci=2, line=485 (Compiled frame) 
- java.lang.ref.Reference$ReferenceHandler.run() @bci=46, line=116 (Compiled frame)

來源

2010-08-10 wishihadabettername

我想查看唯一阻塞的線程的完整堆棧跟蹤。也許你可以在沒有SO限制的情況下重新發布整個轉儲？ – Gary 2010-08-10 14:34:22

我把它放在pastie上，並用鏈接更新了問題（http://pastie.org/1083984） – wishihadabettername 2010-08-10 15:12:46

一種可能性是，你跑出來的堆空間。如果你的堆空間不夠用，JVM將開始以一個巨大的速度燒燬CPU，從而進行瘋狂的垃圾收集。

來源

2010-08-10 18:03:12

我遇到了同樣的問題，實際上堆空間太小。如果您使用jmap，則可以確定堆空間是否確實是您的問題。如果從命令行運行'jmap -heap '，輸出將顯示已使用的堆空間的百分比。 – Roy 2012-09-12 12:05:07

另一種可能性是磁盤或RAM錯誤。這些非常罕見，特別是如果您使用ECC數據傳輸時包含糾錯的ECC RAM，SAS或SCSI。但它可能發生！

我讀過IBM和Sun內核工程師看到的一些非常奇怪的錯誤，這些錯誤支持大企業，他們只能指責這類東西。

除了讓你知道你可能永遠不會發現它發生的真實原因，它可能永遠不會再發生多年，這並沒有多大幫助。 :)

來源

2010-08-10 18:12:19

JVM消耗所有的CPU，大部分線程被阻塞。 JVM的錯誤？

回答

相關問題