2010-07-26 120 views
12

Hudson在構建了幾個項目後,使用以下堆棧跟蹤重複失敗,錯誤爲「設備上沒有剩餘空間」,雖然有足夠的空間磁盤。任何文件夾都沒有配額限制。以下是不同系統命令的輸出。Hudson出現錯誤「設備上沒有剩餘空間」,儘管系統上有足夠的空間

這是關鍵的系統信息:

Hudson ver. 1.361 
executable-war /opt/hudson/hudson.war 
java.runtime.name OpenJDK Runtime Environment 
java.runtime.version 1.6.0_18-b18 
os.name Linux-Ubuntu 10.04 
os.version 2.6.32-19-generic 

有根據DF

$ df 
Filesystem   1K-blocks  Used Available Use% Mounted on 
/dev/sda1   147550696 67382688 72672840 49%/
none     1535580  304 1535276 1% /dev 
none     1539732  504 1539228 1% /dev/shm 
none     1539732  96 1539636 1% /var/run 
none     1539732   0 1539732 0% /var/lock 
none     1539732   0 1539732 0% /lib/init/rw 
none     147550696 67382688 72672840 49% /var/lib/ureadahead/debugfs 

下一個我檢查打開的文件句柄通過lsof的50%的可用空間,也那是極限內

lsof | wc -l 
694 

然後我通過這個命令檢查了文件句柄

cat /proc/sys/fs/file-nr 
3392 0 306935 

詳細錯誤

[INFO] No tests to run. 
[HUDSON] Recording test results 
[INFO] [jar:jar {execution: default-jar}] 
[INFO] Building jar: /opt/hudson/jobs/EIF_Branch_R1.2__Projects_PkgOnly/workspace/r1.2/projects/CRMServices/deployment/target/eif.deployment.CRMServices-1.2-bt.jar 
[INFO] [antrun:run {execution: default}] 
[INFO] Executing tasks 
    [unzip] Expanding: /opt/hudson/jobs/EIF_Branch_R1.2__Projects_PkgOnly/workspace/r1.2/projects/CRMServices/deployment/src/it/resources/CRMServices.Driver-soapui-project.zip into /opt/hudson/jobs/EIF_Branch_R1.2__Projects_PkgOnly/workspace/r1.2/projects/CRMServices/deployment/src/it/resources 
[HUDSON] Archiving /opt/hudson/jobs/EIF_Branch_R1.2__Projects_PkgOnly/workspace/r1.2/projects/CRMServices/deployment/pom.xml to /opt/hudson/jobs/EIF_Branch_R1.2__Projects_PkgOnly/modules/rogers.bt.deployment$eif.deployment.CRMServices/builds/2010-07-20_12-13-58/archive/rogers.bt.deployment/eif.deployment.CRMServices/1.2-bt/pom.xml 
[HUDSON] Archiving /opt/hudson/jobs/EIF_Branch_R1.2__Projects_PkgOnly/workspace/r1.2/projects/CRMServices/deployment/target/eif.deployment.CRMServices-1.2-bt.jar to /opt/hudson/jobs/EIF_Branch_R1.2__Projects_PkgOnly/modules/rogers.bt.deployment$eif.deployment.CRMServices/builds/2010-07-20_12-13-58/archive/rogers.bt.deployment/eif.deployment.CRMServices/1.2-bt/eif.deployment.CRMServices-1.2-bt.jar 
[HUDSON] Re-archiving /opt/hudson/jobs/EIF_Branch_R1.2__Projects_PkgOnly/workspace/r1.2/projects/CRMServices/deployment/target/eif.deployment.CRMServices-1.2-bt.jar 
[INFO] ------------------------------------------------------------------------ 
[ERROR] FATAL ERROR 
[INFO] ------------------------------------------------------------------------ 
[INFO] Failed to serialize hudson.model.Actionable#actions for class hudson.maven.MavenModuleSetBuild 
No space left on device 
[INFO] ------------------------------------------------------------------------ 
[INFO] Trace 
java.langchannel stopped 
ERROR: Failed to parse POMs 
java.io.IOException: Remote call on Channel to Maven [/opt/bea/jdk160_05/bin/java, -cp, /opt/hudson/plugins/maven-plugin/WEB-INF/lib/maven-agent-1.363.jar:/opt/hudson/tools/Maven_2.2.1/boot/classworlds-1.1.jar, hudson.maven.agent.Main, /opt/hudson/tools/Maven_2.2.1, /opt/hudson/war/WEB-INF/lib/remoting-1.363.jar, /opt/hudson/plugins/maven-plugin/WEB-INF/lib/maven-interceptor-1.363.jar, 55951, /opt/hudson/plugins/maven-plugin/WEB-INF/lib/maven2.1-interceptor-1.2.jar] failed 
       at hudson.remoting.Channel.call(Channel.java:564) 
       at hudson.maven.ProcessCache$MavenProcess.call(ProcessCache.java:156) 
       at hudson.maven.MavenModuleSetBuild$RunnerImpl.doRun(MavenModuleSetBuild.java:483) 
       at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:416) 
       at hudson.model.Run.run(Run.java:1253) 
       at hudson.maven.MavenModuleSetBuild.run(MavenModuleSetBuild.java:306) 
       at hudson.model.ResourceController.execute(ResourceController.java:88) 
       at hudson.model.Executor.run(Executor.java:127) 
Caused by: java.lang.Error: Unable to load resource hudson/maven/Messages.properties 
       at hudson.remoting.RemoteClassLoader.findResource(RemoteClassLoader.java:198) 
       at java.lang.ClassLoader.getResource(ClassLoader.java:977) 
       at java.lang.Class.getResource(Class.java:2074) 
       at org.jvnet.localizer.ResourceBundleHolder.get(ResourceBundleHolder.java:83) 
       at org.jvnet.localizer.ResourceBundleHolder.get(ResourceBundleHolder.java:102) 
       at org.jvnet.localizer.ResourceBundleHolder.get(ResourceBundleHolder.java:102) 
       at org.jvnet.localizer.ResourceBundleHolder.format(ResourceBundleHolder.java:139) 
       at hudson.maven.Messages.MavenBuilder_AsyncFailed(Messages.java:233) 
       at hudson.maven.MavenBuilder.call(MavenBuilder.java:184) 
       at hudson.maven.MavenModuleSetBuild$Builder.call(MavenModuleSetBuild.java:696) 
       at hudson.maven.MavenModuleSetBuild$Builder.call(MavenModuleSetBuild.java:640) 
       at hudson.remoting.UserRequest.perform(UserRequest.java:114) 
       at hudson.remoting.UserRequest.perform(UserRequest.java:48) 
       at hudson.remoting.Request$2.run(Request.java:270) 
       at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) 
       at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) 
       at java.util.concurrent.FutureTask.run(FutureTask.java:138) 
       at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885) 
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) 
       at java.lang.Thread.run(Thread.java:619) 
Caused by: java.io.IOException: No space left on device 
       at java.io.FileOutputStream.writeBytes(Native Method) 
       at java.io.FileOutputStream.write(FileOutputStream.java:247) 
       at hudson.remoting.RemoteClassLoader.makeResource(RemoteClassLoader.java:267) 
       at hudson.remoting.RemoteClassLoader.findResource(RemoteClassLoader.java:194) 
       ... 19 more 
FATAL: : No space left on device 
hudson.util.IOException2: : No space left on device 
       at hudson.XmlFile.write(XmlFile.java:168) 
       at hudson.model.Run.save(Run.java:1383) 
       at hudson.maven.MavenModuleSetBuild$RunnerImpl.post2(MavenModuleSetBuild.java:595) 
       at hudson.model.AbstractBuild$AbstractRunner.post(AbstractBuild.java:528) 
       at hudson.model.Run.run(Run.java:1276) 
       at hudson.maven.MavenModuleSetBuild.run(MavenModuleSetBuild.java:306) 
       at hudson.model.ResourceController.execute(ResourceController.java:88) 
       at hudson.model.Executor.run(Executor.java:127) 
Caused by: com.thoughtworks.xstream.io.StreamException: : No space left on device 
       at com.thoughtworks.xstream.core.util.QuickWriter.flush(QuickWriter.java:73) 
       at com.thoughtworks.xstream.io.xml.PrettyPrintWriter.endNode(PrettyPrintWriter.java:288) 
       at com.thoughtworks.xstream.io.WriterWrapper.endNode(WriterWrapper.java:37) 
       at com.thoughtworks.xstream.io.path.PathTrackingWriter.endNode(PathTrackingWriter.java:48) 
       at com.thoughtworks.xstream.core.TreeMarshaller.start(TreeMarshaller.java:99) 
       at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.marshal(AbstractTreeMarshallingStrategy.java:38) 
       at com.thoughtworks.xstream.XStream.marshal(XStream.java:840) 
       at com.thoughtworks.xstream.XStream.marshal(XStream.java:829) 
       at com.thoughtworks.xstream.XStream.toXML(XStream.java:804) 
       at hudson.XmlFile.write(XmlFile.java:165) 
       ... 7 more 
Caused by: java.io.IOException: No space left on device 
       at java.io.FileOutputStream.writeBytes(FileOutputStream.java) 
       at java.io.FileOutputStream.write(FileOutputStream.java:260) 
       at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:212) 
       at java.io.BufferedWriter.flush(BufferedWriter.java:236) 
       at hudson.util.AtomicFileWriter.flush(AtomicFileWriter.java:91) 
       at com.thoughtworks.xstream.io.xml.PrettyPrintWriter.endNode(PrettyPrintWriter.java:288) 
       at com.thoughtworks.xstream.io.path.PathTrackingWriter.endNode(PathTrackingWriter.java:49) 
       at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.marshal(AbstractTreeMarshallingStrategy.java:38) 
       at hudson.XmlFile.write(XmlFile.java:165) 
       at hudson.model.Run.save(Run.java:1384) 
       at hudson.maven.MavenModuleSetBuild$RunnerImpl.post2(MavenModuleSetBuild.java:595) 
       at hudson.model.AbstractBuild$AbstractRunner.post(AbstractBuild.java:528) 

回答

22

「無空間留在設備上」錯誤是沒有必要引起用完的存儲容量,因爲它表明,它也可以通過運行出的i是起因 - 文件系統上的節點。換句話說,給定的文件系統只能包含如此多的文件。運行df將提示一切正常。

有關更多信息,請參閱this article

您可能需要刪除一些不再需要的文件,或者將Hudson放在不同的文件系統上。

這種情況並不少見,你知道你將有很多小文件來構建一個帶有明確更大的inode表的文件系統。

+0

非常感謝許多skaffman 當我以root身份運行hudson時,問題消失了。所以這可能是因爲哈德森在寫字的地方它沒有訪問過... 謝謝和問候, 的Sohel – 2010-08-20 14:55:53

+2

輝煌鏈接「DF -i」救了我的燻肉這個 – 2017-03-07 17:59:36

0

這是由於內存溢出。

運行此命令檢查可用的內存大小

df 

如果您發現100%或接近100%的任何磁盤,請從該目錄中刪除數據,但要小心,然後再試一次。

在大多數情況下,/ tmp目錄將是100%,所以重新啓動計算機

sudo reboot 

會抹掉tmp目錄下的所有文件。

+0

這個特殊的解決方案對我的作品。謝謝 – 2017-01-06 09:33:39

0

也做ipcs -uipcs -p看看「空間不足」是SHM內存已滿和什麼應用程序正在使用它。如果分配的分段等於sysctl kernel.shmmni,則表示您是「完整」shm。您還可以與ipcs -l輸出進行比較,因爲這顯示了系統限制。

最新版本的java可能會濫用shm用法,直到它不是免費的。要麼關閉正在濫用shm的程序,要麼增加kernel.shmmnikernel.shmmax sysctl來增加可用的shm空間

相關問題