COS正在用完/ var/lib/docker的inode卷

Question

我正試圖讓我們在GCP上運行一些服務。

一個我看到目前的問題是，我得非常快速啓動虛擬機似乎用盡inode爲/ var/lib中/泊塢窗文件系統。我以爲這是在容器優化的os中調整的東西之一？

wouter@nbwm-cron ~ $ df -hi Filesystem Inodes IUsed IFree IUse% Mounted on /dev/root 78K 13K 65K 17%/ devtmpfs 463K 204 463K 1% /dev tmpfs 464K 1 464K 1% /dev/shm tmpfs 464K 500 463K 1% /run tmpfs 464K 13 464K 1% /sys/fs/cgroup tmpfs 464K 9 464K 1% /mnt/disks tmpfs 464K 16K 448K 4% /tmp /dev/sda8 4.0K 11 4.0K 1% /usr/share/oem /dev/sda1 1013K 998K 15K 99% /var tmpfs 464K 45 464K 1% /var/lib/cloud overlayfs 464K 39 464K 1% /etc

wouter@nbwm-cron ~ $ docker images REPOSITORY TAG IMAGE ID CREATED SIZE <name>/stackdriver-agent latest 0c4b075e7550 3 days ago 1.423 GB <none> <none> 96d027d3feea 4 days ago 905.2 MB gcr.io/<project>/nbwm-ops/docker-php5 latest 5d2c59c7dd7a 2 weeks ago 1.788 GB

nbwm-cron wouter # tune2fs -l /dev/sda1 tune2fs 1.43.3 (04-Sep-2016) Filesystem volume name: STATE Last mounted on: /var Filesystem UUID: ca44779b-ffd5-405a-bd3e-528071b45f73 Filesystem magic number: 0xEF53 Filesystem revision #: 1 (dynamic) Filesystem features: has_journal ext_attr resize_inode dir_index filetype needs_recovery extent 64bit flex_bg sparse_super large_file huge_file uninit_bg dir_nlink extra_isize Filesystem flags: signed_directory_hash Default mount options: user_xattr acl Filesystem state: clean Errors behavior: Remount read-only Filesystem OS type: Linux Inode count: 1036320 Block count: 4158971 Reserved block count: 0 Free blocks: 4062454 Free inodes: 1030756 First block: 0 Block size: 4096 Fragment size: 4096 Group descriptor size: 64 Reserved GDT blocks: 747 Blocks per group: 32768 Fragments per group: 32768 Inodes per group: 8160 Inode blocks per group: 510 Flex block group size: 16 Filesystem created: Thu Jun 15 22:39:33 2017 Last mount time: Wed Jun 28 13:51:31 2017 Last write time: Wed Jun 28 13:51:31 2017 Mount count: 5 Maximum mount count: -1 Last checked: Thu Nov 19 19:00:00 2009 Check interval: 0 (<none>) Lifetime writes: 67 MB Reserved blocks uid: 0 (user root) Reserved blocks gid: 0 (group root) First inode: 11 Inode size: 256 Required extra isize: 32 Desired extra isize: 32 Journal inode: 8 Default directory hash: half_md4 Directory Hash Seed: 66aa0e7f-57da-41d0-86f7-d93270e53030 Journal backup: inode blocks

如何調整文件系統有更多的inode可用？

Answer 1

這是存在docker中的overlay存儲驅動程序的已知問題，由overlay2驅動程序解決。 新的cos-61 releases使用docker 17.03和overlay2存儲驅動程序。你能否試試看看問題是否再次發生？

謝謝！

Answer 2

編輯：爲2017年10月10日的，GKE是使kubernetes 1.6.11和1.7.8可用COS 61個圖像與碼頭工人17.03：https://cloud.google.com/container-engine/release-notes#october-10-2017

我有同樣的問題與COS 57.9202.64.0對GKE（泊塢窗1.11.2與overlay驅動程序），它解決了使用Spotify的搬運工-GC：

# docker run --rm --userns host -v /var/run/docker.sock:/var/run/docker.sock -v /etc:/etc spotify/docker-gc 

前：

# df -hi /var/lib/docker/ 
Filesystem  Inodes IUsed IFree IUse% Mounted on 
/dev/sda1  6.0M 5.0M 1.1M 83% /var 

# df -h /var/lib/docker/ 
Filesystem  Size Used Avail Use% Mounted on 
/dev/sda1  95G 84G 11G 89% /var 

# du --inodes -s /var/lib/docker/* 
180  /var/lib/docker/containers 
4093 /var/lib/docker/image 
4  /var/lib/docker/network 
4906733 /var/lib/docker/overlay 
1  /var/lib/docker/tmp 
1  /var/lib/docker/trust 
25  /var/lib/docker/volumes 

後：

# df -hi /var/lib/docker/ 
Filesystem  Inodes IUsed IFree IUse% Mounted on 
/dev/sda1  6.0M 327K 5.7M 6% /var/lib/docker 

# df -h /var/lib/docker/ 
Filesystem  Size Used Avail Use% Mounted on 
/dev/sda1  95G 6.6G 88G 7% /var/lib/docker 

# du --inodes -s /var/lib/docker/* 
218  /var/lib/docker/containers 
1792 /var/lib/docker/image 
4  /var/lib/docker/network 
279002 /var/lib/docker/overlay 
1  /var/lib/docker/tmp 
1  /var/lib/docker/trust 
25  /var/lib/docker/volumes 

注：

使用通常的docker rmi $(docker images --filter "dangling=true" -q --no-trunc)和docker rm $(docker ps -qa --no-trunc --filter "status=exited")無助於恢復資源/var/lib/docker/overlay。

而且，我不相信這是可能升級GKE節點池COS 61的圖像。