kube-apiserver 报 too many open files 错误信息

发布 : 2019-05-05 分类 : 运维 浏览 :

kube-apiserver 报 too many open files 错误信息

引言

最近使用kubectl命令时,经常会卡住,应该是与kube-apiserver连接有问题,经查看日志发现有too many open files 的错误,以下是解决过程记录

问题描述

日志错误如下:

1
2
3
4
5
6
7
8

Apr 28 21:38:50 k8s-master-7 kube-apiserver: I0428 21:38:50.430345 103893 logs.go:49] http: Accept error: accept tcp [::]:6443: accept4: too many open files; retrying in 20ms
Apr 28 21:38:50 k8s-master-7 kube-apiserver: I0428 21:38:50.450568 103893 logs.go:49] http: Accept error: accept tcp [::]:6443: accept4: too many open files; retrying in 40ms
Apr 28 21:38:50 k8s-master-7 kube-apiserver: I0428 21:38:50.490833 103893 logs.go:49] http: Accept error: accept tcp [::]:6443: accept4: too many open files; retrying in 80ms
Apr 28 21:38:50 k8s-master-7 kube-apiserver: I0428 21:38:50.571101 103893 logs.go:49] http: Accept error: accept tcp [::]:6443: accept4: too many open files; retrying in 160ms
Apr 28 21:38:50 k8s-master-7 kube-apiserver: I0428 21:38:50.628440 103893 wrap.go:42] GET /api/v1/namespaces/hotfix/secrets/registry?resourceVersion=0: (679.961µs) 200 [[kubelet/v1.10.5 (linux/amd64) kubernetes/32ac1c9] 10.10.1.2:45969]
Apr 28 21:38:50 k8s-master-7 kube-apiserver: I0428 21:38:50.629772 103893 wrap.go:42] GET /api/v1/namespaces/hotfix/secrets/registry?resourceVersion=0: (577.741µs) 200 [[kubelet/v1.10.5 (linux/amd64) kubernetes/32ac1c9] 10.10.1.2:45969]
Apr 28 21:38:50 k8s-master-7 kube-apiserver: I0428 21:38:50.731451 103893 logs.go:49] http: Accept error: accept tcp [::]:6443: accept4: too many open files; retrying in 320ms

问题解决

错误很明显应该是文件描述符不够用了,但是我ulimit 查看一下系统的配置,open files是65535,这样看系统的配置应该是正常的。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
#  ulimit -a
core file size (blocks, -c) unlimited
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 127857
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 65535
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 127857
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited

在看一下kube-apiserver服务的limit,kube-apiserver服务竟然的是1024

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
# systemctl status kube-apiserver.service  | grep PID
Main PID: 45112 (kube-apiserver)
# cat /proc/45112/limits
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 8388608 unlimited bytes
Max core file size 0 unlimited bytes
Max resident set unlimited unlimited bytes
Max processes 127857 127857 processes
Max open files 1024 1024 files
Max locked memory 65536 65536 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 127857 127857 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us

经查找相关文档原来需要在kube-apiserver.service配置中[Service]中添加 LimitNOFILE= 选项,可以LimitNOFILE=65535,或LimitNOFILE=infinity

1
2
3
4
5
6
7
8
9
10
11
12
13
cat /usr/lib/systemd/system/kube-apiserver.service
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes

[Service]
EnvironmentFile=-/xdfapp/server/kubernetes/config/kube-apiserver
ExecStart=/xdfapp/server/kubernetes/bin/kube-apiserver $KUBE_APISERVER_OPTS
Restart=on-failure
LimitNOFILE=infinity

[Install]
WantedBy=multi-user.target

重启服务验证limit是否有变化

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
# systemctl daemon-reload
# systemctl restart kube-apiserver
# systemctl status kube-apiserver | grep PID
Main PID: 126057 (kube-apiserver)
# cat /proc/126057/limits
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 8388608 unlimited bytes
Max core file size 0 unlimited bytes
Max resident set unlimited unlimited bytes
Max processes 127857 127857 processes
Max open files 65536 65536 files
Max locked memory 65536 65536 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 127857 127857 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us

ok搞定

本文作者 : WGY
原文链接 : http://geeklive.cn/2019/05/05/kube-apiserver/undefined/kube-apiserver/
版权声明 : 本博客所有文章除特别声明外,均采用 CC BY-NC-SA 4.0 许可协议。转载请注明出处!
留下足迹