kubernetes无法删除pod问题记录

环境信息

# uname -a
Linux wf-b19-04-vm-op-k8s-master-7 3.10.0-862.el7.x86_64 #1 SMP Fri Apr 20 16:44:24 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
# docker -v
Docker version 1.13.1, build dded712/1.13.1
# kubectl version
Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.5", GitCommit:"32ac1c9073b132b8ba18aa830f46b77dcceb0723", GitTreeState:"clean", BuildDate:"2018-06-21T11:46:00Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.5", GitCommit:"32ac1c9073b132b8ba18aa830f46b77dcceb0723", GitTreeState:"clean", BuildDate:"2018-06-21T11:34:22Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}

问题现象

更新了deployment后，经常会出现老的pod无法被删除的情况，卡在Terminating状态

# kubectl get pods -o wide | grep Terminating
test2-688945d85b-2rstv       0/1       Terminating   0          8d        10.254.42.46   10.10.0.199

简单粗暴的解决办法

使用在删除时添加参数 –force –grace-period=0，但是删除时会有警告，看意思是，强制删除不保证资源就立即停止了，资源可能会无限期的运行在集群上，虽然是警告但这不是一个好办法还是要找到具体原因。

# kubectl delete pod test2-688945d85b-r8rrw --force  --grace-period=0
warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely.
pod "test2-688945d85b-r8rrw" deleted

–force 强制删除

–grace-period=0，grace-period表示过渡存活期，默认30s，在删除POD之前允许POD慢慢终止其上的容器进程，从而优雅退出，0表示立即终止POD

排查记录

1. 复现问题

先查看一下pod

# kubectl get pods  -o wide

将Running状态的pod删除

  
2. 查看docker的日志
 看docker日志中的报错意思应该是容器停止
```shell
# journalctl -xe --no-pager  -u docker
May 08 14:30:07 wf-b19-06-op-k8s-node-199 dockerd[2525]: time="2019-05-08T02:30:07.245507385-04:00" level=error msg="Handler for POST /v1.27/containers/470343975c9cc155ae7139b8b6959d6ec694918835c38a444904cded1278d555/stop returned error: Container 470343975c9cc155ae7139b8b6959d6ec694918835c38a444904cded1278d555 is already stopped"
May 08 14:33:17 wf-b19-06-op-k8s-node-199 dockerd[2525]: time="2019-05-08T02:33:17.451296517-04:00" level=error msg="Handler for POST /v1.27/containers/a4908b6bcd80d22779ad57fe0b496d1631520cae736da26a972f52eb6fae6c71/stop returned error: Container a4908b6bcd80d22779ad57fe0b496d1631520cae736da26a972f52eb6fae6c71 is already stopped"
May 08 14:33:17 wf-b19-06-op-k8s-node-199 dockerd[2525]: time="2019-05-08T02:33:17.459284356-04:00" level=error msg="Error removing mounted layer 5db030c7a9b9a67e9caecdaa3cfd9b45257e4599bf8fdf4de6fb84d29bf98934: remove /xdfapp/data/docker/overlay/7a8b4e41f72acaa1974ebc2caa7215fabd014973cb73d703f6c1dd156a58e7a7/merged: device or resource busy"
May 08 14:33:17 wf-b19-06-op-k8s-node-199 dockerd[2525]: time="2019-05-08T02:33:17.459669385-04:00" level=error msg="Handler for DELETE /v1.27/containers/5db030c7a9b9a67e9caecdaa3cfd9b45257e4599bf8fdf4de6fb84d29bf98934 returned error: Driver overlay failed to remove root filesystem 5db030c7a9b9a67e9caecdaa3cfd9b45257e4599bf8fdf4de6fb84d29bf98934: remove /xdfapp/data/docker/overlay/7a8b4e41f72acaa1974ebc2caa7215fabd014973cb73d703f6c1dd156a58e7a7/merged: device or resource busy"
May 08 14:33:18 wf-b19-06-op-k8s-node-199 dockerd[2525]: time="2019-05-08T02:33:18.504333845-04:00" level=error msg="Handler for POST /v1.27/containers/a4908b6bcd80d22779ad57fe0b496d1631520cae736da26a972f52eb6fae6c71/stop returned error: Container a4908b6bcd80d22779ad57fe0b496d1631520cae736da26a972f52eb6fae6c71 is already stopped"

参考文档

http://kuring.me/post/docker-rm-failed/