author ｜ Luo Bingli
source ｜ Erda official account
One night , The customer encountered such a problem ：K8s The cluster has been failing to expand , All nodes cannot join the cluster normally . After many twists and turns, there is no solution , Customers feed back problems to us , Hope to get technical support . The whole investigation process of this problem is quite interesting , In this paper, the investigation ideas and methods are summarized and shared with you , Hope to be able to help you in the investigation of such problems and reference .
Operation and maintenance students in the customer’s K8s When the cluster is expanding the capacity of nodes , It is found that the newly added node has failed to be added all the time . The preliminary investigation results are as follows ：
- On the new node , visit K8s master service vip The Internet is not working .
- On the new node , Direct access K8s master hostIP + 6443 Normal network .
- On the new node , Access containers for other nodes IP It’s ok ping through .
- On the new node , visit coredns service vip Normal network .
The customer uses Kubernetes The version is 1.13.10, The kernel version of the host is 4.18（centos 8.2）.
Received feedback from the front-line colleagues , We have a preliminary suspicion that ipvs The problem of . According to the past experience of network troubleshooting , We did some routine checks on the scene first ：
- Confirm kernel module ip_tables Whether to load （ normal ）
- confirm iptable forward Default accpet （ normal ）
- Confirm whether the host network is normal （ normal ）
- Confirm if the container network is normal （ normal ）
After eliminating the usual problems , It can basically narrow the scope , Now let’s continue based on ipvs Relevant levels of investigation .
1. adopt ipvsadm Order the investigation
10.96.0.1 It’s a customer cluster K8s master service vip.
As shown in the figure above , We can see that there is an abnormal connection , be in SYN_RECV The state of , And it can be observed that , Startup time kubelet + kube-proxy There is a normal connection , The explanation is after starting up ,K8s service There’s something wrong with the network .
2. tcpdump Caught analysis
Grab the bag at both ends , And pass telnet 10.96.0.1 443 Command to confirm .
Conclusion ： Find out SYN The packet is not sent out on this machine .
3. A preliminary summary
Through the above investigation , We can narrow it down again ： The problem is basically kube-proxy On the body . We used ipvs Pattern , I also rely on iptables Configure some network forwarding 、snat、drop etc. .
According to the above investigation process , We’ve narrowed it down again , Start analyzing the suspect kube-proxy.
4. see kube-proxy journal
As shown in the figure above ： Exception log found ,iptables-restore Command execution exception . adopt Google、 Community view , Identify the problem .
relevant issue For links, please refer to ：
Check… By code （1.13.10 edition pkg/proxy/ipvs/proxier.go:1427）, It can be found that this version does not judge KUBE-MARK-DROP Whether there is and creates logic . When the chain does not exist , There will be logical flaws , Lead to iptable Command execution failed .
K8s master service vip no , Related to the actual container ip Yes. , The reason for this situation , And the following iptable The rules are about ：
iptable -t nat -A KUBE-SERVICES ! -s 18.104.22.168/8 -m comment --comment "Kubernetes service cluster ip + port for masquerade purpose" -m set --match-set KUBE-CLUSTER-IP dst,dst -j KUBE-MARK-MASQ
6. Root cause research
We already know that kube-proxy 1.13.10 There is a flaw in the version , Without creating KUBE-MARK-DROP In the case of chains , perform iptables-restore Command configuration rules . But why K8s 1.13.10 The version runs in centos8.2 4.18 An error will be reported on the operating system of the kernel , Run in the centos7.6 3.10 The kernel’s operating system is normal ？
Let’s look at kube-proxy Source code , You can find kube-proxy Actually, it’s execution iptables Command to configure rules . Since the kube-proxy Report errors iptables-restore Command fails , We’re looking for one 4.18 The kernel machine , Get into kube-proxy Let’s see .
Go to the container and execute iptables-save command , You can find kube-proxy There’s really no creation in the container KUBE-MARK-DROP chain （ Meet code expectations ）. Continue on the host iptables-save command , But I found that there was KUBE-MARK-DROP chain .
There are two questions ：
- Why? 4.18 The kernel host’s iptables Yes KUBE-MARK-DROP chain ？
- Why? 4.18 The kernel host’s iptables Rules and kube-proxy The rules in the container are inconsistent ？
The first doubt , Doubt by feeling, except kube-proxy, There will be other programs operating iptables, Keep rolling K8s Code .
Conclusion ： It turns out that except for kube-proxy, also kubelet It will also modify iptables The rules . Specific code can be viewed ：pkg/kubelet/kubelet_network_linux.go
The second question , Continue to feel ······Google Let’s have a look at why kube-proxy The container mounts the host /run/xtables.lock In the case of documents , Hosts and containers iptables The rules for viewing are inconsistent .
Conclusion ：CentOS 8 Get rid of… On the Internet iptables, use nftables Framework as the default network packet filter tool .
thus , All the mysteries have been solved .
The team has done a lot of customer project delivery , Here are some more questions to answer ：
- Question 1 ： Why do so many customer environments encounter this situation for the first time ？
Because of the need K8s 1.13.10 + centos 8.2 Operating system of , This combination is rare , And problems will arise . upgrade K8s 1.16.0+ There is no such problem .
- Question two ： Why use K8s 1.13.10 + 5.5 The kernel doesn’t have this problem ？
Because with centos 8 Operating system , We manually upgrade 5.5 After version , The default is still used iptables frame .
Can pass iptables -v command , To confirm whether to use nftables.
Digression ：nftables Who is holy ？ Than iptables Good yao ？ This is another point worth further learning , There’s no more depth here .
Summary and perception
In view of the above problems , Let’s summarize the solutions ：
- Adjust the kernel version to 3.10（centos 7.6+）, Or manually upgrade the kernel version to 5.0 +;
- upgrade Kubernetes edition , The current confirmation 1.16.10+ Version does not have this problem .
That’s what we’re doing Kubernetes A little experience in network troubleshooting , Hope to be able to effectively check , It helps to pinpoint the cause .
If for Erda There are other things you want to know about the project , welcome Add small assistant wechat （Erda202106） Join the community ！
Welcome to open source
Erda As an open source one-stop cloud native PaaS platform , Have DevOps、 Microservice observation governance 、 Platform level capabilities such as multi cloud management and fast data governance . Click the link below to participate in open source , Discuss with many developers 、 communication , Build an open source community . Welcome to pay attention 、 Contribution code and Star！
- Erda Github Address ：https://github.com/erda-project/erda
- Erda Cloud Official website ：https://www.erda.cloud/