Three Interacting Bugs Caused GKE API Timeouts, Ending in a Linux Kernel TCP Fix
Google traced intermittent Kubernetes API timeouts on GKE to three interacting bugs spanning Private Service Connect, Envoy and the Linux kernel. The final fix was a kernel patch, backported to LTS branches.
TL;DR
- Intermittent Kubernetes API timeouts on Google Kubernetes Engine (GKE) came from three bugs across different layers, not one.
- The causes were packet reordering in Private Service Connect (PSC), Envoy's half-close behavior, and a Linux kernel bug that discarded certain reordered FIN packets.
- The final fix was a kernel patch, merged into Linux 6.9.7 and backported to the 6.6 and 6.1 LTS branches, reaching GKE in version 1.33.5 and later.
What went wrong
In January, a customer running several GKE clusters with Private Service Connect for private control-plane access saw intermittent timeouts when pods tried to reach the Kubernetes API server. Packet captures showed the control plane answering new TCP connection requests with stale ACK packets that belonged to connections already closed. Each symptom looked like a plain connection failure, which made the cause hard to pin down.
A FIN packet is the TCP segment that signals one side is done sending. A socket stuck in FIN_WAIT_2 is one that has sent its FIN and is waiting for the peer's FIN, which never arrives in a usable form. In this case, the sockets stayed in that state indefinitely.
Three causes, one symptom
According to Akash Thakur, senior technical solutions engineer at Google Cloud, engineers from GKE, Private Service Connect, networking, the kernel team and the Linux upstream community worked together to find three separate root causes. The investigation pointed to three interacting factors: packet reordering inside PSC, Envoy's half-close behavior, which kept TCP connections from closing cleanly, and a Linux kernel bug that dropped some reordered FIN packets.
Thakur noted that most networking cases resolve at a single layer, such as a misconfigured firewall rule or an exhausted connection pool. Here, the same "i/o timeout" symptom appeared whichever layer was examined, and each layer offered an explanation that was only partly complete.
How the fix came together
The remediation took several weeks and three coordinated steps. Google first rolled back a Private Service Connect change to restore connectivity. It then corrected proxy protocol handling and added regression tests. Finally, a Linux kernel patch written by Eric Dumazet, formerly a software engineer at Google, fixed the underlying TCP issue.
The article's own commentary is useful here. Justin Garrison, field CTO at Sidero, pointed out that GKE was running a kernel from three years earlier (6.1), and that the bug had already been fixed upstream and had to be backported. Rohan Karne, managing director at Google Cloud, described the work as methodically ruling out the obvious suspects until packet captures revealed the kernel root cause.
The case also has limits. The report is an anonymized incident and gives no population failure rate, so it does not show how common this problem is or explain every API timeout on GKE.
Why it matters
For developers running workloads on GKE or other managed Kubernetes platforms, the case shows that a timeout can originate several layers below the application. Platform teams should correlate network symptoms with infrastructure rollouts, since that correlation helped link the three bugs. Packet captures at multiple layers were essential. Teams that depend on a managed kernel should also check which kernel version their nodes run and whether upstream fixes have reached it, because the fix only arrived through a backport to LTS branches.
Sources
This article was written by Blozhik from the reporting below. Read the originals for full detail.
InHow Google Traced GKE API Timeouts to a Linux Kernel TCP BugInfoQ