Explaining Buildkite's incident takes the node vocabulary Kubernetes hides from service owners
Lorin Hochstein rates Buildkite's public write-up above GitHub's on detail, and the first thing his own retelling has to do is teach clusters, nodes and headroom, the layer a platform keeps out of sight.
The Engineer · Build desk
What happened
- Lorin Hochstein published his own reading of Buildkite's public incident write-up, days after writing about a GitHub Actions incident at the other CI-as-a-service provider.
- Retelling the failure mode in his own words required him to define Kubernetes clusters, nodes and pods first, noting that on Amazon's EKS he believes a node is an EC2 instance.
- He labels the unused part of a node headroom, a term he says is not explicitly a Kubernetes concept, meaning the additional compute jobs that node could still run.
Why it matters
- What it limits constraint The engineer paged first sees only the layer the platform exposes, so finishing the diagnosis needs someone with cluster access, and that handoff happens while the queue is still growing.
- Who has to decide decision Anyone alerting on pending pods has to split the alert, because a slow image pull and a cluster with no room to schedule need different responses and one threshold cannot serve both.
- What it sets up precedent A write-up detailed enough for an outsider to reconstruct and republish raises what readers can reasonably ask of the next CI vendor's incident report.
Next story · Build · 3d ago
A one-commit checkout and a trailing || true let 41 PRs skip tests under green CIOne report1 publisher
Red, green and gray on Hochstein's example graph do different jobs. Red is the desired replica count the autoscaler computed. Green counts pods in the running state, which the Kubernetes docs define as bound to a node with all containers created and at least one still running [13]. Pending is the gray band, and the docs put two unlike delays in that single state: a pending pod "has been accepted by the Kubernetes cluster, but one or more of the containers has not been set up and made ready to run", and that "includes time a Pod spends waiting to be scheduled as well as the time spent downloading container images over the network" [12]. A pending count on its own does not say which of the two you have [17].
Image download is bounded by the network [12]. Waiting to be scheduled is bounded by whether any node still has headroom [9]. The HorizontalPodAutoscaler increases or decreases the size of a replicaset in response to load [11], and a replicaset is a group of identical pods, added only because the service needs more compute [10]. Nothing in that loop creates a node. With every node full, a scale-out decision raises the desired count and the pending count by the same amount [16].
Hochstein is precise about who can see any of this. Service owners know pods, he wrote, "because that's what's exposed to you", while "those implementation details are deliberately hidden from the service owners" [8]. The engineer paged for a CI queue that has stopped draining is looking at pods.
For this failure to transfer to another cluster, several things have to hold. The scaling signal has to act on pod count and not on node count [11]. The node pool has to stay fixed while that happens. And the pods have to be large enough relative to the node that headroom runs out before the replica target is met [9]. Buildkite is a CI company [3], and Hochstein notes that a service consuming from a job queue sees the number of pending jobs vary at any one time [10].
He also notes that Buildkite's write-up says scale out, and that he uses scale up and scale out interchangeably, which is more disclosure than most glossaries offer [15].
Hochstein calls what the report describes "a fascinating failure mode" [4]. The text in hand ends at the autoscaling graph, before the node-level cause is named [5]. On the comparison with the GitHub Actions incident he had written about days earlier [1], he is unambiguous: Buildkite's "incident report contains a lot more detail than GitHub's" [2].
What to watch
- Whether Buildkite publishes node-level numbers, such as instance type and per-pod resource requests, that would let an outsider reproduce the failure.
- Whether the full Buildkite report ties its scale-out to a cluster autoscaler or to a fixed node pool.
- Whether GitHub adds node or cluster detail to its Actions incident report.
Clarity's read
What the record supports and how the coverage leans. The claims behind it follow.
Reality
- Evidence55
- Adoption
- Not scored
- Hype gap0
- Incentives20
- Confidence60
Claim ledger
Ranked by verification strength, evidence, and original report placement.
- [1]
Lorin Hochstein published his own observations on Buildkite's public incident write-up, having earlier written about a GitHub Actions incident.
- [2]
Hochstein wrote of the Buildkite write-up: "their incident report contains a lot more detail than GitHub's."
- [3]
Buildkite is another CI company, and recently released a public write-up of an incident it experienced.
- [4]
Hochstein wrote that the Buildkite write-up "describes what I found to be a fascinating failure mode".
- [5]
The text of Hochstein's post available here ends at the autoscaling graph example, before the node-level cause of the Buildkite failure is described.
- [6]
A Kubernetes cluster is made up of multiple nodes, and Hochstein writes that on EKS, Amazon's Kubernetes offering, he believes those nodes are EC2 instances.
- [7]
Each node exists to run pods, which are the compute workloads run on Kubernetes.
- [8]
Hochstein wrote that a service owner on Kubernetes is almost certainly familiar with pods "because that's what's exposed to you", but is likely not familiar with nodes and clusters because "those implementation details are deliberately hidden from the service owners".
- [9]
Hochstein labels the blank part of a node "headroom", says it is not explicitly a Kubernetes concept, and defines it as "the amount of additional compute jobs that could still be run on this node".
- [10]
A service on Kubernetes is deployed as a replicaset, a group of identical pods, and the only reason to have multiple pods is the need for additional compute resources; a request-driven service sees request volume vary and a queue consumer sees the number of pending jobs vary at any one time.
- [11]
The HorizontalPodAutoscaler increases or decreases the size of a replicaset depending on the load.
- [12]
The Kubernetes pod lifecycle docs define pending as: "The Pod has been accepted by the Kubernetes cluster, but one or more of the containers has not been set up and made ready to run. This includes time a Pod spends waiting to be scheduled as well as the time spent downloading container images over the network."
- [13]
The Kubernetes pod lifecycle docs define running as: "The Pod has been bound to a node, and all of the containers have been created. At least one container is still running, or is in the process of starting or restarting."
- [14]
In Hochstein's example autoscaling graph, the red desired replica count starts at 6 and the autoscaler later adjusts it to 8; green shows 6 pods running at the start and 8 at the end, and gray shows pods in the pending state.
- [15]
Hochstein notes that the Buildkite write-up uses the term scale out, and says he uses scale up and scale out interchangeably in his post.
- [16]
The HorizontalPodAutoscaler changes replicaset size and not node count, so pods it adds can only run where node headroom exists; with full nodes, a scale-out raises the desired count and the pending count by the same amount.
- [17]
Because pending covers both waiting to be scheduled and image download, a pending pod count alone does not identify which delay is occurring.
Sources
2 independent publishers whose own reporting we read for this story.
- Thoughts on the Buildkite Aug 25 incident
daily.dev
1 article · September 21, 2026
- Lorin Hochstein's analysis
surfingcomplexity.blog
1 article · September 20, 2026
Topics and entities
Topics
- Continuous integration
- Kubernetes autoscaling
- Incident Analysis
Entities
- Buildkite
- CoreDNS
- Amazon EKS
- Kubernetes
- HorizontalPodAutoscaler
- GitHub Actions