Architecture

The Build Fleet

Ephemeral CI agents, where their dependencies come from, and why a fleet that worked for a year starts failing on a busy Tuesday.

scroll to render

What you're looking at

Pull requests queue work; an autoscaler starts one short-lived pod per job. Each pod checks out the repository into a workspace volume, resolves dependencies, builds, and dies. Because the pods are ephemeral, everything they need has to come over the network every time unless something is deliberately cached — and everything they send goes out through a shared NAT gateway.

Why a single egress address is a ceiling

Public package registries throttle by source address. A fleet that egresses through one NAT address looks, to an upstream registry, like one very busy client. Combined with no persistent dependency cache, every build cold-downloads the full graph from that one address. That is a fixed ceiling, and nothing about it changes until a busy day crosses it — which is why it reads as a sudden outage rather than a capacity problem.

The fix is a mirror, not a retry

Pointing builds at a pull-through remote repository in your own registry means the agent only ever talks to you, and your registry fetches upstream server-side and caches it. Per-address throttling becomes structurally impossible rather than less likely. Adding NAT addresses buys relief; persisting the cache makes builds fast; the mirror is what makes the failure mode go away.

The other failure: storage, and a poisoned cache

A workspace on an ephemeral volume counts against the container's storage limit, so a big checkout plus a cold dependency download can evict the pod mid-job. The tear-down then produces errors naming whatever process died with it, which sends you chasing the wrong component. Worse, the cache layer only writes on a miss and never overwrites a hit — so if the first build on a new cache key is killed mid-download, it saves a partial archive that every later build then hits, re-downloads around, and never refreshes. A closed loop that only breaks when one run is allowed to finish.

Inspect it yourself

Separate eviction from cancellation
```
kubectl -n NS get events --sort-by=.lastTimestamp | grep -iE 'evict|oom'
```
Lost-agent and process-spawn errors are tear-down artifacts of a pod dying mid-job, never the cause. The event says whether it was evicted or simply cancelled by a rapid re-push.
Check the limit, not the disk
```
kubectl -n NS exec POD -- df -h /tmp/work
```
Free space on the node alongside an eviction message means it was limit enforcement, not physical exhaustion. That reframes the entire investigation.
Count your egress addresses
```
gcloud compute routers nats describe NAT --router ROUTER --region REGION --format='value(natIps)'
```
One manually assigned address for a whole CI fleet is the finding. Compare it against the workload gateway, which usually has several.
Spot a poisoned partial cache
```
grep -E 'cache hit|Downloading' build.log | head -30
```
A reported cache hit followed by downloads of everything means the stored archive is a sliver. Changing one line in the requirements file changes the key, and if that first build dies, the partial is what every later build inherits.

Read the source

Components

PULL REQUESTS— bursty
AUTOSCALER— one pod per job
AGENT PODS— ephemeral
WORKSPACE— counts to limit
DEP CACHE— saves on miss only
NAT GATEWAY— shared egress
REMOTE MIRROR— pull-through
PUBLIC REGISTRY— throttles per IP
ARTIFACTS— signed images

Flows

pr→scaler
scaler→agentsscale out
agents→work
cache⇢agentsrestore
agents→nat
nat→mirror
mirror→publicserver-side
nat⇢public429 per-IP
agents→artifactspush