The Question

Why do containers slow down unexpectedly when you scale them on modern multi-core CPUs, even when you have plenty of hardware resources available?

Simple Explanation

Imagine you're running a restaurant kitchen. You have multiple chefs (CPU cores), each with their own prep station (cache). When orders come in, chefs grab ingredients from a central pantry (memory).

This works fine with a few chefs. But when you pack dozens of chefs into the same kitchen, problems emerge:

  • The pantry door gets crowded — chefs wait in line to grab ingredients (memory bandwidth saturation)
  • Chefs yell across the room — "Who has the salt?" — creating noise that slows everyone (cache coherency traffic)
  • Some pantries are farther away — chefs on the east side wait longer than those near the pantry (NUMA topology)

Containers are like individual orders. Your orchestration tool (Kubernetes, Docker Swarm) keeps adding orders, assuming more chefs means faster service. But the kitchen itself becomes the bottleneck.

How It Actually Works

The Mount Namespace Problem

When you launch a container, it gets its own mount namespace — an isolated view of the filesystem. Modern container images use layered filesystems (OverlayFS), where each layer stacks on top of another.

Netflix's Titus platform discovered something alarming: nodes were stalling for 30+ seconds during health checks when launching hundreds of containers simultaneously. The kernel was spending excessive time managing mount namespaces — a problem invisible to most monitoring tools.

Each mount operation requires kernel locks. When hundreds of containers launch in parallel, those locks serialize operations that should be concurrent. The more CPU cores you have, the worse the contention becomes.

NUMA: The Hidden Geography of Memory

Modern server CPUs like AMD EPYC and Intel Xeon use Non-Uniform Memory Access (NUMA) architecture. Each CPU socket has directly attached memory (local) and memory attached to other sockets (remote).

Cross-NUMA memory access incurs 40% or higher latency compared to local access. In practical terms, this means a container scheduled on CPU socket 0 but allocated memory on socket 1 runs significantly slower — sometimes 15-40% slower for memory-intensive workloads.

Most container orchestrators ignore NUMA topology entirely. They see "128 cores available" and schedule containers wherever they fit. But those 128 cores span multiple NUMA nodes, and naive scheduling creates a hidden performance penalty.

Cache Coherency: The Invisible Tax

Each CPU core has its own cache (L1, L2, sometimes L3). When multiple cores access the same memory, the CPU must ensure all caches agree — a process called cache coherency.

In many-core systems, cache coherency delays can reach 1000+ cycles. That's not milliseconds — that's CPU cycles. A modern CPU executing billions of instructions per second can waste significant time just waiting for cache lines to synchronize.

This problem compounds with SMT (Simultaneous Multithreading), marketed as Hyper-Threading by Intel. When two threads share a physical core, they also share caches. A memory-heavy container on one thread can evict cache lines needed by its sibling thread, causing cache thrashing that hurts both containers.

Why cgroup CPU Limits Kill Throughput

Container CPU limits use the Linux kernel's CFS Bandwidth Control. You set a quota (e.g., 500ms of CPU time per 100ms period), and the kernel throttles the container when it exceeds that quota.

Here's the problem: research shows CPU limits can reduce throughput by 2-3x or more, even at low utilization. When a container hits its limit mid-request, that request stalls until the next quota period. For latency-sensitive applications, this creates unpredictable spikes.

A Datadog study found that over 65% of Kubernetes workloads use less than half their requested CPU. Teams over-provision to avoid throttling, wasting resources while still hitting performance walls during traffic spikes.

Real-World Example: Netflix Titus

Netflix runs thousands of EC2 instances launching hundreds of thousands of containers daily through Titus, their container platform. They pack containers densely — sometimes hundreds per host.

When they scaled up, they hit walls:

  • Nodes would stall during container launches, timing out health checks
  • Performance flattened at 75% CPU utilization despite adding more containers
  • Noisy neighbor effects caused unpredictable latency spikes

Their solution? Netflix built titus-isolate, a subsystem that uses machine learning to predict optimal container placement. It considers CPU topology, cache sharing patterns, and historical performance to place containers intelligently rather than randomly. This reduced peak CPU usage and improved P99 latency by 13% for key services.

Why It Matters

If you're running containers at scale, these invisible bottlenecks affect you. Traditional monitoring shows CPU utilization and memory usage, but misses:

  • NUMA imbalance — containers scheduled on wrong sockets
  • Cache contention — too many containers sharing the same cache
  • Mount namespace serialization — kernel lock contention during scale-out
  • CFS throttling — artificial limits causing latency spikes

The solution isn't more hardware — it's smarter orchestration. Tools like Kubernetes Topology Manager can enforce NUMA alignment. Setting CPU requests without limits avoids throttling. And understanding your workload patterns helps you pack containers efficiently rather than densely.

Modern CPUs are incredibly powerful, but their complexity creates new failure modes. The orchestrators that win will be the ones that understand hardware topology, not just resource quotas.

Further Reading