An experiment by developer Artem Yarulin has managed to run millions of web servers on a single virtual machine with 10 GB of RAM and one CPU core, with the stated goal of reaching 15 million. To get there, he had to abandon the classic one-process-per-server model and the Linux kernel network stack itself, which runs out of memory at around 2.5 million servers, and replace it with smoltcp, a user-space TCP/IP stack fed through AF_XDP or DPDK. Free Linux has analysed the work and reproduced some of its measurements.

Key points of the 15 million web server experiment in 20 seconds

  • The target: 15 million web servers on a 10 GB, single-core VM.
  • With the kernel stack, each socket takes about 3.4 KB and memory runs out at 2.55 million.
  • With smoltcp in user space, each server costs about 620 bytes idle, reaching 16 million.
  • It is a proof of concept, not a production design.

The premise is deliberately over the top: give every person on Earth their own web server. The real value lies in the journey, because each obstacle that appears when multiplying servers matches a limit any administrator can run into when scaling connections: ephemeral ports, kernel memory per socket and the cost of processes.

From one process per server to 16 bytes per guest

The first attempt was the naive one: one minimal C server per process. Each consumes around 1.5 MB, so roughly 7,000 fit in 10 GB. Reaching 15 million would take on the order of 20 TB of memory.

The fix was to put every server inside a single process. Each runs as a guest isolated with LFI (Lightweight Fault Isolation), a software sandboxing technique developed at Stanford, and a user-space scheduler switches between them. Each guest’s state takes about 16 bytes, so 15 million fit in around 220 MB.

The next wall was ports. A server is identified by protocol, IP address and port, and Linux’s default ephemeral port range (32768 to 60999) only offers 28,232 of them. Server number 28,232 failed. The way out is AnyIP: a local route that makes the kernel answer for an entire subnet as its own. The author used 10.64.0.0/16 for IPv4, and with IPv6 a single /64 would do.

# Make the machine answer for the whole subnet as local (AnyIP)
ip route add local 10.64.0.0/16 dev lo

# Check the ephemeral port range
sysctl net.ipv4.ip_local_port_rangeCode language: PHP (php)

The real limit: kernel memory per socket

With addressing solved, the underlying obstacle surfaced. Every socket() call reserves about 3.4 KB of kernel memory, whether the socket is idle, listening or active. With the Linux network stack, the VM ran out of memory at around 2.55 million servers, and requests per second dropped as their number grew.

Free Linux reproduced that measurement on a Linux 7.0.14 kernel and got between 3,550 and 3,580 bytes per socket: 200,000 sockets used about 710 MB. Measurements through MemAvailable varied between runs, so growth of the kernel slab cache was taken as the reference. This is not waste a tunable can trim: those structures underpin TCP behaviour, from timers to buffers and connection state.

# See how much memory TCP socket structures use in the slab cache
sudo grep -E 'TCP|sock_inode_cache' /proc/slabinfo
sudo slabtop -o | head -20Code language: PHP (php)
ApproachCost per serverCapacity on 10 GB
One process per server~1.5 MB~7,000 (calculated)
Kernel network stack~3.4 KB per socket plus guest stateOut of memory at ~2.55 million
smoltcp in user space~620 B idle, ~1 KB active16 million

Bypassing the kernel: smoltcp with AF_XDP or DPDK

Going beyond 2.5 million meant dropping a level and reimplementing TCP/IP in user space. The author chose smoltcp 0.14.0, a Rust TCP/IP stack under the 0BSD licence, originally aimed at embedded systems. With it, each server takes about 620 bytes idle and around 1 KB active, and the same VM reaches 16 million.

smoltcp needs packets delivered without going through the kernel stack, and the experiment tests three paths: TUN/TAP, AF_XDP and DPDK. The last two matter most to an administrator:

  • DPDK (Data Plane Development Kit) is the fastest route. The program takes exclusive control of the network card, with a core dedicated to busy-polling it. The card disappears from the operating system, so if it is the same interface SSH uses, you lose access.
  • AF_XDP is close behind. The card stays under kernel control and an eBPF program diverts only the servers’ traffic to user space, so SSH and other services keep working.

The tests used the xsk-rs library for AF_XDP and rust-dpdk-net for DPDK. A minimal C server reached about 65,000 requests per second, although the work does not publish per-server latency or a detailed throughput comparison between AF_XDP and DPDK.

What sysadmins can take away

Nobody needs 15 million servers on one VM, but the exercise leaves useful lessons. The ephemeral port range and addressing are often the first bottleneck when scaling outbound connections or per-port services, and AnyIP is a clean way to get many addresses without configuring them one by one. Kernel memory per socket is a fixed cost worth factoring in when sizing load balancers, proxies or servers with millions of open connections.

The choice between AF_XDP and DPDK is not only about speed either: DPDK requires dedicating the card and a core, which means having a separate management interface, while AF_XDP coexists with the rest of the system. The work’s limits are worth remembering: it is a proof of concept, the runtime does not offer full POSIX compatibility, and Free Linux did not repeat the 16 million benchmark, so the smoltcp, AF_XDP and DPDK figures are the author’s. His code is available on GitHub.

Frequently asked questions

How much memory does a socket use on Linux?

According to the experiment, about 3.4 KB of kernel memory per socket() call. Free Linux measured 3,550 to 3,580 bytes on Linux 7.0.14.

What is the difference between AF_XDP and DPDK?

DPDK takes exclusive control of the network card and is faster, but the interface disappears from the OS. AF_XDP diverts only part of the traffic with eBPF and leaves the card under kernel control.

What is smoltcp?

A TCP/IP stack written in Rust that runs in user space, under the 0BSD licence. In the experiment, each server uses about 620 bytes idle with it.

What is AnyIP?

A local route that makes the system answer for every address in a subnet as its own, without configuring them individually.

Sources:

Scroll to Top