For years, high availability meant buying better servers, building clusters, adding redundancy, and hoping everything would keep running when the next failure occurred.
Netflix proved that wasn’t enough.
About 15 years ago, the company started deliberately shutting down production servers at random. Not because they were broken, but to verify whether its architecture could survive the loss of any server without users noticing.
That tool became known as Chaos Monkey, and it laid the foundations for what is now called Chaos Engineering, one of the most influential practices in modern site reliability engineering (SRE).
With Kubernetes, microservices, distributed systems, and AI agents increasingly making infrastructure decisions automatically, intentionally injecting failures is no longer an odd experiment. It’s becoming a best practice.
Chaos Engineering in 20 seconds
- Netflix created Chaos Monkey to randomly terminate production servers.
- The idea evolved into platforms capable of simulating latency, network failures, storage issues, and even entire cloud region outages.
- Today, tools such as AWS Fault Injection Service, LitmusChaos, and Chaos Mesh make these experiments accessible to any organization.
- The goal isn’t to break systems—it’s to discover weaknesses before your users do.
The biggest mistake is assuming everything works
Every systems administrator has heard statements like:
“We’re highly available.”
“We’ve got redundancy.”
“If a node fails, nothing happens.”
The reality is that many of those assumptions have never actually been tested.
The secondary node has never taken over.
Backups have never been restored.
Database replication has been failing silently for weeks.
The load balancer contains a configuration error nobody noticed.
Everything appears reliable… until the day it isn’t.
Chaos Engineering starts from a simple principle:
Don’t assume your infrastructure works.
Prove it.
Chaos Monkey changed the way we think about availability
When Netflix migrated to AWS, it realized failures were no longer exceptional events.
In an environment with thousands of virtual machines, servers were constantly being created, terminated, or failing.
Expecting every instance to remain healthy all the time simply wasn’t realistic.
So Netflix built Chaos Monkey.
Its behavior was incredibly simple.
During business hours, it randomly selected an EC2 instance and terminated it.
That was it.
If the application kept running, the architecture was healthy.
If users experienced an outage, the problem wasn’t Chaos Monkey.
The architecture had failed the test.
Then Netflix started testing real-world failures
Netflix soon realized that shutting down servers covered only a small fraction of possible failure scenarios.
Distributed systems fail in many different ways.
That’s why the Simian Army was created.
Each “monkey” simulated a different type of failure.
Among the most well-known were:
- Latency Monkey, which injected artificial latency between services.
- Chaos Gorilla, capable of simulating the complete loss of an AWS Availability Zone.
- Chaos Kong, which tested what would happen if an entire AWS Region disappeared.
Later, Netflix evolved toward far more sophisticated platforms such as ChAP (Chaos Automation Platform) and FIT (Failure Injection Testing).
Instead of randomly breaking infrastructure, every experiment became hypothesis-driven and carefully controlled.
The objective shifted from creating failures to scientifically validating system behavior.
Kubernetes has made Chaos Engineering even more valuable
Traditional infrastructures changed relatively slowly.
Kubernetes doesn’t.
Pods are constantly recreated.
Nodes join and leave clusters.
Containers restart.
Services scale automatically.
Infrastructure is dynamic by design.
That makes Chaos Engineering particularly useful.
Common experiments now include:
- removing Kubernetes worker nodes;
- restarting critical pods;
- blocking traffic between namespaces;
- degrading storage performance;
- simulating network partitions.
The goal is straightforward:
Verify that Kubernetes behaves exactly as expected under real failure conditions.
You no longer need to build your own platform
When Netflix pioneered Chaos Engineering, almost no tools existed.
Today, there are plenty.
AWS Fault Injection Service (AWS FIS)
AWS provides a fully managed service that allows engineers to inject failures into production or testing environments.
It can simulate:
- EC2 instance termination;
- network interruptions;
- CPU stress;
- storage degradation;
- ECS and EKS failures;
- database disruptions.
LitmusChaos
One of the most popular open-source Chaos Engineering frameworks for Kubernetes.
It includes dozens of ready-made experiments for cloud-native environments.
Chaos Mesh
Widely adopted in Kubernetes environments.
It supports fault injection for networking, CPU, memory, I/O, DNS, and processes.
Gremlin
A commercial Chaos Engineering platform used by many enterprise organizations to automate reliability testing at scale.
What should a systems administrator actually test?
You don’t need to simulate an entire data center outage on day one.
Start small.
Examples include:
| Hypothesis | Experiment |
|---|---|
| Pods survive node failures | Shut down a Kubernetes worker |
| Load balancer removes unhealthy servers | Terminate a backend VM |
| Application tolerates service latency | Inject 500 ms of network delay |
| Cluster maintains quorum | Temporarily disconnect a node |
| Disaster recovery works | Restore a complete backup in a test environment |
Every experiment should answer three questions:
- What do I expect to happen?
- Which metrics will confirm success?
- How will I stop the experiment if something unexpected occurs?
AI will make Chaos Engineering even more important
The next major evolution will come from autonomous infrastructure.
AI agents are increasingly making operational decisions involving:
- auto-scaling;
- workload placement;
- resource optimization;
- automated remediation;
- infrastructure tuning.
Soon, engineers won’t just need to validate infrastructure.
They’ll also need to validate how AI agents behave when facing incomplete, inconsistent, or unexpected information.
Chaos Engineering will likely evolve from testing distributed systems to testing autonomous decision-making itself.
The goal isn’t to break infrastructure
One misconception still persists.
Chaos Engineering isn’t about destroying systems.
It’s about building confidence.
Every failure discovered during a controlled experiment prevents a future production incident.
Every validated hypothesis reduces uncertainty.
Every weakness uncovered makes the platform more resilient.
Ultimately, the best infrastructure isn’t the one that never fails.
It’s the one that continues working when failure inevitably arrives.
Frequently Asked Questions
Is Chaos Engineering only useful for large enterprises?
No. Any organization running critical services, Kubernetes clusters, virtual machines, or distributed applications can benefit from controlled failure testing.
Is it safe to run Chaos Engineering experiments in production?
Yes—provided experiments are carefully designed, limited in scope, continuously monitored, and include automatic rollback or abort mechanisms.
Which open-source Chaos Engineering tools are the most widely used?
LitmusChaos and Chaos Mesh are among the most popular for Kubernetes environments, while AWS Fault Injection Service is commonly used on AWS infrastructure.
What skills should a systems administrator have before getting started?
A solid understanding of the infrastructure architecture, monitoring, disaster recovery procedures, and system observability is essential before introducing controlled failures.