A timed out purchase can still become an order. Design a bounded staging experiment to check what your service preserves when a dependency fails.
A checkout request times out, the customer tries again, and the first order appears later. Your service has recovered its connection. Has it also created a duplicate purchase? Chaos engineering gives a team a controlled way to investigate questions like this before an outage forces the answer.
The scenarios, thresholds, and expected outcomes here are hypothetical experiment designs, not measured results. Start in an isolated staging environment with synthetic accounts and a dependency simulator. The goal is to design and interpret a first experiment without injecting faults into production.
Start with a claim you can disprove
Connect steady state to a customer outcome
The Principles of Chaos Engineering describe experiments around measurable steady state, a hypothesis, and realistic disruptions. The principles favor production experiments as an advanced practice because environment and traffic affect behavior. Staging is a useful place to learn the method and rehearse containment. Passing there does not establish production resilience.
Choose a claim with an observable consequence. For an order service, propose that a brief loss of the delivery estimate dependency leaves order submission available, clearly marks the estimate as unavailable, and creates one order per accepted purchase intent. Define the permitted response time and recovery deadline before choosing an injection tool.
Steady state is the normal range of those outputs under a stated workload. Record successful submissions alongside offered requests, latency including failures, and durable order counts. Give each metric a denominator and observation window. CPU and connection counts help explain system behavior, but a healthy CPU graph cannot prove that customers received their orders.
Keep the comparison interpretable
Observe the unchanged staging build under the intended workload before applying a fault. Keep the request mix and dataset stable during the experiment. Where practical, use a comparable unaffected cohort to help distinguish fault effects from shared environmental changes. A cohort sharing the same exhausted database may no longer be an unaffected control.
Avoid a deployment or scaling change during this first run. A shifting baseline makes attribution difficult. If the baseline already breaches its acceptance criteria, investigate that problem before introducing another variable. Record sample counts so a quiet minute with no failures does not become a claim about reliability.
Choose chaos engineering tools for your failure model
Match the injection boundary to the question
A chaos engineering tool needs to reproduce the failure your hypothesis names. For a distributed system of microservices, map the request path and shared dependencies before selecting an action. Candidate failure scenarios include a slow response, an unavailable dependency, or a response lost after a write. Each exercises different recovery behavior.
Start with a scoped simulator when the question concerns one dependency's response. It can simulate latency for synthetic requests while preserving the caller's real timeout handling. Record what it cannot represent, such as network routing failures or provider internals. This scope makes the first experiment easier to interpret without claiming complete system reliability.
Use AWS FIS when the question requires a supported AWS resource action and its targeting controls. Inspect the action's effect and cleanup before choosing it. Azure Chaos Studio has its own resource models and supported actions. Verify the model's current status and scope before adopting a template. A cloud tool's default scenario can affect more resources than your hypothesis requires.
Review the plan with the service owner and the DevOps or site reliability engineering team responsible for recovery. Rehearse manually before deciding what to automate. Reliable targeting, usable monitoring tools, and a tested stop path are prerequisites for repeatable chaos testing. Automation should preserve those controls and the evidence needed to explain a failed run.
Build a controlled chaos experiment
Plan the fault and its recovery together
Choose a bounded failure
Select one caller and one dependency boundary for fault injection. In the order example, a dedicated staging simulator can delay delivery estimate responses for a fixed interval. The application should still execute its real timeout and fallback code. A mock that returns the fallback directly cannot establish whether that code works.
Specify the eligible resources, maximum affected count, duration, and intensity. Blast radius includes indirect effects through shared queues, databases, or identity services. Confirm that staging cannot send real purchases, messages, or billing requests. A staging label offers little containment when its credentials still reach a live provider.
For AWS, the FIS target documentation distinguishes explicit resources, selection criteria, and selection modes. A count of one limits the number selected from the eligible population. It does not make an overly broad population safe. Review actual resource identities and the dependencies they share before accepting that scope.
Define a safe stop
Write abort conditions independently of the hypothesis. The experiment might allow missing delivery estimates while requiring an immediate stop for duplicate orders, an expanding queue, an unexpected target, or lost monitoring. Assign an operator who can stop the run and another person who watches application outcomes. Both need access that remains usable during the fault.
In AWS FIS stop conditions, CloudWatch alarms define thresholds that stop an experiment when triggered. Attach an alarm for unacceptable application impact. Include a fixed duration and a manual stop path as separate controls. A duration bounds intended injection time; it does not bound every resulting side effect.
Rehearse the stop path in the isolated environment before the real test window. Establish what the operator does if telemetry disappears. Specify which signal, evaluation window, and owner authorize starting again. After an abort, review the evidence and recovery before scheduling another run.
Verify recovery
Define an observation period after the fault ends. Check that new submissions succeed, delayed work drains within the agreed deadline, and orders retain their intended contents. Follow requests accepted during the interruption through to durable outcomes. A successful request after the interruption says nothing about an earlier request left pending.
Keep separate timestamps for requesting the stop, confirming fault removal, restoring application behavior, and finishing reconciliation. Record any manual intervention. If an operator clears a queue to restore responsiveness, the report must account for its missing work. Recovery that required intervention differs from the autonomous recovery in the hypothesis.
Make the controls independent of the failure
Check targeting and permissions before injection
Use a dedicated experiment identity with narrowly scoped permissions. The FIS experiment role guide recommends least privilege through specific resources or tags. Review both the role and template. A correctly scoped template can later change while an unnecessarily broad role continues to permit destructive actions.
AWS documents a target preview through the actions mode in FIS experiment options. Preview skips actions and helps inspect resolved targets and logging. It does not verify permission to execute fault actions, and later resource changes or random selection can change actual targets. Record the previewed resource identities and require a separate permission review for the chosen action.
For broader emergency containment, FIS safety levers stop running experiments and prevent new ones within an AWS account and Region. Their scope can include other teams' experiments. Coordinate ownership and access before using that control in a shared account. Prevent an unattended schedule from restarting a fault while recovery is underway.
Budget for alarm delay and missing data
An abort threshold needs time to observe a breach and act. CloudWatch alarm documentation describes evaluation over periods and configurable treatment of missing data. Review the metric dimensions and publishing interval with the evaluation window. An alarm watching the wrong environment can remain healthy throughout the experiment.
Choose a fault duration and margin that account for detection and stopping time. Do not assume a brief injection will always end before damage occurs. Rehearse alarm behavior with safe synthetic signals, and verify that the stop actually follows. If the experiment can break its own monitoring path, require another observation path or postpone it.
Separate stopping cleanup and rollback
Understand what the selected action reverses
Stopping ends the experiment's activity. Cleanup removes temporary injection configuration or resources. Rollback returns an application deployment or configuration to a previous version. None of those statements establishes that every business transaction is correct. Reconciliation accounts for work already accepted and any durable effects.
The FIS stopping guide says unfinished post actions complete before the experiment stops. Inspect the selected action's documented behavior and confirm its result. A completed stop request cannot undo an email already sent or repair an order duplicated while a caller was retrying.
Azure's Chaos Studio Workspaces scenario documentation illustrates why action semantics matter. Its Service Bus topic state action is discrete and does not revert itself; reactivation requires a later action. Some other scenarios restore their injected configuration automatically. Check the exact resource model and action, since cleanup guarantees differ. Workspaces and Scenarios are documented as public preview and not intended for production use.
Write the cleanup procedure during planning. Name the temporary configuration to remove and the application state to inspect afterward. Give any deployment rollback its own trigger, such as a regression found during diagnosis. Changing a release while a fault remains active adds another variable and can leave the injection in place.
Examine retries and durable side effects
Count attempts alongside completed work
A retry can hide a brief failure from the caller while increasing work at the dependency. AWS explains amplification and delayed recovery in Timeouts retries and backoff with jitter. Inspect every retrying layer, including the browser, service client, and queue consumer. Their budgets can multiply even when each looks reasonable alone.
For the staging experiment, record dependency attempts per logical submission, concurrency, and queued work. Require retries to stay within a stated attempt budget and the caller's overall deadline. Observe whether backoff spreads attempts and whether they stop when the request is no longer useful. An unchanged success rate alongside growing resource consumption warrants investigation.
Use the existing API rate limit testing guide to develop assertions around bounded retries. Keep offered traffic fixed for this experiment so increased dependency traffic can be attributed to application behavior. Capacity exploration belongs in a separate workload design, as discussed in the k6 load testing guide.
Verify the ambiguous completion case
The order example needs a later experiment for a different failure boundary. Let an isolated order dependency commit a synthetic order, then delay its response beyond the caller's timeout. The caller cannot infer from that timeout whether the write happened. Repeating the purchase with a new identity can create another valid order.
AWS's guidance on idempotent mutating operations describes reusing a token across retries and coordinating token storage with the mutation. Define the token's scope, retention window, and handling of changed payloads. A token header alone does not implement the guarantee.
Check the persisted order count and relevant downstream events through independent reads. Confirm that the same logical request retains its identity across retries and that concurrent attempts cannot bypass duplicate handling. Keep this experiment separate from delivery estimate latency so a failure has a clear boundary to investigate.
Interpret evidence before expanding scope
Distinguish resilience from an ineffective fault
First establish that the intended fault reached the intended request path. Correlate the injection interval with dependency observations and application requests. Cached responses or a selector matching the wrong workload can produce an uneventful graph without challenging the hypothesis. Classify that run as inconclusive.
Compare outcomes with the baseline and the unaffected cohort where available. Separate expected estimate failures from unexpected submission failures. Include affected sample counts and the complete recovery interval. A single successful run supports only the tested build, workload, environment, and fault. Record staging differences, such as smaller pools or substituted providers, beside the conclusion.
Turn a discovered weakness into a retest
Preserve the build revision, sanitized experiment definition, resolved targets, and a common timeline. Note the abort reason and any intervention. Give each finding an owner and a concrete requirement. A retry storm might lead to a shared retry budget; a duplicate order needs a durable correctness fix and regression coverage.
Repeat the same bounded experiment after the fix with comparable conditions. Change one dimension in subsequent experiments, such as duration or affected count, only after reviewing the previous evidence. Production experimentation requires its own operational design and authorization beyond the staging evidence established here.
Check whether people understand degraded behavior
After engineering verifies containment and recovery, create a BetaRoast feedback round using a safe hosted build with a controlled degraded response. Ask participants whether they understand that the estimate is unavailable, whether the order succeeded, and what action they would take next. Use synthetic data and explain the task.
BetaRoast collects human feedback. It does not inject infrastructure faults or certify resilience. An unclear pending message can still encourage repeated submission even when backend protections work. Convert that observation into a clearer status requirement and test it alongside the technical recovery path.
Frequently asked questions
How is chaos engineering different from load testing?
Load testing varies demand to examine behavior under a workload. A chaos experiment challenges a stated resilience hypothesis with a controlled disruption. Demand can be part of either design. For the first experiment, keeping it stable makes fault effects easier to interpret.
Can a staging experiment prove production resilience?
No. It provides evidence about the tested staging conditions and helps rehearse controls. Production traffic, topology, and dependency behavior can differ. The Principles of Chaos favor production experimentation as an advanced practice, which requires additional containment and operational preparation.
Does stopping an experiment roll back its effects?
Only effects explicitly reversed by the action's cleanup are covered. Confirm fault removal and application recovery separately. Durable writes and messages need reconciliation, and a deployment rollback has its own purpose and trigger.
What should happen when an abort condition fires?
Stop injection through the rehearsed control, confirm its effect, and follow the recovery procedure. Preserve the timeline and account for pending work. Review the breached condition before another run, even when the application appears healthy again.
Define steady state through customer outcomes, limit the fault to an isolated staging boundary, and rehearse independent stop controls. Verify fault removal, application recovery, and durable state separately. Use each result to refine the hypothesis and regression tests while keeping the limits of staging evidence explicit.