WAF-REL-090 – Chaos Engineering & Fault Injection

Säule: Reliability | Schweregrad: Mittel | Kategorie: Logging Monitoring | Automatisierbar: High

Beschreibung

Production and staging workloads MUST undergo structured chaos engineering experiments at minimum quarterly. Experiments MUST be based on hypothesis-driven testing against defined failure modes. Results MUST be documented and deviations from expected behavior MUST trigger reliability improvements. Chaos experiments MUST be run in staging before production. Production chaos MUST have defined blast radius limits, rollback procedures, and active monitoring during execution. Ad-hoc or unsanctioned "accidental chaos" is not a substitute.

Begründung

Risiko	Beschreibung
Risk	Untested workloads surprise teams during real outages because known weaknesses were never exercised.
Risk	Missing chaos experiments prevent targeted discovery of resilience gaps.

Risiko

Beschreibung

Risk

Untested workloads surprise teams during real outages because known weaknesses were never exercised.

Risk

Missing chaos experiments prevent targeted discovery of resilience gaps.

Anforderungen

Production and staging workloads MUST undergo structured chaos engineering experiments at minimum quarterly.
Experiments MUST be based on hypothesis-driven testing against defined failure modes.
Results MUST be documented and deviations from expected behavior MUST trigger reliability improvements.
Chaos experiments MUST be run in staging before production.
Production chaos MUST have defined blast radius limits, rollback procedures, and active monitoring during execution.

Implementierungsleitfaden

Document the requirement in a version-controlled policy or standard.
Assign ownership and establish a review cadence (e.g., quarterly).

Reifegrade

Level	Name	Kriterien
L1	Ad hoc – control not consistently applied	* Manual evidence exists for some workloads, but coverage is unknown.
L2	Standardised – policy defined and communicated	* A version-controlled policy, ADR, or standard describes the requirement.
L3	Integrated – automated validation in CI/CD	* Terraform checks run automatically on pull requests and report results.
L4	Auditable – metrics and exceptions tracked	* Control coverage, failures, and approved exceptions are reported quarterly.
L5	Optimised – continuous improvement based on data	* Control effectiveness is measured and iteratively improved.

Level

Name

Kriterien

Ad hoc – control not consistently applied

* Manual evidence exists for some workloads, but coverage is unknown.

Standardised – policy defined and communicated

* A version-controlled policy, ADR, or standard describes the requirement.

Integrated – automated validation in CI/CD

* Terraform checks run automatically on pull requests and report results.

Auditable – metrics and exceptions tracked

* Control coverage, failures, and approved exceptions are reported quarterly.

Optimised – continuous improvement based on data

* Control effectiveness is measured and iteratively improved.

Checks

waf-rel-090.tf.aws.fis-experiment-template

Engine: terraform | Provider: aws | Automated: Ja

AWS FIS experiment templates must define stop conditions

AWS Fault Injection Simulator experiment templates must define stop_condition blocks to automatically abort experiments when error budget thresholds are exceeded.

Scope: resource — aws_fis_experiment_template

Assertions: * description not empty — FIS experiment must have a descriptive description with hypothesis. * stop_condition attribute exists — FIS experiment must define at least one stop_condition. * role_arn not empty — FIS experiment must specify an IAM role ARN.

Remediation: Add a stop_condition block to the aws_fis_experiment_template referencing a CloudWatch alarm that triggers when error budget is consumed.

Compliant

resource "aws_fis_experiment_template" "az_failure" {
  description = "Hypothesis: Service recovers from AZ failure within 60s without SLO violation"
  role_arn    = aws_iam_role.fis.arn

  stop_condition {
    source = "aws:cloudwatch:alarm"
    value  = aws_cloudwatch_metric_alarm.slo_burn_rate.arn
  }

  action {
    name      = "terminate-instances-az1"
    action_id = "aws:ec2:terminate-instances"
    target {
      key   = "Instances"
      value = "az1-instances"
    }
  }

  target {
    name           = "az1-instances"
    resource_type  = "aws:ec2:instance"
    selection_mode = "PERCENT(25)"
    resource_tag {
      key   = "availability-zone"
      value = "eu-west-1a"
    }
  }
}

Non-Compliant

resource "aws_fis_experiment_template" "az_failure" {
  description = "AZ failure test"
  role_arn    = aws_iam_role.fis.arn
  # Missing: stop_condition – experiment has no abort mechanism
  # WAF-REL-090 violation
  action {
    name      = "terminate-instances"
    action_id = "aws:ec2:terminate-instances"
    target {
      key   = "Instances"
      value = "all-instances"
    }
  }
  target {
    name           = "all-instances"
    resource_type  = "aws:ec2:instance"
    selection_mode = "ALL"
  }
}

waf-rel-090.tf.azurerm.chaos-studio-experiment

Engine: terraform | Provider: azurerm | Automated: Ja

Azure Chaos Studio experiments must target non-production environments first

Azure Chaos Studio experiments must have a name and location configured and should target resources tagged with the correct environment.

Scope: resource — azurerm_chaos_studio_experiment

Assertions: * name not empty — Chaos Studio experiment must have a descriptive name. * location not empty — Chaos Studio experiment must specify a location. * identity attribute exists — Chaos Studio experiment must define an identity for execution.

Remediation: Configure azurerm_chaos_studio_experiment with name, resource_group_name, location, and identity. Define the experiment steps in the schema_version block.

Compliant

resource "azurerm_chaos_studio_experiment" "pod_failure" {
  name                = "payment-api-pod-failure-staging"
  location            = azurerm_resource_group.staging.location
  resource_group_name = azurerm_resource_group.staging.name

  identity {
    type = "SystemAssigned"
  }

  schema_version = "2.0"
  steps {
    name = "pod-failure-step"
    branches {
      name = "main-branch"
      actions {
        action_urn = "urn:csci:microsoft:azureKubernetesServiceChaosMesh:podChaos/2.2"
        duration   = "PT5M"
      }
    }
  }
}

Non-Compliant

# No chaos experiments configured – WAF-REL-090 violation
# Resilience claims not validated by fault injection

waf-rel-090.tf.google.fault-injection-test

Engine: terraform | Provider: google | Automated: Ja

GCP Cloud Run services should have fault injection testing configured

GCP workloads should use Traffic Director or Cloud Service Mesh fault injection to validate resilience patterns. Alternatively, load testing pipelines should include fault scenarios.

Scope: resource — google_compute_url_map

Assertions: * name not empty — URL map must have a name. * default_route_action attribute exists — URL map should define default_route_action for fault injection testing.

Remediation: Configure fault_injection_policy in the default_route_action of the URL map to inject delays or abort responses for chaos testing.

Compliant

resource "google_compute_url_map" "api" {
  name = "payment-api-url-map"

  default_route_action {
    fault_injection_policy {
      delay {
        fixed_delay {
          seconds = 5
          nanos   = 0
        }
        percentage = 10  # 10% of requests get 5s delay
      }
      abort {
        http_status = 503
        percentage  = 5  # 5% abort for chaos testing
      }
    }
    weighted_backend_services {
      backend_service = google_compute_backend_service.api.id
      weight          = 1000
    }
  }
}

Non-Compliant

resource "google_compute_url_map" "api" {
  name            = "payment-api-url-map"
  default_service = google_compute_backend_service.api.id
  # No fault injection configured – resilience not testable
  # WAF-REL-090 violation
}

waf-rel-090.tf.stackit.rollback-tested

Engine: terraform | Provider: stackit | Automated: Ja

Production and staging workloads for stackit

Production and staging workloads must be documented in IaC via a rollback-tested tag/label on stackit_project. This validates the control is tracked as code.

Scope: resource — stackit_project

Assertions: * labels key exists — stackit_project is missing the rollback-tested tag/label.

Remediation: Add a rollback-tested tag/label to the stackit_project resource to show the stackit implementation of WAF-REL-090 is tracked in IaC.

Compliant

resource "stackit_project" "example" {
  labels = {
    rollback-tested = "true"
  }
}

Non-Compliant

resource "stackit_project" "example" {
  # Missing rollback-tested labels
}

waf-rel-090.tf.ovh.rollback-tested

Engine: terraform | Provider: ovh | Automated: Ja

Production and staging workloads for ovh

Production and staging workloads must be documented in IaC via a rollback-tested tag/label on ovh_cloud_project. This validates the control is tracked as code.

Scope: resource — ovh_cloud_project

Assertions: * tags key exists — ovh_cloud_project is missing the rollback-tested tag/label.

Remediation: Add a rollback-tested tag/label to the ovh_cloud_project resource to show the ovh implementation of WAF-REL-090 is tracked in IaC.

Compliant

resource "ovh_cloud_project" "example" {
  tags = {
    rollback-tested = "true"
  }
}

Non-Compliant

resource "ovh_cloud_project" "example" {
  # Missing rollback-tested tags
}

waf-rel-090.tf.hcloud.rollback-tested

Engine: terraform | Provider: hcloud | Automated: Ja

Production and staging workloads for hcloud

Production and staging workloads must be documented in IaC via a rollback-tested tag/label on hcloud_server. This validates the control is tracked as code.

Scope: resource — hcloud_server

Assertions: * labels key exists — hcloud_server is missing the rollback-tested tag/label.

Remediation: Add a rollback-tested tag/label to the hcloud_server resource to show the hcloud implementation of WAF-REL-090 is tracked in IaC.

Compliant

resource "hcloud_server" "example" {
  labels = {
    rollback-tested = "true"
  }
}

Non-Compliant

resource "hcloud_server" "example" {
  # Missing rollback-tested labels
}

Evidenz

Art Erforderlich Beschreibung

Art	Erforderlich	Beschreibung
`iac`	Erforderlich	Terraform configuration implementing the control.
`governance`	Erforderlich	Policy, ADR, or standard documenting the requirement.
`config`	Optional	Cloud console or API output confirming runtime state.

iac

Erforderlich

Terraform configuration implementing the control.

governance

Erforderlich

Policy, ADR, or standard documenting the requirement.

config

Optional

Cloud console or API output confirming runtime state.

WAF-REL-090 – Chaos Engineering & Fault Injection

Beschreibung

Begründung

Anforderungen

Implementierungsleitfaden

Reifegrade

Checks

waf-rel-090.tf.aws.fis-experiment-template

waf-rel-090.tf.azurerm.chaos-studio-experiment

waf-rel-090.tf.google.fault-injection-test

waf-rel-090.tf.stackit.rollback-tested

waf-rel-090.tf.ovh.rollback-tested

waf-rel-090.tf.hcloud.rollback-tested

Evidenz

Verwandte Controls