Wombat

WAF-OPS-060 – Runbook & Operational Documentation Coverage

Säule: Excellence | Schweregrad: Mittel | Kategorie: Identity Access | Automatisierbar: High

Beschreibung

Every production workload MUST have a runbook covering all known failure modes, routine operational tasks, and escalation paths. Runbooks MUST be version-controlled, regularly reviewed (at minimum quarterly), and accessible to all on-call engineers without authentication barriers at 3 AM. Alert rules that trigger on-call pages MUST link to the relevant runbook section.

Begründung

Risiko Beschreibung

Risk

Runbooks not linked to alerts slow response to recurring disturbances.

Risk

Missing escalation instructions lead to uncoordinated ad-hoc reactions.

Anforderungen

  • Every production workload MUST have a runbook covering all known failure modes, routine operational tasks, and escalation paths.

  • Runbooks MUST be version-controlled, regularly reviewed (at minimum quarterly), and accessible to all on-call engineers without authentication barriers at 3 AM.

  • Alert rules that trigger on-call pages MUST link to the relevant runbook section.

Implementierungsleitfaden

  1. Document the requirement in a version-controlled policy or standard.

  2. Assign ownership and establish a review cadence (e.g., quarterly).

  3. Implement the required configuration as code in Terraform or equivalent IaC.

Reifegrade

Level Name Kriterien

L1

Ad hoc – control not consistently applied

* Manual evidence exists for some workloads, but coverage is unknown.

L2

Standardised – policy defined and communicated

* A version-controlled policy, ADR, or standard describes the requirement.

L3

Integrated – automated validation in CI/CD

* Terraform checks run automatically on pull requests and report results.

L4

Auditable – metrics and exceptions tracked

* Control coverage, failures, and approved exceptions are reported quarterly.

L5

Optimised – continuous improvement based on data

* Control effectiveness is measured and iteratively improved.

Checks

waf-ops-060.tf.aws.cloudwatch-alarm-runbook-annotation

Engine: terraform | Provider: aws | Automated: Ja

CloudWatch alarms must include runbook URL in description

All CloudWatch metric alarms that trigger on-call notifications MUST include a runbook URL in the alarm_description to reduce mean time to resolution.

Scope: resource — aws_cloudwatch_metric_alarm

Assertions: * alarm_description matches pattern https?:// — CloudWatch alarm description must include a runbook URL (http:// or https://).

Remediation: Include a runbook URL in the alarm_description field: alarm_description = "Error rate exceeded. Runbook: https://wiki/runbooks/service-errors"

Compliant
resource "aws_cloudwatch_metric_alarm" "error_rate" {
  alarm_name          = "payment-service-error-rate"
  comparison_operator = "GreaterThanThreshold"
  evaluation_periods  = 2
  metric_name         = "5XXError"
  namespace           = "AWS/ApplicationELB"
  period              = 60
  statistic           = "Sum"
  threshold           = 10
  alarm_description   = "Payment service error rate high. Runbook: https://wiki/runbooks/payment-5xx"
  alarm_actions       = [aws_sns_topic.oncall.arn]
}
Non-Compliant
resource "aws_cloudwatch_metric_alarm" "error_rate" {
  alarm_name          = "payment-service-error-rate"
  comparison_operator = "GreaterThanThreshold"
  evaluation_periods  = 2
  metric_name         = "5XXError"
  namespace           = "AWS/ApplicationELB"
  period              = 60
  statistic           = "Sum"
  threshold           = 10
  alarm_description   = "High error rate"
  # No runbook URL – WAF-OPS-060 Violation
  alarm_actions       = [aws_sns_topic.oncall.arn]
}

waf-ops-060.tf.aws.prometheus-alert-runbook-label

Engine: terraform | Provider: aws | Automated: Ja

Prometheus alert rules must include runbook_url annotation

Prometheus alert rules managed in AWS Managed Prometheus MUST include a runbook_url annotation so on-call engineers have immediate access to remediation steps.

Scope: resource — aws_prometheus_rule_group_namespace

Assertions: * data matches pattern runbook_url — Prometheus alert rules must include runbook_url annotation.

Remediation: Add runbook_url: "https://wiki/runbooks/<service>" annotation to all Prometheus alert rules.

Compliant
resource "aws_prometheus_rule_group_namespace" "alerts" {
  name         = "payment-service-alerts"
  workspace_id = aws_prometheus_workspace.main.id
  data         = <<-EOT
    groups:
      - name: payment-service
        rules:
          - alert: HighErrorRate
            expr: rate(http_requests_total{status=~"5.."}[5m]) > 0.1
            for: 2m
            annotations:
              summary: "High error rate on payment service"
              runbook_url: "https://wiki/runbooks/payment-errors"
  EOT
}
Non-Compliant
resource "aws_prometheus_rule_group_namespace" "alerts" {
  name         = "payment-service-alerts"
  workspace_id = aws_prometheus_workspace.main.id
  data         = <<-EOT
    groups:
      - name: payment-service
        rules:
          - alert: HighErrorRate
            expr: rate(http_requests_total{status=~"5.."}[5m]) > 0.1
            # No runbook_url annotation – WAF-OPS-060 Violation
  EOT
}

waf-ops-060.tf.azurerm.monitor-alert-runbook

Engine: terraform | Provider: azurerm | Automated: Ja

Azure Monitor alerts must include runbook or knowledge base links

Every azurerm_monitor_metric_alert that pages on-call MUST include a runbook or knowledge base URL in its description or action webhook payload so responders can act immediately.

Scope: resource — azurerm_monitor_metric_alert

Assertions: * description matches pattern https?:// — Azure Monitor alert description must include a runbook URL.

Remediation: Add a runbook URL to the description field of every azurerm_monitor_metric_alert. Use action groups with webhook payloads that include the runbook link and alert context.

Compliant
resource "azurerm_monitor_metric_alert" "errors" {
  name                = "payment-service-errors"
  resource_group_name = azurerm_resource_group.main.name
  scopes              = [azurerm_application_insights.app.id]

  criteria {
    metric_namespace = "Microsoft.Insights/components"
    metric_name      = "requests/failed"
    aggregation      = "Count"
    operator         = "GreaterThan"
    threshold        = 10
  }

  description = "High failed request rate. Runbook: https://wiki/runbooks/payment-errors"

  action {
    action_group_id = azurerm_monitor_action_group.oncall.id
  }
}
Non-Compliant
resource "azurerm_monitor_metric_alert" "errors" {
  name                = "payment-service-errors"
  resource_group_name = azurerm_resource_group.main.name
  scopes              = [azurerm_application_insights.app.id]
  description       = "High failed request rate"
  # No runbook link provided
}

waf-ops-060.tf.google.monitoring-alert-runbook

Engine: terraform | Provider: google | Automated: Ja

GCP Monitoring alert policies must include runbook or documentation links

Every google_monitoring_alert_policy that triggers on-call notifications MUST include a runbook or knowledge base URL in its documentation block so responders have immediate remediation guidance.

Scope: resource — google_monitoring_alert_policy

Assertions: * documentation attribute exists — GCP alert policy must include a documentation block with runbook link.

Remediation: Set the documentation field of every google_monitoring_alert_policy to include alert context and a runbook URL. Keep documentation under version control alongside the Terraform configuration.

Compliant
resource "google_monitoring_alert_policy" "errors" {
  display_name = "Payment Service High Error Rate"
  documentation {
    content = "High 5xx error rate on payment service. Runbook: https://wiki/runbooks/payment-errors"
    mime_type = "text/markdown"
  }

  conditions {
    display_name = "5xx rate above threshold"
    condition_threshold {
      filter          = "metric.type=\"logging.googleapis.com/user/payment_service_5xx\""
      duration        = "300s"
      comparison      = "COMPARISON_GT"
      threshold_value = 10
    }
  }

  notification_channels = [google_monitoring_notification_channel.oncall.id]
}
Non-Compliant
resource "google_monitoring_alert_policy" "errors" {
  display_name = "Payment Service High Error Rate"
  # No documentation block with runbook link

  conditions {
    display_name = "5xx rate above threshold"
    condition_threshold {
      filter     = "metric.type=\"logging.googleapis.com/user/payment_service_5xx\""
      duration   = "300s"
      comparison = "COMPARISON_GT"
    }
  }
}

waf-ops-060.tf.stackit.runbook-defined

Engine: terraform | Provider: stackit | Automated: Ja

Every production workload for stackit

Every production workload must be documented in IaC via a runbook-defined tag/label on stackit_project. This validates the control is tracked as code.

Scope: resource — stackit_project

Assertions: * labels key exists — stackit_project is missing the runbook-defined tag/label.

Remediation: Add a runbook-defined tag/label to the stackit_project resource to show the stackit implementation of WAF-OPS-060 is tracked in IaC.

Compliant
resource "stackit_project" "example" {
  labels = {
    runbook-defined = "true"
  }
}
Non-Compliant
resource "stackit_project" "example" {
  # Missing runbook-defined labels
}

waf-ops-060.tf.ovh.runbook-defined

Engine: terraform | Provider: ovh | Automated: Ja

Every production workload for ovh

Every production workload must be documented in IaC via a runbook-defined tag/label on ovh_cloud_project. This validates the control is tracked as code.

Scope: resource — ovh_cloud_project

Assertions: * tags key exists — ovh_cloud_project is missing the runbook-defined tag/label.

Remediation: Add a runbook-defined tag/label to the ovh_cloud_project resource to show the ovh implementation of WAF-OPS-060 is tracked in IaC.

Compliant
resource "ovh_cloud_project" "example" {
  tags = {
    runbook-defined = "true"
  }
}
Non-Compliant
resource "ovh_cloud_project" "example" {
  # Missing runbook-defined tags
}

waf-ops-060.tf.hcloud.runbook-defined

Engine: terraform | Provider: hcloud | Automated: Ja

Every production workload for hcloud

Every production workload must be documented in IaC via a runbook-defined tag/label on hcloud_server. This validates the control is tracked as code.

Scope: resource — hcloud_server

Assertions: * labels key exists — hcloud_server is missing the runbook-defined tag/label.

Remediation: Add a runbook-defined tag/label to the hcloud_server resource to show the hcloud implementation of WAF-OPS-060 is tracked in IaC.

Compliant
resource "hcloud_server" "example" {
  labels = {
    runbook-defined = "true"
  }
}
Non-Compliant
resource "hcloud_server" "example" {
  # Missing runbook-defined labels
}

Evidenz

Art Erforderlich Beschreibung

iac

Erforderlich

Terraform configuration implementing the control.

governance

Erforderlich

Policy, ADR, or standard documenting the requirement.

config

Optional

Cloud console or API output confirming runtime state.

Verwandte Controls

Framework Controls

ISO/IEC 20000-1

* Knowledge management * Incident management

ITIL 4

* Knowledge management * Incident management

SOC 2 Type II

* CC9.1 – Risk Mitigation Activities