WAF-OPS-070 – Post-Incident Review Process

Säule: Excellence | Schweregrad: Mittel | Kategorie: Data Governance | Automatisierbar: High

Beschreibung

Every production incident with user-visible impact or SLO violation MUST trigger a post-incident review (postmortem) within 5 business days. Postmortems MUST be blameless, structured, and result in documented action items with owners and due dates tracked through completion. Postmortem findings MUST be shared across teams to prevent recurrence of the same class of incidents.

Begründung

Risiko	Beschreibung
Risk	Undocumented incidents prevent organizational learning and recurrence prevention.
Risk	Missing post-mortems leave systemic weaknesses unrecognized and uncorrected.

Risiko

Beschreibung

Risk

Undocumented incidents prevent organizational learning and recurrence prevention.

Risk

Missing post-mortems leave systemic weaknesses unrecognized and uncorrected.

Anforderungen

Every production incident with user-visible impact or SLO violation MUST trigger a post-incident review (postmortem) within 5 business days.
Postmortems MUST be blameless, structured, and result in documented action items with owners and due dates tracked through completion.
Postmortem findings MUST be shared across teams to prevent recurrence of the same class of incidents.

Implementierungsleitfaden

Document the requirement in a version-controlled policy or standard.
Assign ownership and establish a review cadence (e.g., quarterly).
Implement the required configuration as code in Terraform or equivalent IaC.

Reifegrade

Level	Name	Kriterien
L1	Ad hoc – control not consistently applied	* Manual evidence exists for some workloads, but coverage is unknown.
L2	Standardised – policy defined and communicated	* A version-controlled policy, ADR, or standard describes the requirement.
L3	Integrated – automated validation in CI/CD	* Terraform checks run automatically on pull requests and report results.
L4	Auditable – metrics and exceptions tracked	* Control coverage, failures, and approved exceptions are reported quarterly.
L5	Optimised – continuous improvement based on data	* Control effectiveness is measured and iteratively improved.

Level

Name

Kriterien

Ad hoc – control not consistently applied

* Manual evidence exists for some workloads, but coverage is unknown.

Standardised – policy defined and communicated

* A version-controlled policy, ADR, or standard describes the requirement.

Integrated – automated validation in CI/CD

* Terraform checks run automatically on pull requests and report results.

Auditable – metrics and exceptions tracked

* Control coverage, failures, and approved exceptions are reported quarterly.

Optimised – continuous improvement based on data

* Control effectiveness is measured and iteratively improved.

Checks

waf-ops-070.tf.aws.incident-management-sns-topic

Engine: terraform | Provider: aws | Automated: Ja

SNS incident notification topic must be configured for production alarms

An SNS topic for incident notifications MUST exist and MUST be referenced by production CloudWatch alarms. This ensures incidents are captured and tracked.

Scope: resource — aws_sns_topic

Assertions: * name not empty — SNS topic must have a non-empty name.

Remediation: Create an aws_sns_topic for incident notifications and reference it in all production CloudWatch alarm alarm_actions.

Compliant

resource "aws_sns_topic" "incidents" {
  name         = "production-incident-notifications"
  display_name = "Production Incidents"
}

resource "aws_sns_topic_subscription" "pagerduty" {
  topic_arn = aws_sns_topic.incidents.arn
  protocol  = "https"
  endpoint  = var.pagerduty_endpoint
}

resource "aws_cloudwatch_metric_alarm" "critical_error" {
  alarm_name    = "payment-service-critical"
  alarm_actions = [aws_sns_topic.incidents.arn]
  /* ... */
}

Non-Compliant

resource "aws_cloudwatch_metric_alarm" "critical_error" {
  alarm_name = "payment-service-critical"
  # No alarm_actions – incidents not captured
  # WAF-OPS-070 Violation
}

waf-ops-070.tf.azurerm.action-group-incident

Engine: terraform | Provider: azurerm | Automated: Ja

Azure Monitor action group must be configured for incident routing

Azure Monitor action groups for production incident routing MUST include at least one receiver (email, webhook, or PagerDuty) to ensure incidents are captured and reviewed.

Scope: resource — azurerm_monitor_action_group

Assertions: * name not empty — Action group must have a non-empty name. * short_name not empty — Action group must have a short_name for SMS notifications.

Remediation: Define an azurerm_monitor_action_group with at least one receiver type (email_receiver, webhook_receiver, or azure_app_push_receiver).

Compliant

resource "azurerm_monitor_action_group" "oncall" {
  name                = "production-oncall"
  resource_group_name = azurerm_resource_group.main.name
  short_name          = "oncall"

  email_receiver {
    name          = "oncall-engineer"
    email_address = var.oncall_email
  }

  webhook_receiver {
    name        = "pagerduty"
    service_uri = var.pagerduty_webhook_url
  }
}

Non-Compliant

resource "azurerm_monitor_action_group" "oncall" {
  name                = "production-oncall"
  resource_group_name = azurerm_resource_group.main.name
  short_name          = "oncall"
  # No receivers configured – incidents not routed
  # WAF-OPS-070 Violation
}

waf-ops-070.tf.google.monitoring-notification-channel

Engine: terraform | Provider: google | Automated: Ja

GCP notification channel must route production alerts to on-call

Every production google_monitoring_alert_policy MUST reference at least one google_monitoring_notification_channel so that incidents with user-visible impact or SLO violations are captured and routed to on-call engineers.

Scope: resource — google_monitoring_alert_policy

Assertions: * notification_channels attribute exists — GCP alert policy must route to at least one notification channel.

Remediation: Create google_monitoring_notification_channel resources for email, PagerDuty, or Slack and reference them in google_monitoring_alert_policy notification_channels.

Compliant

resource "google_monitoring_notification_channel" "oncall" {
  display_name = "On-call PagerDuty"
  type         = "pagerduty"
  labels = {
    service_key = var.pagerduty_service_key
  }
}

resource "google_monitoring_alert_policy" "errors" {
  display_name = "Payment Service High Error Rate"
  documentation {
    content   = "Runbook: https://wiki/runbooks/payment-errors"
    mime_type = "text/markdown"
  }
  notification_channels = [google_monitoring_notification_channel.oncall.id]

  conditions {
    display_name = "5xx rate above threshold"
    condition_threshold {
      filter          = "metric.type=\"logging.googleapis.com/user/payment_service_5xx\""
      duration        = "300s"
      comparison      = "COMPARISON_GT"
      threshold_value = 10
    }
  }
}

Non-Compliant

resource "google_monitoring_alert_policy" "errors" {
  display_name = "Payment Service High Error Rate"
  # No notification_channels – incident never reaches on-call
}

waf-ops-070.tf.stackit.incident-response-tag

Engine: terraform | Provider: stackit | Automated: Ja

STACKIT workloads must have an incident response tag

STACKIT workloads must have an incident response tag. This check validates the Terraform configuration uses the expected resource and attribute.

Scope: resource — stackit_ske_cluster, stackit_server

Assertions: * labels.incident-runbook key exists — STACKIT workload is missing the 'incident-runbook' label.

Remediation: Review the Terraform configuration for the listed resource types and ensure the required attribute or tag is present. See the control narrative for provider-specific guidance.

Compliant

# Example resource meets the requirement.
resource "stackit_ske_cluster" "example" {
  # required attribute/tag/label present
}

Non-Compliant

# Example resource does not meet the requirement.
resource "stackit_ske_cluster" "example" {
  # Missing required attribute/tag/label
}

waf-ops-070.tf.ovh.incident-response-tag

Engine: terraform | Provider: ovh | Automated: Ja

OVHcloud workloads must have an incident response tag

OVHcloud workloads must have an incident response tag. This check validates the Terraform configuration uses the expected resource and attribute.

Scope: resource — ovh_cloud_project_kube, ovh_cloud_project_instance

Assertions: * tags.incident-runbook key exists — OVHcloud workload is missing the 'incident-runbook' tag.

Remediation: Review the Terraform configuration for the listed resource types and ensure the required attribute or tag is present. See the control narrative for provider-specific guidance.

Compliant

# Example resource meets the requirement.
resource "ovh_cloud_project_kube" "example" {
  # required attribute/tag/label present
}

Non-Compliant

# Example resource does not meet the requirement.
resource "ovh_cloud_project_kube" "example" {
  # Missing required attribute/tag/label
}

waf-ops-070.tf.hcloud.incident-response-tag

Engine: terraform | Provider: hcloud | Automated: Ja

Hetzner Cloud workloads must have an incident response tag

Hetzner Cloud workloads must have an incident response tag. This check validates the Terraform configuration uses the expected resource and attribute.

Scope: resource — hcloud_server

Assertions: * labels.incident-runbook key exists — Hetzner Cloud server is missing the 'incident-runbook' label.

Remediation: Review the Terraform configuration for the listed resource types and ensure the required attribute or tag is present. See the control narrative for provider-specific guidance.

Compliant

# Example resource meets the requirement.
resource "hcloud_server" "example" {
  # required attribute/tag/label present
}

Non-Compliant

# Example resource does not meet the requirement.
resource "hcloud_server" "example" {
  # Missing required attribute/tag/label
}

Evidenz

Art Erforderlich Beschreibung

Art	Erforderlich	Beschreibung
`iac`	Erforderlich	Terraform configuration implementing the control.
`governance`	Erforderlich	Policy, ADR, or standard documenting the requirement.
`config`	Optional	Cloud console or API output confirming runtime state.

iac

Erforderlich

Terraform configuration implementing the control.

governance

Erforderlich

Policy, ADR, or standard documenting the requirement.

config

Optional

Cloud console or API output confirming runtime state.

WAF-OPS-070 – Post-Incident Review Process

Beschreibung

Begründung

Anforderungen

Implementierungsleitfaden

Reifegrade

Checks

waf-ops-070.tf.aws.incident-management-sns-topic

waf-ops-070.tf.azurerm.action-group-incident

waf-ops-070.tf.google.monitoring-notification-channel

waf-ops-070.tf.stackit.incident-response-tag

waf-ops-070.tf.ovh.incident-response-tag

waf-ops-070.tf.hcloud.incident-response-tag

Evidenz

Verwandte Controls