WAF-OPS-040 – Alerting on Symptoms, Not Causes

Säule: Excellence | Schweregrad: Hoch | Kategorie: Logging Monitoring | Automatisierbar: High

Beschreibung

All production alerting MUST be based on user-visible symptoms (elevated error rates, latency exceeding SLO thresholds, availability degradation) rather than internal causes (CPU usage, memory consumption, disk I/O). Every alert that pages an on-call engineer MUST be actionable and MUST have a linked runbook. Alert fatigue through non-actionable alerts MUST be actively managed and reduced.

Begründung

Risiko	Beschreibung
Risk	Alerting on infrastructure instead of symptom metrics creates alert fatigue and misses customer-impacting outages.
Risk	Missing action assignment on alerts prolongs time to resolution.

Risiko

Beschreibung

Risk

Alerting on infrastructure instead of symptom metrics creates alert fatigue and misses customer-impacting outages.

Risk

Missing action assignment on alerts prolongs time to resolution.

Anforderungen

All production alerting MUST be based on user-visible symptoms (elevated error rates, latency exceeding SLO thresholds, availability degradation) rather than internal causes (CPU usage, memory consumption, disk I/O).
Every alert that pages an on-call engineer MUST be actionable and MUST have a linked runbook.
Alert fatigue through non-actionable alerts MUST be actively managed and reduced.

Implementierungsleitfaden

Document the requirement in a version-controlled policy or standard.
Assign ownership and establish a review cadence (e.g., quarterly).
Implement the required configuration as code in Terraform or equivalent IaC.

Reifegrade

Level	Name	Kriterien
L1	Ad hoc – control not consistently applied	* Manual evidence exists for some workloads, but coverage is unknown.
L2	Standardised – policy defined and communicated	* A version-controlled policy, ADR, or standard describes the requirement.
L3	Integrated – automated validation in CI/CD	* Terraform checks run automatically on pull requests and report results.
L4	Auditable – metrics and exceptions tracked	* Control coverage, failures, and approved exceptions are reported quarterly.
L5	Optimised – continuous improvement based on data	* Control effectiveness is measured and iteratively improved.

Level

Name

Kriterien

Ad hoc – control not consistently applied

* Manual evidence exists for some workloads, but coverage is unknown.

Standardised – policy defined and communicated

* A version-controlled policy, ADR, or standard describes the requirement.

Integrated – automated validation in CI/CD

* Terraform checks run automatically on pull requests and report results.

Auditable – metrics and exceptions tracked

* Control coverage, failures, and approved exceptions are reported quarterly.

Optimised – continuous improvement based on data

* Control effectiveness is measured and iteratively improved.

Checks

waf-ops-040.tf.aws.cloudwatch-alarm-symptom-based

Engine: terraform | Provider: aws | Automated: Ja

CloudWatch alarms must use symptom-based metrics

CloudWatch alarms MUST use application-level symptom metrics (error rates, latency, request counts) rather than exclusively infrastructure metrics. Alarms MUST include alarm_description with actionable information.

Scope: resource — aws_cloudwatch_metric_alarm

Assertions: * alarm_description not empty — CloudWatch alarm must have a non-empty alarm_description explaining the symptom and action. * alarm_actions not empty — CloudWatch alarm must have at least one alarm_action (SNS topic, Auto Scaling, etc.).

Remediation: Add alarm_description with symptom description and runbook link. Ensure alarm_actions routes to an SNS topic connected to on-call alerting.

Compliant

resource "aws_cloudwatch_metric_alarm" "high_error_rate" {
  alarm_name          = "payment-service-high-error-rate"
  comparison_operator = "GreaterThanThreshold"
  evaluation_periods  = 2
  metric_name         = "5XXError"
  namespace           = "AWS/ApplicationELB"
  period              = 60
  statistic           = "Sum"
  threshold           = 10
  alarm_description   = "Payment service 5XX error rate > 10/min. Runbook: https://wiki/runbooks/payment-errors"
  alarm_actions       = [aws_sns_topic.pagerduty.arn]
  dimensions = {
    LoadBalancer = aws_lb.main.arn_suffix
  }
}

Non-Compliant

resource "aws_cloudwatch_metric_alarm" "high_cpu" {
  alarm_name          = "high-cpu"
  comparison_operator = "GreaterThanThreshold"
  evaluation_periods  = 1
  metric_name         = "CPUUtilization"
  namespace           = "AWS/EC2"
  period              = 300
  statistic           = "Average"
  threshold           = 80
  # No alarm_description, no runbook link – WAF-OPS-040 Violation
  # Cause-based metric, not symptom-based
}

waf-ops-040.tf.azurerm.monitor-alert-symptom

Engine: terraform | Provider: azurerm | Automated: Ja

Azure Monitor metric alerts must include symptom description and action group

Azure Monitor metric alerts for production workloads MUST include a descriptive name indicating the symptom and MUST reference an action group for notification.

Scope: resource — azurerm_monitor_metric_alert

Assertions: * description not empty — Azure Monitor alert must have a description explaining the symptom. * action attribute exists — Azure Monitor alert must reference at least one action group.

Remediation: Add a description field explaining the user-visible symptom and runbook location. Reference an azurerm_monitor_action_group in the action block.

Compliant

resource "azurerm_monitor_metric_alert" "error_rate" {
  name                = "payment-service-error-rate-high"
  resource_group_name = azurerm_resource_group.main.name
  scopes              = [azurerm_application_insights.main.id]
  description         = "Payment service error rate > 1%. Runbook: https://wiki/runbooks/payment-errors"

  criteria {
    metric_namespace = "Microsoft.Insights/Components"
    metric_name      = "requests/failed"
    aggregation      = "Count"
    operator         = "GreaterThan"
    threshold        = 10
  }

  action {
    action_group_id = azurerm_monitor_action_group.oncall.id
  }
}

Non-Compliant

resource "azurerm_monitor_metric_alert" "cpu_alert" {
  name                = "cpu-high"
  resource_group_name = azurerm_resource_group.main.name
  scopes              = [azurerm_virtual_machine.app.id]
  # No description, no action group – WAF-OPS-040 Violation
  criteria {
    metric_namespace = "Microsoft.Compute/virtualMachines"
    metric_name      = "Percentage CPU"
    aggregation      = "Average"
    operator         = "GreaterThan"
    threshold        = 80
  }
}

waf-ops-040.tf.google.monitoring-alert-symptom

Engine: terraform | Provider: google | Automated: Ja

GCP Monitoring alert policies must use symptom-based conditions

GCP Monitoring alert policies MUST include a user_labels field identifying the service and symptom type, and MUST have notification channels configured.

Scope: resource — google_monitoring_alert_policy

Assertions: * display_name not empty — Alert policy must have a descriptive display_name. * notification_channels not empty — Alert policy must have at least one notification channel.

Remediation: Add notification_channels referencing PagerDuty, Slack, or email channels. Name the alert with the service and symptom (e.g., 'payment-service-high-latency').

Compliant

resource "google_monitoring_alert_policy" "latency_alert" {
  display_name = "payment-service-latency-p99-high"
  combiner     = "OR"

  conditions {
    display_name = "p99 latency > 500ms"
    condition_threshold {
      filter          = "metric.type=\"custom.googleapis.com/payment/latency_p99\""
      comparison      = "COMPARISON_GT"
      threshold_value = 500
      duration        = "60s"
    }
  }

  notification_channels = [google_monitoring_notification_channel.pagerduty.name]

  documentation {
    content = "Payment service p99 latency exceeded 500ms. Runbook: https://wiki/runbooks/payment-latency"
  }
}

Non-Compliant

resource "google_monitoring_alert_policy" "cpu_alert" {
  display_name = "high-cpu"
  combiner     = "OR"
  conditions {
    display_name = "CPU > 80%"
    condition_threshold {
      filter     = "metric.type=\"compute.googleapis.com/instance/cpu/utilization\""
      comparison = "COMPARISON_GT"
      threshold_value = 0.8
    }
  }
  # No notification channels – WAF-OPS-040 Violation
}

waf-ops-040.tf.stackit.observability-dashboard

Engine: terraform | Provider: stackit | Automated: Ja

All production alerting for stackit

All production alerting must be documented in IaC via a observability-dashboard tag/label on stackit_project. This validates the control is tracked as code.

Scope: resource — stackit_project

Assertions: * labels key exists — stackit_project is missing the observability-dashboard tag/label.

Remediation: Add a observability-dashboard tag/label to the stackit_project resource to show the stackit implementation of WAF-OPS-040 is tracked in IaC.

Compliant

resource "stackit_project" "example" {
  labels = {
    observability-dashboard = "true"
  }
}

Non-Compliant

resource "stackit_project" "example" {
  # Missing observability-dashboard labels
}

waf-ops-040.tf.ovh.observability-dashboard

Engine: terraform | Provider: ovh | Automated: Ja

All production alerting for ovh

All production alerting must be documented in IaC via a observability-dashboard tag/label on ovh_cloud_project. This validates the control is tracked as code.

Scope: resource — ovh_cloud_project

Assertions: * tags key exists — ovh_cloud_project is missing the observability-dashboard tag/label.

Remediation: Add a observability-dashboard tag/label to the ovh_cloud_project resource to show the ovh implementation of WAF-OPS-040 is tracked in IaC.

Compliant

resource "ovh_cloud_project" "example" {
  tags = {
    observability-dashboard = "true"
  }
}

Non-Compliant

resource "ovh_cloud_project" "example" {
  # Missing observability-dashboard tags
}

waf-ops-040.tf.hcloud.observability-dashboard

Engine: terraform | Provider: hcloud | Automated: Ja

All production alerting for hcloud

All production alerting must be documented in IaC via a observability-dashboard tag/label on hcloud_server. This validates the control is tracked as code.

Scope: resource — hcloud_server

Assertions: * labels key exists — hcloud_server is missing the observability-dashboard tag/label.

Remediation: Add a observability-dashboard tag/label to the hcloud_server resource to show the hcloud implementation of WAF-OPS-040 is tracked in IaC.

Compliant

resource "hcloud_server" "example" {
  labels = {
    observability-dashboard = "true"
  }
}

Non-Compliant

resource "hcloud_server" "example" {
  # Missing observability-dashboard labels
}

Evidenz

Art Erforderlich Beschreibung

Art	Erforderlich	Beschreibung
`iac`	Erforderlich	Terraform configuration implementing the control.
`governance`	Erforderlich	Policy, ADR, or standard documenting the requirement.
`config`	Optional	Cloud console or API output confirming runtime state.

iac

Erforderlich

Terraform configuration implementing the control.

governance

Erforderlich

Policy, ADR, or standard documenting the requirement.

config

Optional

Cloud console or API output confirming runtime state.

WAF-OPS-040 – Alerting on Symptoms, Not Causes

Beschreibung

Begründung

Anforderungen

Implementierungsleitfaden

Reifegrade

Checks

waf-ops-040.tf.aws.cloudwatch-alarm-symptom-based

waf-ops-040.tf.azurerm.monitor-alert-symptom

waf-ops-040.tf.google.monitoring-alert-symptom

waf-ops-040.tf.stackit.observability-dashboard

waf-ops-040.tf.ovh.observability-dashboard

waf-ops-040.tf.hcloud.observability-dashboard

Evidenz

Verwandte Controls