WAF-OPS-040 – Alerting on Symptoms, Not Causes

Pillar: Excellence | Severity: High | Category: Logging Monitoring | Automatable: High

Description

All production alerting MUST be based on user-visible symptoms (elevated error rates, latency exceeding SLO thresholds, availability degradation) rather than internal causes (CPU usage, memory consumption, disk I/O). Every alert that pages an on-call engineer MUST be actionable and MUST have a linked runbook. Alert fatigue through non-actionable alerts MUST be actively managed and reduced.

Rationale

Risk	Description
Risk	Alerting on infrastructure instead of symptom metrics creates alert fatigue and misses customer-impacting outages.
Risk	Missing action assignment on alerts prolongs time to resolution.

Risk

Description

Risk

Alerting on infrastructure instead of symptom metrics creates alert fatigue and misses customer-impacting outages.

Risk

Missing action assignment on alerts prolongs time to resolution.

Requirements

All production alerting MUST be based on user-visible symptoms (elevated error rates, latency exceeding SLO thresholds, availability degradation) rather than internal causes (CPU usage, memory consumption, disk I/O).
Every alert that pages an on-call engineer MUST be actionable and MUST have a linked runbook.
Alert fatigue through non-actionable alerts MUST be actively managed and reduced.

Implementation Guidance

Document the requirement in a version-controlled policy or standard.
Assign ownership and establish a review cadence (e.g., quarterly).
Implement the required configuration as code in Terraform or equivalent IaC.

Maturity Levels

Level	Name	Criteria
L1	Ad hoc – control not consistently applied	* Manual evidence exists for some workloads, but coverage is unknown.
L2	Standardised – policy defined and communicated	* A version-controlled policy, ADR, or standard describes the requirement.
L3	Integrated – automated validation in CI/CD	* Terraform checks run automatically on pull requests and report results.
L4	Auditable – metrics and exceptions tracked	* Control coverage, failures, and approved exceptions are reported quarterly.
L5	Optimised – continuous improvement based on data	* Control effectiveness is measured and iteratively improved.

Level

Name

Criteria

Ad hoc – control not consistently applied

* Manual evidence exists for some workloads, but coverage is unknown.

Standardised – policy defined and communicated

* A version-controlled policy, ADR, or standard describes the requirement.

Integrated – automated validation in CI/CD

* Terraform checks run automatically on pull requests and report results.

Auditable – metrics and exceptions tracked

* Control coverage, failures, and approved exceptions are reported quarterly.

Optimised – continuous improvement based on data

* Control effectiveness is measured and iteratively improved.

Checks

waf-ops-040.tf.aws.cloudwatch-alarm-symptom-based

Engine: terraform | Provider: aws | Automated: Yes

CloudWatch alarms must use symptom-based metrics

CloudWatch alarms MUST use application-level symptom metrics (error rates, latency, request counts) rather than exclusively infrastructure metrics. Alarms MUST include alarm_description with actionable information.

Scope: resource — aws_cloudwatch_metric_alarm

Assertions: * alarm_description not empty — CloudWatch alarm must have a non-empty alarm_description explaining the symptom and action. * alarm_actions not empty — CloudWatch alarm must have at least one alarm_action (SNS topic, Auto Scaling, etc.).

Remediation: Add alarm_description with symptom description and runbook link. Ensure alarm_actions routes to an SNS topic connected to on-call alerting.

Compliant

resource "aws_cloudwatch_metric_alarm" "high_error_rate" {
  alarm_name          = "payment-service-high-error-rate"
  comparison_operator = "GreaterThanThreshold"
  evaluation_periods  = 2
  metric_name         = "5XXError"
  namespace           = "AWS/ApplicationELB"
  period              = 60
  statistic           = "Sum"
  threshold           = 10
  alarm_description   = "Payment service 5XX error rate > 10/min. Runbook: https://wiki/runbooks/payment-errors"
  alarm_actions       = [aws_sns_topic.pagerduty.arn]
  dimensions = {
    LoadBalancer = aws_lb.main.arn_suffix
  }
}

Non-Compliant

resource "aws_cloudwatch_metric_alarm" "high_cpu" {
  alarm_name          = "high-cpu"
  comparison_operator = "GreaterThanThreshold"
  evaluation_periods  = 1
  metric_name         = "CPUUtilization"
  namespace           = "AWS/EC2"
  period              = 300
  statistic           = "Average"
  threshold           = 80
  # No alarm_description, no runbook link – WAF-OPS-040 Violation
  # Cause-based metric, not symptom-based
}

waf-ops-040.tf.azurerm.monitor-alert-symptom

Engine: terraform | Provider: azurerm | Automated: Yes

Azure Monitor metric alerts must include symptom description and action group

Azure Monitor metric alerts for production workloads MUST include a descriptive name indicating the symptom and MUST reference an action group for notification.

Scope: resource — azurerm_monitor_metric_alert

Assertions: * description not empty — Azure Monitor alert must have a description explaining the symptom. * action attribute exists — Azure Monitor alert must reference at least one action group.

Remediation: Add a description field explaining the user-visible symptom and runbook location. Reference an azurerm_monitor_action_group in the action block.

Compliant

resource "azurerm_monitor_metric_alert" "error_rate" {
  name                = "payment-service-error-rate-high"
  resource_group_name = azurerm_resource_group.main.name
  scopes              = [azurerm_application_insights.main.id]
  description         = "Payment service error rate > 1%. Runbook: https://wiki/runbooks/payment-errors"

  criteria {
    metric_namespace = "Microsoft.Insights/Components"
    metric_name      = "requests/failed"
    aggregation      = "Count"
    operator         = "GreaterThan"
    threshold        = 10
  }

  action {
    action_group_id = azurerm_monitor_action_group.oncall.id
  }
}

Non-Compliant

resource "azurerm_monitor_metric_alert" "cpu_alert" {
  name                = "cpu-high"
  resource_group_name = azurerm_resource_group.main.name
  scopes              = [azurerm_virtual_machine.app.id]
  # No description, no action group – WAF-OPS-040 Violation
  criteria {
    metric_namespace = "Microsoft.Compute/virtualMachines"
    metric_name      = "Percentage CPU"
    aggregation      = "Average"
    operator         = "GreaterThan"
    threshold        = 80
  }
}

waf-ops-040.tf.google.monitoring-alert-symptom

Engine: terraform | Provider: google | Automated: Yes

GCP Monitoring alert policies must use symptom-based conditions

GCP Monitoring alert policies MUST include a user_labels field identifying the service and symptom type, and MUST have notification channels configured.

Scope: resource — google_monitoring_alert_policy

Assertions: * display_name not empty — Alert policy must have a descriptive display_name. * notification_channels not empty — Alert policy must have at least one notification channel.

Remediation: Add notification_channels referencing PagerDuty, Slack, or email channels. Name the alert with the service and symptom (e.g., 'payment-service-high-latency').

Compliant

resource "google_monitoring_alert_policy" "latency_alert" {
  display_name = "payment-service-latency-p99-high"
  combiner     = "OR"

  conditions {
    display_name = "p99 latency > 500ms"
    condition_threshold {
      filter          = "metric.type=\"custom.googleapis.com/payment/latency_p99\""
      comparison      = "COMPARISON_GT"
      threshold_value = 500
      duration        = "60s"
    }
  }

  notification_channels = [google_monitoring_notification_channel.pagerduty.name]

  documentation {
    content = "Payment service p99 latency exceeded 500ms. Runbook: https://wiki/runbooks/payment-latency"
  }
}

Non-Compliant

resource "google_monitoring_alert_policy" "cpu_alert" {
  display_name = "high-cpu"
  combiner     = "OR"
  conditions {
    display_name = "CPU > 80%"
    condition_threshold {
      filter     = "metric.type=\"compute.googleapis.com/instance/cpu/utilization\""
      comparison = "COMPARISON_GT"
      threshold_value = 0.8
    }
  }
  # No notification channels – WAF-OPS-040 Violation
}

waf-ops-040.tf.stackit.observability-dashboard

Engine: terraform | Provider: stackit | Automated: Yes

All production alerting for stackit

All production alerting must be documented in IaC via a observability-dashboard tag/label on stackit_project. This validates the control is tracked as code.

Scope: resource — stackit_project

Assertions: * labels key exists — stackit_project is missing the observability-dashboard tag/label.

Remediation: Add a observability-dashboard tag/label to the stackit_project resource to show the stackit implementation of WAF-OPS-040 is tracked in IaC.

Compliant

resource "stackit_project" "example" {
  labels = {
    observability-dashboard = "true"
  }
}

Non-Compliant

resource "stackit_project" "example" {
  # Missing observability-dashboard labels
}

waf-ops-040.tf.ovh.observability-dashboard

Engine: terraform | Provider: ovh | Automated: Yes

All production alerting for ovh

All production alerting must be documented in IaC via a observability-dashboard tag/label on ovh_cloud_project. This validates the control is tracked as code.

Scope: resource — ovh_cloud_project

Assertions: * tags key exists — ovh_cloud_project is missing the observability-dashboard tag/label.

Remediation: Add a observability-dashboard tag/label to the ovh_cloud_project resource to show the ovh implementation of WAF-OPS-040 is tracked in IaC.

Compliant

resource "ovh_cloud_project" "example" {
  tags = {
    observability-dashboard = "true"
  }
}

Non-Compliant

resource "ovh_cloud_project" "example" {
  # Missing observability-dashboard tags
}

waf-ops-040.tf.hcloud.observability-dashboard

Engine: terraform | Provider: hcloud | Automated: Yes

All production alerting for hcloud

All production alerting must be documented in IaC via a observability-dashboard tag/label on hcloud_server. This validates the control is tracked as code.

Scope: resource — hcloud_server

Assertions: * labels key exists — hcloud_server is missing the observability-dashboard tag/label.

Remediation: Add a observability-dashboard tag/label to the hcloud_server resource to show the hcloud implementation of WAF-OPS-040 is tracked in IaC.

Compliant

resource "hcloud_server" "example" {
  labels = {
    observability-dashboard = "true"
  }
}

Non-Compliant

resource "hcloud_server" "example" {
  # Missing observability-dashboard labels
}

Evidence

Kind Required Description

Kind	Required	Description
`iac`	Required	Terraform configuration implementing the control.
`governance`	Required	Policy, ADR, or standard documenting the requirement.
`config`	Optional	Cloud console or API output confirming runtime state.

iac

Required

Terraform configuration implementing the control.

governance

Required

Policy, ADR, or standard documenting the requirement.

config

Optional

Cloud console or API output confirming runtime state.

Related Controls

Framework	Controls
ISO/IEC 20000-1	* Incident management * Monitoring, measurement, analysis, and evaluation
ITIL 4	* Monitoring and event management * Incident management
SRE Book (Google)	* Chapter 6: Monitoring Distributed Systems – Four Golden Signals * Chapter 5: Eliminating Toil

Framework

Controls

ISO/IEC 20000-1

* Incident management * Monitoring, measurement, analysis, and evaluation

ITIL 4

* Monitoring and event management * Incident management

SRE Book (Google)

* Chapter 6: Monitoring Distributed Systems – Four Golden Signals * Chapter 5: Eliminating Toil