WAF-PERF-050 – Performance Monitoring & SLO Definition

Pillar: Efficiancy | Severity: High | Category: Logging Monitoring | Automatable: High

Description

All production services MUST have Service Level Objectives (SLOs) defined for performance-relevant metrics: latency (P95 and P99), error rate, and availability. Service Level Indicators (SLIs) MUST be instrumented and measured continuously. Performance dashboards MUST be available for all production services showing current SLO compliance and historical trends. Alerting MUST be configured to notify on-call teams when SLO burn rate exceeds acceptable thresholds. SLOs MUST be reviewed and updated quarterly or after major architectural changes.

Rationale

Risk	Description
Risk	Services without SLOs have no measurable reliability commitment and hide user impact.
Risk	Missing error budgets allow recurring instability to be accepted without consequence.

Risk

Description

Risk

Services without SLOs have no measurable reliability commitment and hide user impact.

Risk

Missing error budgets allow recurring instability to be accepted without consequence.

Requirements

All production services MUST have Service Level Objectives (SLOs) defined for performance-relevant metrics: latency (P95 and P99), error rate, and availability.
Service Level Indicators (SLIs) MUST be instrumented and measured continuously.
Performance dashboards MUST be available for all production services showing current SLO compliance and historical trends.
Alerting MUST be configured to notify on-call teams when SLO burn rate exceeds acceptable thresholds.
SLOs MUST be reviewed and updated quarterly or after major architectural changes.

Implementation Guidance

Document the requirement in a version-controlled policy or standard.
Assign ownership and establish a review cadence (e.g., quarterly).
Implement the required configuration as code in Terraform or equivalent IaC.

Maturity Levels

Level	Name	Criteria
L1	Ad hoc – control not consistently applied	* Manual evidence exists for some workloads, but coverage is unknown.
L2	Standardised – policy defined and communicated	* A version-controlled policy, ADR, or standard describes the requirement.
L3	Integrated – automated validation in CI/CD	* Terraform checks run automatically on pull requests and report results.
L4	Auditable – metrics and exceptions tracked	* Control coverage, failures, and approved exceptions are reported quarterly.
L5	Optimised – continuous improvement based on data	* Control effectiveness is measured and iteratively improved.

Level

Name

Criteria

Ad hoc – control not consistently applied

* Manual evidence exists for some workloads, but coverage is unknown.

Standardised – policy defined and communicated

* A version-controlled policy, ADR, or standard describes the requirement.

Integrated – automated validation in CI/CD

* Terraform checks run automatically on pull requests and report results.

Auditable – metrics and exceptions tracked

* Control coverage, failures, and approved exceptions are reported quarterly.

Optimised – continuous improvement based on data

* Control effectiveness is measured and iteratively improved.

Checks

waf-perf-050.tf.aws.cloudwatch-latency-alarm

Engine: terraform | Provider: aws | Automated: Yes

AWS CloudWatch alarms must monitor P99 latency for load balancers

Load balancers must have CloudWatch alarms configured for P99 target response time to detect latency degradation before it becomes visible to users.

Scope: resource — aws_cloudwatch_metric_alarm

Assertions: * alarm_actions not empty — CloudWatch alarm must have at least one action (SNS topic) configured. * evaluation_periods greater than or equal 2 — Alarm must evaluate over at least 2 periods to avoid spurious alerts. * alarm_description not empty — Alarm must have a description explaining what it monitors and what to do.

Remediation: Ensure CloudWatch alarms have alarm_actions pointing to SNS topics for notification. Set evaluation_periods >= 2 to reduce false positives. Add alarm_description with runbook link.

Compliant

resource "aws_cloudwatch_metric_alarm" "p99_latency" {
  alarm_name          = "payment-api-p99-latency"
  alarm_description   = "P99 latency > 500ms for payment API. Runbook: https://wiki/perf-runbook"
  comparison_operator = "GreaterThanThreshold"
  evaluation_periods  = 3
  metric_name         = "TargetResponseTime"
  namespace           = "AWS/ApplicationELB"
  period              = 60
  statistic           = "p99"
  threshold           = 0.5
  alarm_actions       = [aws_sns_topic.alerts.arn]
  dimensions = {
    LoadBalancer = aws_lb.app.arn_suffix
  }
}

Non-Compliant

resource "aws_cloudwatch_metric_alarm" "latency" {
  alarm_name          = "latency-alarm"
  comparison_operator = "GreaterThanThreshold"
  evaluation_periods  = 1
  metric_name         = "TargetResponseTime"
  namespace           = "AWS/ApplicationELB"
  period              = 60
  statistic           = "Average"  # Average hides P99 spikes – WAF-PERF-050 violation
  threshold           = 1.0
  # No alarm_actions – alert fires but nobody is notified
}

waf-perf-050.tf.azurerm.app-insights-availability

Engine: terraform | Provider: azurerm | Automated: Yes

Azure Application Insights must be configured for SLO monitoring

Azure Application Insights must be deployed and configured for all production services to enable performance monitoring, distributed tracing, and SLO tracking.

Scope: resource — azurerm_application_insights

Assertions: * application_type not empty — Application Insights application_type must be set (web or other). * retention_in_days greater than or equal 1 — Application Insights retention must be at least 30 days for trend analysis.

Remediation: Set application_type to "web" for HTTP services. Set retention_in_days >= 90 for adequate trend analysis. Configure availability tests and custom metric alerts for SLOs.

Compliant

resource "azurerm_application_insights" "app" {
  name                = "appi-payment-api"
  resource_group_name = azurerm_resource_group.main.name
  location            = azurerm_resource_group.main.location
  application_type    = "web"
  retention_in_days   = 90
  tags = {
    workload    = "payment-api"
    environment = "production"
  }
}

Non-Compliant

resource "azurerm_application_insights" "app" {
  name             = "appi-payment-api"
  application_type = "other"  # Wrong type for web service
  retention_in_days = 7       # Too short – WAF-PERF-050 violation
}

waf-perf-050.tf.google.monitoring-uptime-check

Engine: terraform | Provider: google | Automated: Yes

GCP services must have uptime checks and alert policies configured

GCP production services must have Cloud Monitoring uptime checks and alerting policies for latency and availability SLO monitoring.

Scope: resource — google_monitoring_alert_policy

Assertions: * notification_channels not empty — Alert policy must have at least one notification channel. * display_name not empty — Alert policy must have a descriptive display name. * enabled is true — Alert policy must be enabled.

Remediation: Ensure alert policies have notification_channels configured, are enabled, and have descriptive display names. Create SLO-based alerts using google_monitoring_slo resource.

Compliant

resource "google_monitoring_alert_policy" "latency" {
  display_name = "payment-api-p99-latency"
  enabled      = true
  conditions {
    display_name = "P99 latency > 500ms"
    condition_threshold {
      filter          = "metric.type=\"loadbalancing.googleapis.com/https/backend_latencies\" resource.type=\"https_lb_rule\""
      comparison      = "COMPARISON_GT"
      threshold_value = 500
      duration        = "60s"
      aggregations {
        alignment_period   = "60s"
        per_series_aligner = "ALIGN_PERCENTILE_99"
      }
    }
  }
  notification_channels = [google_monitoring_notification_channel.slack.id]
}

Non-Compliant

resource "google_monitoring_alert_policy" "latency" {
  display_name = "latency"
  enabled      = false  # Alert disabled – WAF-PERF-050 violation
  conditions { }
  # No notification_channels
}

waf-perf-050.tf.stackit.cdn-configured

Engine: terraform | Provider: stackit | Automated: Yes

All production services for stackit

All production services must be documented in IaC via a cdn-configured tag/label on stackit_project. This validates the control is tracked as code.

Scope: resource — stackit_project

Assertions: * labels key exists — stackit_project is missing the cdn-configured tag/label.

Remediation: Add a cdn-configured tag/label to the stackit_project resource to show the stackit implementation of WAF-PERF-050 is tracked in IaC.

Compliant

resource "stackit_project" "example" {
  labels = {
    cdn-configured = "true"
  }
}

Non-Compliant

resource "stackit_project" "example" {
  # Missing cdn-configured labels
}

waf-perf-050.tf.ovh.cdn-configured

Engine: terraform | Provider: ovh | Automated: Yes

All production services for ovh

All production services must be documented in IaC via a cdn-configured tag/label on ovh_cloud_project. This validates the control is tracked as code.

Scope: resource — ovh_cloud_project

Assertions: * tags key exists — ovh_cloud_project is missing the cdn-configured tag/label.

Remediation: Add a cdn-configured tag/label to the ovh_cloud_project resource to show the ovh implementation of WAF-PERF-050 is tracked in IaC.

Compliant

resource "ovh_cloud_project" "example" {
  tags = {
    cdn-configured = "true"
  }
}

Non-Compliant

resource "ovh_cloud_project" "example" {
  # Missing cdn-configured tags
}

waf-perf-050.tf.hcloud.cdn-configured

Engine: terraform | Provider: hcloud | Automated: Yes

All production services for hcloud

All production services must be documented in IaC via a cdn-configured tag/label on hcloud_server. This validates the control is tracked as code.

Scope: resource — hcloud_server

Assertions: * labels key exists — hcloud_server is missing the cdn-configured tag/label.

Remediation: Add a cdn-configured tag/label to the hcloud_server resource to show the hcloud implementation of WAF-PERF-050 is tracked in IaC.

Compliant

resource "hcloud_server" "example" {
  labels = {
    cdn-configured = "true"
  }
}

Non-Compliant

resource "hcloud_server" "example" {
  # Missing cdn-configured labels
}

Evidence

Kind Required Description

Kind	Required	Description
`iac`	Required	Terraform configuration implementing the control.
`governance`	Required	Policy, ADR, or standard documenting the requirement.
`config`	Optional	Cloud console or API output confirming runtime state.

iac

Required

Terraform configuration implementing the control.

governance

Required

Policy, ADR, or standard documenting the requirement.

config

Optional

Cloud console or API output confirming runtime state.

Related Controls

Framework	Controls
ISO/IEC 25010	* Performance Efficiency – Time Behavior * Performance Efficiency – Capacity
SRE Book (Google)	* Service Level Objectives * Error Budgets

Framework

Controls

ISO/IEC 25010

* Performance Efficiency – Time Behavior * Performance Efficiency – Capacity

SRE Book (Google)

* Service Level Objectives * Error Budgets