WAF-AGN-090 – Cost & Performance Monitoring

Beschreibung

Agent operations must be economic and performant. Token usage and compute costs MUST be monitored. Budget alerts MUST be configured to prevent cost overruns. Model selection MUST be optimized for cost-performance tradeoffs. Efficient caching and context management MUST be implemented.

Begründung

Risiko Beschreibung

Risk

Unbounded agent compute consumption leads to exploding costs and increased carbon footprint.

Risk

Inefficient agent architectures miss latency requirements and waste resources.

Anforderungen

  • Agent operations must be economic and performant.

  • Token usage and compute costs MUST be monitored.

  • Budget alerts MUST be configured to prevent cost overruns.

  • Model selection MUST be optimized for cost-performance tradeoffs.

  • Efficient caching and context management MUST be implemented.

Implementierungsleitfaden

  1. Document the requirement in a version-controlled policy or standard.

  2. Assign ownership and establish a review cadence (e.g., quarterly).

  3. Implement the required configuration as code in Terraform or equivalent IaC.

Reifegrade

Level Name Kriterien

L1

Ad hoc – control not consistently applied

* Manual evidence exists for some workloads, but coverage is unknown.

L2

Standardised – policy defined and communicated

* A version-controlled policy, ADR, or standard describes the requirement.

L3

Integrated – automated validation in CI/CD

* Terraform checks run automatically on pull requests and report results.

L4

Auditable – metrics and exceptions tracked

* Control coverage, failures, and approved exceptions are reported quarterly.

L5

Optimised – continuous improvement based on data

* Control effectiveness is measured and iteratively improved.

Checks

waf-agn-090.tf.aws.token-monitoring-configured

Engine: terraform | Provider: aws | Automated: Ja

Agent token usage must be monitored

Token usage (input, output, total) for all agents must be tracked and monitored. Usage must be exported to CloudWatch with daily aggregation. Anomalies in token usage must trigger alerts.

Scope: resource — aws_cloudwatch_metric_alarm, aws_cloudwatch_dashboard

Assertions: * alarm_name not empty — Token usage monitoring must be configured.

Remediation: Configure CloudWatch Metrics for Bedrock token usage. Create dashboards showing token consumption by agent. Set alarms for unusual token spikes that may indicate runaway prompts or model issues.

Compliant
resource "aws_cloudwatch_metric_alarm" "token_usage" {
  alarm_name          = "agent-token-spike"
  comparison_operator = "GreaterThanThreshold"
  evaluation_periods  = 1
  metric_name         = "InputTokenCount"
  namespace           = "AWS/Bedrock"
  period              = 300
  statistic           = "Sum"
  threshold           = 50000
  unit                = "Count"
  alarm_description   = "Agent token usage anomaly detected"
}

resource "aws_cloudwatch_dashboard" "agent_dashboard" {
  dashboard_name = "agent-metrics"

  dashboard_body = jsonencode({
    widgets = [{
      type   = "metric"
      x      = 0
      y      = 0
      width  = 12
      height = 6
      properties = {
        metrics = [
          ["AWS/Bedrock", "InputTokenCount", "AgentName", "customer-agent"]
        ]
        period = 300
        stat   = "Sum"
        region = "us-east-1"
      }
    }]
  })
}
Non-Compliant
# No token monitoring configured
# WAF-AGN-090 Violation
# Cannot track or control costs

waf-agn-090.tf.aws.budget-alerts-configured

Engine: terraform | Provider: aws | Automated: Ja

Budget alerts must be configured for agent costs

Cost budgets must be set for agent operations with alerts at 50%, 75%, 90%, and 100% of budget. Alerts must be sent to designated recipients before thresholds are reached.

Scope: resource — aws_budgets_budget

Assertions: * budget_name not empty — Budget alerts must be configured for agent costs.

Remediation: Configure AWS Cost Explorer budgets for Bedrock spending. Set up budget alerts via SNS topics. Define budget periods (monthly, quarterly) and threshold percentages.

Compliant
resource "aws_budgets_budget" "agent_budget" {
  budget_name = "agent-monthly-budget"

  budget_type      = "COST"
  limit_amount     = "500"
  limit_unit       = "USD"
  time_unit        = "MONTHLY"

  cost_types {
    include_tax           = true
    include_subscription  = true
    include_recurring     = true
    include_other_subscription = true
    include_discounts     = true
    use_amortized         = false
    use_blended           = false
  }

  time_period_start = "2026-01-01_00:00"
  time_period_end   = "2026-12-31_23:59"

  notification {
    notification_type       = "ACTUAL"
    comparison_operator     = "GREATER_THAN"
    threshold               = 50
    threshold_type          = "PERCENTAGE"
    notification_state      = "ALARM"
    subscriber_email_addresses = ["team-ops@example.com"]
  }

  notification {
    notification_type       = "ACTUAL"
    comparison_operator     = "GREATER_THAN"
    threshold               = 75
    threshold_type          = "PERCENTAGE"
    notification_state      = "ALARM"
    subscriber_email_addresses = ["team-ops@example.com", "finance@example.com"]
  }

  notification {
    notification_type       = "ACTUAL"
    comparison_operator     = "GREATER_THAN"
    threshold               = 90
    threshold_type          = "PERCENTAGE"
    notification_state      = "ALARM"
    subscriber_email_addresses = ["team-ops@example.com", "finance@example.com"]
  }

  notification {
    notification_type       = "ACTUAL"
    comparison_operator     = "GREATER_THAN"
    threshold               = 100
    threshold_type          = "PERCENTAGE"
    notification_state      = "ALARM"
    subscriber_email_addresses = ["team-ops@example.com", "finance@example.com", "management@example.com"]
  }
}
Non-Compliant
# No budget alerts configured
# WAF-AGN-090 Violation
# Cost overruns not detected

waf-agn-090.tf.aws.model-optimization-configured

Engine: terraform | Provider: aws | Automated: Ja

Model selection must be optimized for cost-performance

Agents must use the most cost-effective model that meets quality requirements. Model selection should consider input/output length, latency requirements, and cost per token. Optimization should be reviewed quarterly.

Scope: resource — aws_bedrock_foundation_model

Assertions: * model_id not empty — Model selection must be documented and optimized.

Remediation: Document model selection criteria for each agent use case. Compare Claude Instant vs Claude Sonnet vs Claude Opus based on quality requirements. Use OpenAI GPT-3.5 Turbo vs GPT-4 based on task complexity. Implement model fallback for cost management.

Compliant
# Documented model selection criteria
# - Simple queries: Claude Instant (cheapest)
# - Moderate complexity: Claude Sonnet (balanced)
# - High complexity: Claude Opus (best quality)
# Review model performance quarterly

resource "aws_bedrock_foundation_model" "model_selection" {
  model_id = "anthropic.claude-3-haiku-20240307-v1:0"  # Cost-optimized choice
  # Model selection documented in architecture decision record
}
Non-Compliant
# Using expensive models for simple tasks
# WAF-AGN-090 Violation
# Cost-performance not optimized

waf-agn-090.tf.aws.caching-configured

Engine: terraform | Provider: aws | Automated: Ja

Caching must be implemented to reduce redundant API calls

Caching must be implemented for repeated queries, common prompts, and similar inputs. Cache keys must include input hash and model identifier. Cache TTL must be configured based on data freshness requirements.

Scope: resource — aws_dynamodb_table, aws_lambda_function

Assertions: * name not empty — Caching must be configured to reduce redundant API calls.

Remediation: Implement caching layer (ElastiCache, DynamoDB TTL) for agent responses. Cache based on input hash to handle similar queries efficiently. Configure cache invalidation based on data changes. Monitor cache hit rates.

Compliant
resource "aws_dynamodb_table" "agent_cache" {
  name         = "agent-response-cache"
  billing_mode = "PAY_PER_REQUEST"
  hash_key     = "cache_key"

  attribute {
    name = "cache_key"
    type = "S"
  }

  ttl {
    attribute_name = "expires_at"
    enabled        = true
  }

  # Cache keys based on input hash
  # TTL based on data freshness needs
}

resource "aws_lambda_function" "cache_middleware" {
  function_name = "agent-cache-middleware"
  # Middleware checks cache before API call
}
Non-Compliant
# No caching implemented
# WAF-AGN-090 Violation
# All queries hit API directly

waf-agn-090.tf.azurerm.budget-alert-agent

Engine: terraform | Provider: azurerm | Automated: Ja

Azure Cost Management budget alerts must be configured for AI agent spending

Azure Cost Management budgets must monitor AI service spending (OpenAI, Cognitive Services) with alerts at 75%, 90%, and 100% thresholds.

Scope: resource — azurerm_consumption_budget_subscription, azurerm_consumption_budget_resource_group

Assertions: * notification attribute exists — Budget must have notification thresholds configured. * amount greater than 0 — Budget must define a positive spending limit.

Remediation: Configure azurerm_consumption_budget_subscription or azurerm_consumption_budget_resource_group with filters for AI service resource groups and alert email contacts.

Compliant
resource "azurerm_consumption_budget_subscription" "agent_budget" {
  name            = "agent-spend-budget"
  subscription_id = data.azurerm_subscription.current.id

  amount     = 1000
  time_grain = "Monthly"

  time_period {
    start_date = "2026-01-01T00:00:00Z"
    end_date   = "2026-12-31T23:59:59Z"
  }

  notification {
    enabled   = true
    threshold = 75
    operator  = "EqualTo"
    contact_emails = ["team-ops@example.com"]
  }

  notification {
    enabled   = true
    threshold = 90
    operator  = "EqualTo"
    contact_emails = ["team-ops@example.com", "finance@example.com"]
  }
}
Non-Compliant
resource "azurerm_cognitive_account" "ai" {
  name = "ai-services"
  # No spending budget or alert configured
  # WAF-AGN-090 Violation
}

waf-agn-090.tf.google.budget-alert-agent

Engine: terraform | Provider: google | Automated: Ja

GCP budgets must be configured for agent cost control

GCP budgets must monitor Vertex AI, Cloud Run, and Cloud Logging spending for agent operations with threshold alerts.

Scope: resource — google_billing_budget

Assertions: * threshold_rules attribute exists — Billing budget must define threshold alert rules. * amount attribute exists — Billing budget must define a spending amount.

Remediation: Configure google_billing_budget with filters for Vertex AI / Cloud Run services and alert pubsub/email channels.

Compliant
resource "google_billing_budget" "agent_budget" {
  billing_account = data.google_billing_account.account.id
  display_name    = "Agent Operations Budget"

  amount {
    specified_amount {
      currency_code = "EUR"
      units = "1000"
    }
  }

  threshold_rules {
    threshold_percent = 75
  }

  threshold_rules {
    threshold_percent = 90
  }

  all_updates_rule {
    monitoring_notification_channels = [google_monitoring_notification_channel.agent_budget.id]
  }
}
Non-Compliant
resource "google_cloud_run_service" "agent" {
  name = "agent-service"
  # No billing budget configured
  # WAF-AGN-090 Violation
}

waf-agn-090.tf.stackit.agentic-efficiency-reviewed

Engine: terraform | Provider: stackit | Automated: Ja

Agent operations must be economic and performant for stackit

Agent operations must be economic and performant must be documented in IaC via a agentic-efficiency-reviewed tag/label on stackit_project. This validates the control is tracked as code.

Scope: resource — stackit_project

Assertions: * labels key exists — stackit_project is missing the agentic-efficiency-reviewed tag/label.

Remediation: Add a agentic-efficiency-reviewed tag/label to the stackit_project resource to show the stackit implementation of WAF-AGN-090 is tracked in IaC.

Compliant
resource "stackit_project" "example" {
  labels = {
    agentic-efficiency-reviewed = "true"
  }
}
Non-Compliant
resource "stackit_project" "example" {
  # Missing agentic-efficiency-reviewed labels
}

waf-agn-090.tf.ovh.agentic-efficiency-reviewed

Engine: terraform | Provider: ovh | Automated: Ja

Agent operations must be economic and performant for ovh

Agent operations must be economic and performant must be documented in IaC via a agentic-efficiency-reviewed tag/label on ovh_cloud_project. This validates the control is tracked as code.

Scope: resource — ovh_cloud_project

Assertions: * tags key exists — ovh_cloud_project is missing the agentic-efficiency-reviewed tag/label.

Remediation: Add a agentic-efficiency-reviewed tag/label to the ovh_cloud_project resource to show the ovh implementation of WAF-AGN-090 is tracked in IaC.

Compliant
resource "ovh_cloud_project" "example" {
  tags = {
    agentic-efficiency-reviewed = "true"
  }
}
Non-Compliant
resource "ovh_cloud_project" "example" {
  # Missing agentic-efficiency-reviewed tags
}

waf-agn-090.tf.hcloud.agentic-efficiency-reviewed

Engine: terraform | Provider: hcloud | Automated: Ja

Agent operations must be economic and performant for hcloud

Agent operations must be economic and performant must be documented in IaC via a agentic-efficiency-reviewed tag/label on hcloud_server. This validates the control is tracked as code.

Scope: resource — hcloud_server

Assertions: * labels key exists — hcloud_server is missing the agentic-efficiency-reviewed tag/label.

Remediation: Add a agentic-efficiency-reviewed tag/label to the hcloud_server resource to show the hcloud implementation of WAF-AGN-090 is tracked in IaC.

Compliant
resource "hcloud_server" "example" {
  labels = {
    agentic-efficiency-reviewed = "true"
  }
}
Non-Compliant
resource "hcloud_server" "example" {
  # Missing agentic-efficiency-reviewed labels
}

Evidenz

Art Erforderlich Beschreibung

iac

Erforderlich

Terraform configuration implementing the control.

governance

Erforderlich

Policy, ADR, or standard documenting the requirement.

config

Optional

Cloud console or API output confirming runtime state.

Verwandte Controls

Framework Controls

ISO 27001:2022

* A.8.2 – Privileged access rights * A.12.1 – Control of operational software

ISO 27017

* CLD.9.5 – Segregation in virtual computing environments

CIS Controls v8

* CIS 10 – Data Accuracy

SOC 2 Type II

* CC4.1 – Monitoring activities * CC7.1 – Infrastructure and software monitoring

NIST SP 800-53

* SC-5 – Resource availability * SC-39 – Process isolation

NIST CSF 2.0

* DE.CM – Continuous monitoring * DE.AE – Anomalies and events

GDPR

* Art. 32 – Security of processing * Art. 5(1)(f) – Integrity and confidentiality

PCI DSS v4.0

* Req 10 – Monitor and detect threats

CSA STAR

* CCM Logging and Monitoring

BSI C5:2020

* OPS-01 – Operational monitoring

TISAX

* Information security – Monitoring and logging

ANSSI SecNumCloud

* Domain – Logging and monitoring

BIO

* BIO – Logboekhouding en monitoring

ENS High

* op.exp.6 – Gestión de cambios

UK NCSC CAF

* A4 – Policy and assurance

FedRAMP

* SC-5, SC-39 (Moderate baseline)

CMMC 2.0

* SC.L2-3.8.1 – Limit information system use

IT-Grundschutz

* ORP.1 – Organisation

IRAP

* ISM – Logging and monitoring

CCCS PBMM

* SC-5 – Resource availability

MAS TRM

* Ch.9 – Change management

ISMAP

* Monitoring and logging controls

FISC

* Technical measures – Logging and monitoring