Monitoring & Dashboards¶

SynthOrg exposes runtime telemetry via a Prometheus /metrics endpoint plus structured JSON logs. This guide walks through every metric the application emits, a ready-to-import Grafana dashboard, and suggested alert rules. The canonical metric registration lives in src/synthorg/observability/prometheus_collector.py (pull-refreshed families) and src/synthorg/observability/prometheus_push_metrics.py (push-updated families); bounded label allowlists live in src/synthorg/observability/prometheus_labels.py.

Scraping¶

Point any Prometheus-compatible scraper at the running app:

scrape_configs:
  - job_name: synthorg
    scrape_interval: 30s
    static_configs:
      - targets: ['synthorg:8000']

The endpoint is unauthenticated by default; put it behind your normal scrape-ACL (firewall, sidecar proxy, Kubernetes NetworkPolicy). All metric names are prefixed with synthorg_.

Metric inventory¶

The Dashboard column maps each metric to the row it appears under in the default Grafana overview dashboard (monitoring/grafana/synthorg-overview.json). Rows are collapsible; the only row expanded by default is Health & SLO. A metric marked n/a (scrape-only) is exposed on /metrics and queryable in Prometheus but has no dedicated panel in the bundled dashboard yet. The dashboard exposes two filter variables ($workflow_definition_id, $department) that drill panels down per-entity. No per-agent label is exposed: an unbounded agent_id / agent label is a cardinality bomb, so per-agent cost and task breakdowns live in the structured logs and the REST cost / task APIs rather than in metrics. Default queries aggregate across the full set so the unfiltered view is always meaningful.

Bounded-label values are enforced at record time in src/synthorg/observability/prometheus_labels.py; PromQL filters that reference values outside those allowlists will never match data. The six registry-bound push-time label names (agent_id, department, workflow_definition_id, tool_name, provider, model_id on the metrics noted below) are validated against a registry-bound snapshot rebuilt on every Prometheus scrape; unknown values drop that one sample with a metrics.scrape.failed WARN log per unknown label per scrape. The log repeats on the next scrape if the value is still unknown. The agent_id carried on synthorg_agent_identity_version_changes_total is an OpenMetrics exemplar (validated against the same snapshot), not a label, so per-agent attribution survives without per-agent series.

Info¶

Coordination (push-updated per multi-agent run)¶

Cost & budget (pull-refreshed at scrape)¶

Agents & tasks¶

Providers¶

Tools¶

API¶

Caches¶

Security¶

Audit chain¶

OTLP export health¶

Client transport¶

WebSocket transport¶

Database connection pool¶

Push queue + log shipping¶

Escalation + identity + workflow¶

Decisions¶

Configuration & MCP¶

Suggested PromQL queries¶

Saturation / backlog¶

# Escalation backlog (any department) sustained above 5 for 10m
max_over_time(synthorg_escalation_queue_depth[10m]) > 5

# Workflow p95 latency exceeds 60s
histogram_quantile(0.95, sum by (le) (rate(synthorg_workflow_execution_seconds_bucket[5m]))) > 60

Cost / budget¶

# Burned 80% of the monthly budget
synthorg_budget_used_percent > 80

# Total accumulated cost (per-agent breakdown lives in the REST cost API / logs)
synthorg_cost_total

Coordination health¶

# Coordination overhead sustained above 40% for 10 minutes
avg_over_time(synthorg_coordination_overhead_percent[10m]) > 40

# Coordination efficiency dropped below 0.5 (half of runs wasted)
avg_over_time(synthorg_coordination_efficiency[15m]) < 0.5

Identity lifecycle¶

# Rollback rate over the last hour (audit-relevant spike check)
sum(rate(synthorg_agent_identity_version_changes_total{change_type="rolled_back"}[1h]))

# Churn rate -- identity updates per minute
sum by (change_type) (rate(synthorg_agent_identity_version_changes_total[5m]))

API health¶

# 5xx rate as a fraction of total (clamp_min avoids NaN/Inf in idle windows)
sum(rate(synthorg_api_request_duration_seconds_count{status_class="5xx"}[5m]))
  / clamp_min(sum(rate(synthorg_api_request_duration_seconds_count[5m])), 1)

# Request rate by status class (histogram's auto-emitted _count series)
sum by (status_class) (rate(synthorg_api_request_duration_seconds_count[1m]))

# Error rate by RFC 9457 category
sum by (category) (rate(synthorg_api_error_classification_total[5m]))

# 5xx rate by category (internal vs rate_limit vs provider_error, etc.)
sum by (category) (rate(synthorg_api_error_classification_total{status_class="5xx"}[5m]))

Provider health¶

# Provider error rate per class (hot loop: rate_limit + timeout + connection)
sum by (provider, error_class) (rate(synthorg_provider_errors_total[5m]))

# Token-normalized provider error rate (error events per token volume)
sum by (provider) (rate(synthorg_provider_errors_total[5m]))
  / clamp_min(sum by (provider) (rate(synthorg_provider_tokens_total[5m])), 1)

# Provider call latency p95 by provider + call type
histogram_quantile(
  0.95,
  sum by (le, provider, call_type) (
    rate(synthorg_provider_call_duration_seconds_bucket[5m])
  )
)

# Provider call rate (the histogram's auto-emitted _count series)
sum by (provider, call_type) (rate(synthorg_provider_call_duration_seconds_count[5m]))

Cache hit rate¶

# Hit rate per cache (0.0-1.0)
sum by (cache_name) (rate(synthorg_cache_operations_total{outcome="hit"}[5m]))
  / clamp_min(sum by (cache_name) (rate(synthorg_cache_operations_total[5m])), 1)

# Eviction spike (may indicate undersized cache)
sum by (cache_name) (rate(synthorg_cache_operations_total{outcome="evict"}[5m]))

Autonomy promotion¶

# Promotion grant vs deny rate (governance throughput by outcome)
sum by (outcome) (rate(synthorg_autonomy_promotion_decisions_total[1h]))

# Denial fraction of all promotion decisions over the last day
sum(rate(synthorg_autonomy_promotion_decisions_total{outcome="denied"}[1d]))
  / clamp_min(sum(rate(synthorg_autonomy_promotion_decisions_total[1d])), 1)

Security posture¶

# Denial rate (should be low; spike indicates policy tightening or attack)
rate(synthorg_security_evaluations_total{verdict="deny"}[5m])

# Escalation rate per minute
rate(synthorg_security_evaluations_total{verdict="escalate"}[1m])

Audit chain health¶

# Append-error rate (non-zero = signing pipeline is broken)
rate(synthorg_audit_chain_appends_total{status="error"}[5m])

# Seconds since last append (flat line for > 5m is suspicious)
time() - synthorg_audit_chain_last_append_timestamp_seconds

# Audit log fill ratio (alert when the ring buffer is near capacity).
# At >0.9 the next bursts of activity overwrite the oldest entries
# before an operator can read them; rotate retention or archive.
synthorg_security_audit_log_fill_ratio

Audit log fill ratio¶

The synthorg_security_audit_log_fill_ratio gauge reports the occupancy of the in-memory security audit log as a fraction of its configured max_entries capacity. The log is a ring buffer: once full, the oldest entries are overwritten as new audit events land. A sustained value above 0.9 means the buffer is about to wrap; any unread evidence beyond that point is permanently lost.

Recommended alert rule:

- alert: SynthorgSecurityAuditLogNearCapacity
  expr: synthorg_security_audit_log_fill_ratio > 0.9
  for: 10m
  labels: {severity: warning}
  annotations:
    summary: "Security audit log is {{ $value | humanizePercentage }} full"
    runbook: "increase max_entries, archive entries to long-term storage, or shorten retention"

Grafana panel definition (drop into the Audit & Security row of monitoring/grafana/synthorg-overview.json):

{
  "title": "Security audit log fill ratio",
  "type": "gauge",
  "datasource": "${DS_PROMETHEUS}",
  "fieldConfig": {
    "defaults": {
      "min": 0,
      "max": 1,
      "unit": "percentunit",
      "thresholds": {
        "mode": "absolute",
        "steps": [
          {"color": "green", "value": null},
          {"color": "yellow", "value": 0.75},
          {"color": "red", "value": 0.9}
        ]
      }
    }
  },
  "targets": [
    {"expr": "synthorg_security_audit_log_fill_ratio", "refId": "A"}
  ]
}

OTLP export health¶

# Export-failure rate per kind
sum by (kind) (rate(synthorg_otlp_export_batches_total{outcome="failure"}[5m]))

# Dropped records per kind (queue overflow or retries exhausted)
sum by (kind) (rate(synthorg_otlp_export_dropped_records_total[5m]))

Client transport health¶

# Disconnect rate by transport (alert on transport_error spikes)
sum by (transport, reason) (rate(synthorg_client_disconnects_total[5m]))

# Transport-error rate as a fraction of all disconnects
sum(rate(synthorg_client_disconnects_total{reason="transport_error"}[5m]))
  / clamp_min(sum(rate(synthorg_client_disconnects_total[5m])), 1)

Grafana dashboard¶

Import monitoring/grafana/synthorg-overview.json into any Grafana v10+ instance. The file is Grafana v10-compatible dashboard JSON (authored against the v11 editor, which emits a schema readable by v10) with a single ${DS_PROMETHEUS} template variable bound to your Prometheus data source plus two filter variables: $workflow_definition_id and $department. There are no per-agent filter variables: per-agent metrics labels were removed for cardinality safety, so per-agent cost / task drill-downs come from the REST APIs and structured logs instead. The $department variable is sourced from synthorg_escalation_queue_depth's department label and is empty until the first escalation is recorded; the department-scoped panel shows "No data" until then.

The dashboard organises over fifty panels into thirteen rows. Only Health & SLO is expanded by default; expand the others as needed to keep the unfiltered view scannable.

To install via the Grafana UI: Dashboards → New → Import → Upload JSON file. Via the provisioning API: POST /api/dashboards/db with {"dashboard": <file>, "overwrite": true, "inputs": [...]}.

Alerts¶

The file does not ship alert rules because thresholds are deployment-specific. The suggested PromQL above is ready to drop into Prometheus' rules.yml; pair each query with a labels: severity: warning|critical and a for: duration. Example:

groups:
  - name: synthorg
    rules:
      - alert: SynthorgCoordinationOverheadHigh
        expr: avg_over_time(synthorg_coordination_overhead_percent[10m]) > 40
        for: 10m
        labels: {severity: warning}
        annotations:
          summary: "Coordination overhead is {{ $value }}%"
          runbook: "https://synthorg.io/docs/runbooks/coordination-overhead"

Logfire¶

Logfire's Prometheus integration can scrape the same /metrics endpoint directly; no additional wiring is required on the SynthOrg side. Follow the Logfire documentation for the Prometheus setup and point it at http://synthorg:8000/metrics. All metrics documented above will appear under the same names in Logfire dashboards.

Per-prompt-purpose alerts (operational)¶

These are operator-side operational alerts over your own structured-log / observability backend, in the same vein as the suggested PromQL rules above. They are NOT part of SynthOrg's opt-in product telemetry (which carries only anonymous aggregate signal and never per-entity cost), and they are NOT Prometheus rules: cost and latency sliced by prompt purpose (prompt_class_id, a PromptPurposeId) are deliberately not Prometheus labels, since a per-purpose metric label is the same cardinality hazard as a per-agent one.

The per-purpose data instead lives in the structured cost-recording log/span stream (each CostRecord carries prompt_class_id, latency_ms, cache_hit, retry_count) and the read-role GET /budget/prompt-class-breakdown REST endpoint that backs the dashboard's "Cost by prompt purpose" panel. Configure the two regression alerts as SQL queries (one row per prompt_class_id) in whatever log/observability backend you ship those structured logs to; run them on a rolling window and alert when a purpose's recent window diverges from its trailing baseline:

• Cost regression: a purpose's spend in the last hour exceeds its trailing 7-day hourly mean by the deployment's factor (start at 3x). Group by prompt_class_id; the alert payload names the drifting purpose so operators open the breakdown panel filtered to it.
• p95-latency regression: a purpose's p95 latency_ms in the last hour exceeds its trailing 7-day p95 by the deployment's factor (start at 2x). Group by prompt_class_id.

Thresholds are deployment-specific (low-traffic purposes are noisy on ratio alerts), so pin a minimum call-count floor per window before the ratio fires. Point both alerts at the same notification channel as the Prometheus rules above.

Further reading¶

• Observability design: sink layout, correlation IDs, per-domain routing
• Reference: errors: RFC 9457 error categories
• src/synthorg/observability/prometheus_collector.py: canonical metric registration
• src/synthorg/observability/prometheus_push_metrics.py: push-updated metric families
• src/synthorg/observability/prometheus_labels.py: bounded label value sets