Monitoring & Dashboards¶

SynthOrg exposes runtime telemetry via a Prometheus /metrics endpoint plus structured JSON logs. This guide walks through every metric the application emits, a ready-to-import Grafana dashboard, and suggested alert rules. The canonical metric registration lives in src/synthorg/observability/prometheus_collector.py (pull-refreshed families) and src/synthorg/observability/prometheus_push_metrics.py (push-updated families); bounded label allowlists live in src/synthorg/observability/prometheus_labels.py.

Scraping¶

Point any Prometheus-compatible scraper at the running app:

scrape_configs:
  - job_name: synthorg
    scrape_interval: 30s
    static_configs:
      - targets: ['synthorg:8000']

The endpoint is unauthenticated by default; put it behind your normal scrape-ACL (firewall, sidecar proxy, Kubernetes NetworkPolicy). All metric names are prefixed with synthorg_.

Metric inventory¶

The Dashboard column maps each metric to the row it appears under in the default Grafana overview dashboard (monitoring/grafana/synthorg-overview.json). Rows are collapsible; the only row expanded by default is Health & SLO. A metric marked n/a (scrape-only) is exposed on /metrics and queryable in Prometheus but has no dedicated panel in the bundled dashboard yet. The dashboard exposes one filter variable ($workflow_definition_id) that drills panels down per-entity. No per-agent label is exposed: an unbounded agent_id / agent label is a cardinality bomb, so per-agent cost and task breakdowns live in the structured logs and the REST cost / task APIs rather than in metrics. Default queries aggregate across the full set so the unfiltered view is always meaningful.

Bounded-label values are enforced at record time in src/synthorg/observability/prometheus_labels.py; PromQL filters that reference values outside those allowlists will never match data. The five registry-bound push-time label names (agent_id, workflow_definition_id, tool_name, provider, model_id on the metrics noted below) are validated against a registry-bound snapshot rebuilt on every Prometheus scrape; unknown values drop that one sample with a metrics.scrape.failed WARN log per unknown label per scrape. The log repeats on the next scrape if the value is still unknown. The agent_id carried on synthorg_agent_identity_version_changes_total is an OpenMetrics exemplar (validated against the same snapshot), not a label, so per-agent attribution survives without per-agent series.

Info¶

Coordination (push-updated per multi-agent run)¶

Cost & budget (pull-refreshed at scrape)¶

Agents & tasks¶

Providers¶

Tools¶

API¶

Caches¶

Security¶

Audit chain¶

OTLP export health¶

Client transport¶

WebSocket transport¶

Database connection pool¶

Push queue + log shipping¶

Identity + workflow¶

Decisions¶

Configuration & MCP¶

Suggested PromQL queries¶

Saturation / backlog¶

# Workflow p95 latency exceeds 60s
histogram_quantile(0.95, sum by (le) (rate(synthorg_workflow_execution_seconds_bucket[5m]))) > 60

Cost / budget¶

# Burned 80% of the monthly budget (gate on full measurability -- see below)
synthorg_budget_used_percent > 80 and ignoring(state) synthorg_budget_spend_measurability{state="measured"} == 1

# Total accumulated cost (per-agent breakdown lives in the REST cost API / logs)
synthorg_cost_total

synthorg_budget_used_percent reads 0 whenever the period's spend is not fully metered (any flat-rate connection in the mix, or nothing metered at all), so an unqualified > N alert can silently never fire on such a deployment. Gate it on synthorg_budget_spend_measurability{state="measured"} == 1 as above, or watch that gauge directly to catch a deployment where the percentage has gone structurally unmeasurable. The ignoring(state) is load-bearing, and on() is the wrong tool here: the two metrics differ only by the state label, so plain and matches nothing and the alert never fires however far over budget the deployment runs. on() would fix that by matching on the empty label set, which also discards job and instance, so one target reporting measured would keep every other target's series. ignoring(state) drops the one label that differs and keeps target identity, so each target is gated on its own synthorg_budget_spend_measurability series.

Coordination health¶

# Coordination overhead sustained above 40% for 10 minutes
avg_over_time(synthorg_coordination_overhead_percent[10m]) > 40

# Coordination efficiency dropped below 0.5 (half of runs wasted)
avg_over_time(synthorg_coordination_efficiency[15m]) < 0.5

Identity lifecycle¶

# Rollback rate over the last hour (audit-relevant spike check)
sum(rate(synthorg_agent_identity_version_changes_total{change_type="rolled_back"}[1h]))

# Churn rate -- identity updates per minute
sum by (change_type) (rate(synthorg_agent_identity_version_changes_total[5m]))

API health¶

# 5xx rate as a fraction of total (clamp_min avoids NaN/Inf in idle windows)
sum(rate(synthorg_api_request_duration_seconds_count{status_class="5xx"}[5m]))
  / clamp_min(sum(rate(synthorg_api_request_duration_seconds_count[5m])), 1)

# Request rate by status class (histogram's auto-emitted _count series)
sum by (status_class) (rate(synthorg_api_request_duration_seconds_count[1m]))

# Error rate by RFC 9457 category
sum by (category) (rate(synthorg_api_error_classification_total[5m]))

# 5xx rate by category (internal vs rate_limit vs provider_error, etc.)
sum by (category) (rate(synthorg_api_error_classification_total{status_class="5xx"}[5m]))

Provider health¶

# Provider error rate per class (hot loop: rate_limit + timeout + connection)
sum by (provider, error_class) (rate(synthorg_provider_errors_total[5m]))

# Token-normalised provider error rate (error events per token volume)
sum by (provider) (rate(synthorg_provider_errors_total[5m]))
  / clamp_min(sum by (provider) (rate(synthorg_provider_tokens_total[5m])), 1)

# Provider call latency p95 by provider + call type
histogram_quantile(
  0.95,
  sum by (le, provider, call_type) (
    rate(synthorg_provider_call_duration_seconds_bucket[5m])
  )
)

# Provider call rate (the histogram's auto-emitted _count series)
sum by (provider, call_type) (rate(synthorg_provider_call_duration_seconds_count[5m]))

Cache hit rate¶

# Hit rate per cache (0.0-1.0)
sum by (cache_name) (rate(synthorg_cache_operations_total{outcome="hit"}[5m]))
  / clamp_min(sum by (cache_name) (rate(synthorg_cache_operations_total[5m])), 1)

# Eviction spike (may indicate undersized cache)
sum by (cache_name) (rate(synthorg_cache_operations_total{outcome="evict"}[5m]))

Autonomy promotion¶

# Promotion grant vs deny rate (governance throughput by outcome)
sum by (outcome) (rate(synthorg_autonomy_promotion_decisions_total[1h]))

# Denial fraction of all promotion decisions over the last day
sum(rate(synthorg_autonomy_promotion_decisions_total{outcome="denied"}[1d]))
  / clamp_min(sum(rate(synthorg_autonomy_promotion_decisions_total[1d])), 1)

Security posture¶

# Denial rate (should be low; spike indicates policy tightening or attack)
rate(synthorg_security_evaluations_total{verdict="deny"}[5m])

# Escalation rate per minute
rate(synthorg_security_evaluations_total{verdict="escalate"}[1m])

Audit chain health¶

# Append-error rate (non-zero = signing pipeline is broken)
rate(synthorg_audit_chain_appends_total{status="error"}[5m])

# Seconds since last append (flat line for > 5m is suspicious)
time() - synthorg_audit_chain_last_append_timestamp_seconds

# Audit log fill ratio (alert when the ring buffer is near capacity).
# At >0.9 the next bursts of activity overwrite the oldest entries
# before an operator can read them; rotate retention or archive.
synthorg_security_audit_log_fill_ratio

Audit log fill ratio¶

The synthorg_security_audit_log_fill_ratio gauge reports the occupancy of the in-memory security audit log as a fraction of its configured max_entries capacity. The log is a ring buffer: once full, the oldest entries are overwritten as new audit events land. A sustained value above 0.9 means the buffer is about to wrap; any unread evidence beyond that point is permanently lost.

Recommended alert rule:

- alert: SynthorgSecurityAuditLogNearCapacity
  expr: synthorg_security_audit_log_fill_ratio > 0.9
  for: 10m
  labels: {severity: warning}
  annotations:
    summary: "Security audit log is {{ $value | humanizePercentage }} full"
    runbook: "increase max_entries, archive entries to long-term storage, or shorten retention"

Grafana panel definition (drop into the Audit & Security row of monitoring/grafana/synthorg-overview.json):

{
  "title": "Security audit log fill ratio",
  "type": "gauge",
  "datasource": "${DS_PROMETHEUS}",
  "fieldConfig": {
    "defaults": {
      "min": 0,
      "max": 1,
      "unit": "percentunit",
      "thresholds": {
        "mode": "absolute",
        "steps": [
          {"color": "green", "value": null},
          {"color": "yellow", "value": 0.75},
          {"color": "red", "value": 0.9}
        ]
      }
    }
  },
  "targets": [
    {"expr": "synthorg_security_audit_log_fill_ratio", "refId": "A"}
  ]
}

OTLP export health¶

# Export-failure rate per kind
sum by (kind) (rate(synthorg_otlp_export_batches_total{outcome="failure"}[5m]))

# Dropped records per kind (queue overflow or retries exhausted)
sum by (kind) (rate(synthorg_otlp_export_dropped_records_total[5m]))

Client transport health¶

# Disconnect rate by transport (alert on transport_error spikes)
sum by (transport, reason) (rate(synthorg_client_disconnects_total[5m]))

# Transport-error rate as a fraction of all disconnects
sum(rate(synthorg_client_disconnects_total{reason="transport_error"}[5m]))
  / clamp_min(sum(rate(synthorg_client_disconnects_total[5m])), 1)

Grafana dashboard¶

Import monitoring/grafana/synthorg-overview.json into any Grafana v10+ instance. The file is Grafana v10-compatible dashboard JSON (authored against the v11 editor, which emits a schema readable by v10) with a single ${DS_PROMETHEUS} template variable bound to your Prometheus data source plus one filter variable: $workflow_definition_id. There are no per-agent filter variables: per-agent metrics labels were removed for cardinality safety, so per-agent cost / task drill-downs come from the REST APIs and structured logs instead.

The dashboard organises over fifty panels into thirteen rows. Only Health & SLO is expanded by default; expand the others as needed to keep the unfiltered view scannable.

To install via the Grafana UI: Dashboards → New → Import → Upload JSON file. Via the provisioning API: POST /api/dashboards/db with {"dashboard": <file>, "overwrite": true, "inputs": [...]}.

Alerts¶

The file does not ship alert rules because thresholds are deployment-specific. The suggested PromQL above is ready to drop into Prometheus' rules.yml; pair each query with a labels: severity: warning|critical and a for: duration. Example:

groups:
  - name: synthorg
    rules:
      - alert: SynthorgCoordinationOverheadHigh
        expr: avg_over_time(synthorg_coordination_overhead_percent[10m]) > 40
        for: 10m
        labels: {severity: warning}
        annotations:
          summary: "Coordination overhead is {{ $value }}%"
          runbook: "https://synthorg.io/docs/runbooks/coordination-overhead"

Logfire¶

Logfire's Prometheus integration can scrape the same /metrics endpoint directly; no additional wiring is required on the SynthOrg side. Follow the Logfire documentation for the Prometheus setup and point it at http://synthorg:8000/metrics. All metrics documented above will appear under the same names in Logfire dashboards.

Per-prompt-purpose alerts (operational)¶

These are operator-side operational alerts over your own structured-log / observability backend, in the same vein as the suggested PromQL rules above. They are NOT part of SynthOrg's opt-in product telemetry (which carries only anonymous aggregate signal and never per-entity cost), and they are NOT Prometheus rules: cost and latency sliced by prompt purpose (prompt_class_id, a PromptPurposeId) are deliberately not Prometheus labels, since a per-purpose metric label is the same cardinality hazard as a per-agent one.

The per-purpose data instead lives in the structured cost-recording log/span stream (each CostRecord carries prompt_class_id, latency_ms, cache_read_input_tokens, cache_write_input_tokens, retry_count) and the read-role GET /budget/prompt-class-breakdown REST endpoint that backs the dashboard's "Cost by prompt purpose" panel. Configure the two regression alerts as SQL queries (one row per prompt_class_id) in whatever log/observability backend you ship those structured logs to; run them on a rolling window and alert when a purpose's recent window diverges from its trailing baseline:

• Cost regression: a purpose's spend in the last hour exceeds its trailing 7-day hourly mean by the deployment's factor (start at 3x). Group by prompt_class_id; the alert payload names the drifting purpose so operators open the breakdown panel filtered to it.
• p95-latency regression: a purpose's p95 latency_ms in the last hour exceeds its trailing 7-day p95 by the deployment's factor (start at 2x). Group by prompt_class_id.

Thresholds are deployment-specific (low-traffic purposes are noisy on ratio alerts), so pin a minimum call-count floor per window before the ratio fires. Point both alerts at the same notification channel as the Prometheus rules above.

Further reading¶

• Observability design: sink layout, correlation IDs, per-domain routing
• Reference: errors: RFC 9457 error categories
• src/synthorg/observability/prometheus_collector.py: canonical metric registration
• src/synthorg/observability/prometheus_push_metrics.py: push-updated metric families
• src/synthorg/observability/prometheus_labels.py: bounded label value sets