LLM Call Analytics and Coordination Metrics¶

Every LLM provider call is tracked with comprehensive metadata for financial reporting, debugging, and orchestration overhead analysis. These analytics drive the multi-agent tuning signals (orchestration ratio, coordination efficiency, error amplification) that complement the budget and cost controls.

Per-Call Tracking and Proxy Overhead Metrics¶

Every completion call produces a CompletionResponse with TokenUsage (token counts and cost). The engine layer creates a CostRecord (with agent/task context) and records it into CostTracker. The engine additionally logs proxy overhead metrics at task completion:

• turns_per_task: number of LLM turns to complete the task
• tokens_per_task: total tokens consumed
• cost_per_task: total cost in configured currency
• duration_seconds: wall-clock execution time
• prompt_tokens: estimated system prompt tokens
• prompt_token_ratio: ratio of prompt tokens to total tokens (overhead indicator; warns when >0.3)

These are natural overhead indicators; a task consuming 15 turns and 50k tokens for a one-line fix signals a problem. Metrics are captured in TaskCompletionMetrics, a frozen Pydantic model with a from_run_result() factory method.

Call Categorisation and Orchestration Ratio¶

When multi-agent coordination exists, each CostRecord is tagged with a call category:

The orchestration ratio (coordination / total) is surfaced in metrics and alerts. If coordination tokens consistently exceed productive tokens, the company configuration needs tuning (fewer approval layers, simpler meeting protocols, etc.).

coordination_metrics:
  enabled: false                       # opt-in -- enable for data gathering
  collect:
    - efficiency                       # cheap -- turn counting
    - overhead                         # cheap -- turn counting
    - error_amplification              # requires SAS baseline data
    - message_density                  # requires message counting infrastructure
    - redundancy                       # requires embedding computation on outputs
    - amdahl_ceiling                   # computed from parallelizable fraction
    - straggler_gap                    # computed from per-agent turn times
    - token_speedup_ratio              # computed from token usage + speedup
    - message_overhead                 # computed from pairwise message counts
  error_taxonomy:
    enabled: false                     # opt-in -- enable for targeted diagnosis
    categories:
      - logical_contradiction
      - numerical_drift
      - context_omission
      - coordination_failure

call_analytics:
  track:
    - call_category                    # productive, coordination, system, embedding
    - success                          # true/false
    - retry_count                      # 0 = first attempt succeeded
    - retry_reason                     # rate_limit, timeout, internal_error
    - latency_ms                       # wall-clock time for the call
    - finish_reason                    # stop, tool_use, max_tokens, error
    - cache_hit                        # prompt caching hit/miss (provider-dependent)
  aggregation:
    - per_agent_daily                  # agent spending over time
    - per_task                         # total cost per task
    - per_department                   # department-level rollups
    - per_provider                     # provider reliability and cost comparison
    - orchestration_ratio              # coordination vs productive tokens
  alerts:
    orchestration_ratio:
      info: 0.30                       # info if coordination > 30% of total
      warn: 0.50                       # warn if coordination > 50% of total
      critical: 0.70                   # critical if coordination > 70% of total
    retry_rate_warn: 0.1               # warn if > 10% of calls need retries

Coordination Error Taxonomy¶

When coordination metrics collection is enabled, the system classifies coordination errors into structured categories for targeted diagnosis. Each category supports a pluggable dual-implementation system: a cheap heuristic variant (regex/structural) and an optional LLM-backed semantic variant (accurate, expensive, disabled by default).

Detection scope: detectors operate at SAME_TASK (single execution) or TASK_TREE (parent + delegate executions via parent_task_id linkage). Cross-agent data is sanitized via sanitize_message before inclusion.

Pipeline architecture: detectors implement the Detector protocol and are discovered dynamically from ErrorTaxonomyConfig.detectors (a dict mapping ErrorCategory to per-category variant/scope config). When multiple variants target the same category, a CompositeDetector runs them concurrently and deduplicates findings by (turn_range, description_hash, category).

Downstream sinks: ClassificationSink protocol enables wiring findings into the performance tracker (PerformanceTrackerSink) and notification dispatcher (NotificationDispatcherSink, threshold-filtered).

Cost control: LLM semantic variants share the provider's rate limiter and track per-classification-run cost against classification_budget_per_task.

Error taxonomy classification runs post-execution (never blocks agent work) and logs structured events to the observability layer. Enable via coordination_metrics.error_taxonomy.enabled: true.

Error categories derived from Kim et al., 2025 and the Multi-Agent System Failure Taxonomy (MAST) by Cemri et al. (2025).