Scoring hyperparameters¶

This page tracks the operator-tunable scoring weights and thresholds across the SynthOrg engine. Each entry lists the current default, the setting that controls it, and a rationale for the value. Where rationale reads "audit-set placeholder", the value carries no validated empirical derivation; it ships as a starting point and is subject to revision when an evaluation infrastructure for the relevant scoring path is in place.

This document is tracking-only. Hyperparameter improvements come from manual bug fixes, architectural changes, and prompt engineering; not from auto-tuning sweeps. See issue #1739 for the convention rule that lifted these values from bare numeric literals into settings.

Routing scorer (AgentTaskScorer)¶

The agent-task routing scorer assigns a 0-1 fitness score to every candidate agent for a given subtask. Sum of weights + bonuses is 1.1 with the tag bonus; the caller caps at 1.0.

Rationale. Audit-set placeholders calibrated so primary-skill overlap dominates (0.4) while role and seniority each independently push fit by 0.2. The tag bonus at 0.1 keeps tag-match a tiebreaker rather than a primary axis. min_score=0.1 filters out candidates that score on seniority alone (matches the seniority bonus). No empirical derivation; revisit when routing-decision telemetry is in place.

Model matcher (match_model)¶

Selects the best provider-model fit for a tier-bound ModelRequirement. Three score components: tier base + headroom + priority alignment.

Rationale. Audit-set placeholders chosen so tier match alone gives 0.5, headroom adds up to 0.25, and priority alignment adds up to 0.25. The 2.0 ratio cap means a model with twice the requested context gets the full headroom bonus; beyond that, more headroom is wasted on the priority axis. Balanced partial credit at 0.125 is half of priority_max_bonus. No empirical derivation; revisit alongside matcher-quality telemetry.

Heuristic quality grader (HeuristicRubricGrader)¶

Rule-based fallback grader used when no LLM grader is configured. Grades probes by checking whether each criterion's source text appears in the artifact payload (case-insensitive).

Rationale. Audit-set placeholders. Pass threshold of 0.5 means "at least half the probes match". Pass/fail grades of 0.8/0.3 give a clean PASS-vs-FAIL split that downstream consumers can threshold against. Confidence ceiling 0.9 acknowledges the heuristic is deterministic but not authoritative; bias 0.1 ensures every grading returns at least some confidence. Revisit when LLM-graded evaluations create comparison ground-truth.

Default client feedback (_build_default_client)¶

Synthetic feedback profile attached to default AIClient instances. The strictness multiplier scales a profile's strictness_level onto the 0-1 acceptance curve.

Rationale. Audit-set placeholders. Passing score of 0.5 sits at the midpoint, treating exactly-half-correct interactions as the boundary. Strictness multiplier of 2.0 means a profile with strictness_level=0.5 produces an effective multiplier of 1.0 (the "neutral" point); strictness_level=1.0 doubles sensitivity. The 0.1 floor keeps the multiplier non-zero so feedback weighting never collapses entirely. No empirical derivation; revisit when client simulation calibration data is available.

Limit / timeout settings (cluster #28)¶

Operator-tunable concurrency, retry, and shutdown budgets. The consumer modules (rate-limit stores, CAS retry handler, bus bridge, dispatcher, lifecycle stop sequence, fine-tune chunker) keep module-level constants that mirror these defaults so a service constructed without an explicit settings handle still observes the documented behaviour. Each fallback constant carries a # lint-allow: magic-numbers -- bootstrap fallback for <yaml_path> marker pointing at the canonical setting.

Rationale. Audit-set placeholders. Rate-limiter GC at 1024 acquires balances bookkeeping latency against stale-bucket retention under typical request volumes; the 60s horizon matches the common sliding-window default. CAS attempt budget of 2 (one retry) keeps mutation contention bounded without amplifying load on a hot row. Dispatcher 3 attempts with 0.1s base / 1.0s cap absorbs a transient NATS reconnect without pushing publish latency into the multi-second tail. VRAM table (40GB->128, 16GB->64, 8GB->32) is a conventional GPU-memory-to-batch heuristic for transformer fine-tunes; revisit when distinct GPU profiles surface. Lifecycle stage budgets sum to ~33s under the 40s drain ceiling, leaving ~7s of headroom so the outer hard deadline does not pre-empt normally-progressing stages yet still bounds stuck-stage tail latency; persistence and task-engine claim the largest slices because they own connection-pool drains and in-flight task finalisation respectively. Revisit when telemetry shows real-world shutdown timing distributions.