Skip to content

API Startup Lifecycle

The API boots in two phases. Construction (the create_app body) wires synchronous services. On-startup (_build_lifecycle.on_startup) wires services that need a connected persistence backend. The ordering invariants below are load-bearing: getting them wrong races a dependency and 503s a controller forever, or poisons startup on a transient boot.

Binary preflight

run_binary_preflight (api/lifecycle_helpers/binary_preflight.py) runs first in the construction phase, before anything is wired. The product shells out to a handful of binaries, and an image missing one boots cleanly and fails at the moment the feature is used, which for git is every single dispatch: workspace provisioning is on the critical path of every task and coordination.enable_workspace_isolation defaults on. The whole test suite is blind to it, because tests run where git is on PATH.

The manifest is declarative, one record per binary, and holds exactly the programs the backend needs and cannot obtain for itself. A missing one raises RequiredBinaryMissingError and aborts the boot, naming the binary, the apko package that provides it, and the subsystems it breaks. Refusing to start is the honest outcome: the alternative is a backend that accepts work it can never dispatch.

A record may also declare a min_version, because being on PATH is not the same as being able to do the job. git declares 2.48, the release that added worktree.useRelativePaths: git IGNORES a config key it does not know rather than refusing it, so an older binary accepts the option, reports nothing, and returns a worktree recording the backend's absolute path. The agent opens that worktree through a different mount, and every git command it runs fails with a "not a git repository" naming a path that exists on one side only. The version is read by asking the binary; a version that cannot be read is logged and allowed through, because not knowing is not evidence of an old one and refusing on a line it could not read would take a working deployment down over output the parser did not anticipate.

The probe is bounded at five seconds, so a binary that will not answer cannot hang the boot; it is treated exactly as one whose answer could not be read. It runs the path the presence check already resolved rather than the bare name, so the binary that answers is the one that was found: Windows searches the working directory ahead of PATH for a bare name, which would otherwise let the two halves disagree. The comparison runs over the shared prefix, so a longer version string still satisfies a shorter floor.

"Cannot be read" covers four cases, and the warning names which one, because they need different things done about them: the probe timed out (a wedged binary), it could not be spawned (one that vanished between the PATH lookup and the probe), its output carried no version (a gap in the parser), or it reported fewer components than the floor declares. The last is the one worth stating explicitly, since it arrives wearing a number: tuple ordering would call 2 lower than 2.48 and refuse the boot over a minor version nothing ever reported, so a version too short to compare is treated as unread rather than as old. Decoding is lenient for the same reason, as a strict decode raises a ValueError that no handler here catches and would crash the boot on the one path whose contract is that an unreadable version is survivable.

git is required always; pg_dump / pg_restore only when the configured backend is Postgres, which is why the preflight is handed the resolved backend name (empty when persistence has not resolved one, so only the backend-independent binaries are demanded).

A binary the backend provisions at runtime is deliberately absent. Both tunnel adapters download their vendor CLI on first start (integrations.tunnel.cloudflared_download_enabled / devtunnel_download_enabled, both on by default) and each answers availability() with the live state, including the case where that download is switched off. Checking PATH at boot would report a binary the adapter downloads on demand as missing, and duplicate a better, later report. The same reasoning excludes nix, which provisions inside the sandbox.

Nothing else is listed: there is no create_subprocess_shell under src/synthorg/, and the devcontainer image build goes through aiodocker rather than the docker CLI, so no docker entry can appear.

Construction-phase ordering invariants

  • agent_registry must be built BEFORE auto_wire_meetings.
  • auto_wire_meetings builds the orchestrator with NO protocol registry. The factories bake in organisation-wide strategy policy read from settings, which do not exist at construction time, so the meeting_protocol_registry subsystem installs them on the first reconcile pass (after auto_wire_settings composes the resolver) and installs a replacement when that policy changes. A meeting run before that pass raises MeetingProtocolNotFoundError naming the subsystem rather than running on boot defaults.
  • tunnel_provider is wired unconditionally (not gated by integrations.enabled).
  • message_bus.set_quadratic_alert_sink(DispatcherQuadraticAlertSink(...)) is called after the notification dispatcher is built, binding the in-memory bus's quadratic-fan-out enforcer to the dispatcher through the MessageBus protocol seam (the NATS backend is a no-op). This is a protocol call, not an isinstance(InMemoryMessageBus) + concrete-attr read.
  • ConflictResolutionService is built and installed on CommunicationStateSlice by wire_conflict_resolution_service (api/_comms_conflict_wiring.py): the hierarchy comes from the boot company snapshot and the human resolver reuses the already-wired escalation store/processor/registry. The shared LlmJudgeEvaluator is built here too, from the provider that serves the pinned CONFLICT_JUDGE model (resolve_feature_provider, not a naive first pick); a missing/non-serving provider leaves the judge unwired and the debate/hybrid resolvers fall back to authority.
  • _wire_meeting_conflict_bridge runs AFTER wire_conflict_resolution_service in the same pass (the bridge needs the built service) and AFTER auto_wire_meetings (it installs the bridge on the orchestrator via set_conflict_escalation_hook). It also stores the bridge on CommunicationStateSlice.conflict_escalation_bridge so the on-startup resolver rebind (_wire_resolver_dependents) can inject the persistence-backed resolver, making the meeting_conflict_escalation_enabled kill switch honour dashboard overrides. A no-op when either the orchestrator or the service is absent.
  • PeerDiscoveryClient is built from the peer registry + SSRF network validator and placed on A2aStateSlice.peer_discovery so the gateway's skills/query / skills/negotiate handlers can resolve learned peers.

On-startup ordering invariants

  • Durable security state wires failure-tolerantly after persistence connects, inside install_runtime_services via durability_wiring.py: _try_wire_audit_chain_persistence attaches a DurableAuditChainWriter to each live AuditChainSink (hydrate + verify + drain). It is idempotent and degrades to in-memory-only on failure, logging the dedicated API_AUDIT_CHAIN_PERSISTENCE_DEGRADED event.
  • The closed-loop EvalLoopCoordinator is built from the performance tracker + training service in wire_eval_loop and published on HrStateSlice; the periodic EvalLoopCycleScheduler that drives run_cycle is opt-in (hr.eval_loop_cycle_enabled) and re-reads hr.eval_loop_cycle_paused each tick.
  • The durable hiring pipeline is its own hiring_service subsystem: wire_hiring constructs the HiringService over the registry, the approval store, the settings resolver and the provider catalogue, publishing it on HrStateSlice.hiring_service. The resolver and the catalogue are threaded in unresolved, because both are read when an approval is raised, so a provider configured after boot is offered on the next hire with no restart. Attachment is a hard prerequisite: it calls HiringService.attach_persistence(request_repo=persistence_of(app_state).hiring_requests) then hydrate() so an approved hire survives a restart between approval and instantiation, and a failed attach aborts wiring rather than publishing a non-durable pipeline. Only hydration is isolated, so a hydration fault leaves in-flight requests for the next boot instead of refusing to start. A missing collaborator raises SubsystemDeclinedError naming which, so GET /subsystems can report it.
  • _drain_resume_intents runs LAST in _run_startup, after _wire_approval_gate and the worker-execution-service install hook, because a re-dispatch routes through both. It finishes any approval the previous process decided but died before resuming (see chat-inbound.md); non-fatal, so a drain fault leaves every marker for the next startup to retry rather than refusing to boot.

  • The observability bridge config arm is applied alongside the notification dispatcher: ObservabilityBridgeSettingsSubscriber watches the observability.* keys and recomposes get_observability_bridge_config() so console-level / telemetry-enabled edits take effect via the settings dispatcher.

  • SettingsService auto-wire must precede WorkflowExecutionObserver registration, so it picks up the resolver-driven max_subworkflow_depth instead of the seed default.

  • OntologyService wires after persistence.connect() via _wire_ontology_service.
  • Cost-dial services (BudgetConfig, CostForecastRepository, BenchmarkScoreRepository, the budget.benchmark_provider-selected BenchmarkScoreProvider, CostForecaster, ParetoAnalyzer) wire via _try_wire_cost_dial AFTER persistence connects. It is failure-tolerant (logs an API_APP_STARTUP warning and the controllers 503 if it fails or persistence is absent) and idempotent (skips when already wired), so a transient shared-app boot does not poison startup. The benchmark provider, repo, and ParetoAnalyzer's ModelCapabilityMap are built in api/_benchmark_wiring.py (stub default / measured; seed_benchmark_scores boot-seeds the measured arm from the committed benchmark_seed.json). The approved forecast's forecast_id + ceiling_amount are stamped onto the Task in the work pipeline's intake phase (WorkPipelineService._link_forecast) so the in-loop BudgetChecker enforces the per-brief ceiling and the engine can stamp halt context for the resume banner.
  • The durable memory backend wires via wire_memory_backend (in api/lifecycle_helpers/memory_backend_wiring.py) AFTER persistence connects but BEFORE _install_runtime_services: _construct_agent_engine reads MemoryStateSlice.backend eagerly, so a backend wired later would never reach an agent. It resolves the embedder (resolve_embedder_config) then builds create_memory_backend and connects; when no embedding model resolves it wires NO backend and logs at ERROR rather than silently falling back to the ephemeral keyword store (fail-loud). The same hook starts the consolidation scheduler (_wire_consolidation_scheduler), which is skipped when the interval is never; org memory is its own org_memory_backend subsystem, declared separately below. The embedder settings (memory.embedder_*) and memory.backend are declared on the memory_backend subsystem with rebuild_on_change=True: the backend bakes them in at connect time, so a write to one tears the instance down and builds a replacement rather than leaving stored vectors at an incomparable width.
  • Knowledge substrate wires via wire_knowledge_engine AFTER persistence connects; failure-tolerant and ghost-wired (built regardless of knowledge.enabled), gated only on has_persistence AND has_memory_backend (logs an API_APP_STARTUP warning and the knowledge controllers + MCP handlers 503 if either is absent), so a missing memory backend in dev does not poison startup. knowledge.enabled (Cat-1, default on, hot) is not a wiring gate; it is enforced live per request at the knowledge controllers + MCP handlers via the config resolver, so toggling it takes effect on the next request with no restart.
  • EnvironmentService (per-project reproducible environments) wires in _install_runtime_services behind has_persistence and is threaded into AgentEngineExecutionService via build_runtime_services; the worker provisions ambiently (ActiveSandboxEnvironment contextvar) before the engine run, so a missing workspace logs ENVIRONMENT_PROVISION_SKIPPED rather than silently dropping the declared env.
  • Mid-flight steering splits its wiring in two by dependency: the steering INBOX (read path) is built from persistence.project_brain and injected into the boot AgentEngine in the runtime-services step (persistence-only, memory-independent list_current projection), while the steering SERVICE (write path) wires in _wire_steering_service AFTER _wire_project_brain (memory-gated brain) via partial app_state.wire(CockpitStateSlice, ...) (NOT swap_slice, so the construction-phase steering_notifier and the later steering_service coexist on the slice). Wiring the service inside _wire_cockpit_services would race the brain and 503 forever.
  • The red-team report repo is published on SecurityStateSlice.red_team_reports during _install_runtime_services (decoupled from the review gate, via partial app_state.wire), and _wire_deliverable_receipts reads it so a receipt's red_team section degrades to empty rather than erroring when the subsystem is off.
  • agent_workspace_root is resolved via resolve_agent_workspace_root_env() INSIDE the _install_runtime_services startup closure (its install guard runs it once), NOT at construction: the resolver fail-fasts on a non-absolute SYNTHORG_DB_PATH, and that must not crash pure app construction (dev / :memory: / OpenAPI export, which build the app without running startup). The resolved Path | None is injected into install_runtime_services rather than re-read from the environment inside it.
  • _apply_telemetry_db_layer runs immediately BEFORE telemetry_collector.start. The collector is built pre-persistence (env > default), so this hook re-resolves telemetry.enabled with full DB > env > default precedence and calls telemetry_collector.apply_enabled(enabled=...) before start() reads it, so a DB /settings toggle takes effect on restart. Failure-tolerant: a None resolver or a resolution failure logs an API_APP_STARTUP warning and leaves the construction-time value untouched.
  • The construction-phase notification dispatcher is started by _start_construction_dispatcher ONLY when it is still the live dispatcher on NotificationsStateSlice. The bridge-config startup step (_apply_notification_dispatcher_config, run via _apply_bridge_config) may have rebuilt a replacement with DB-resolved timeouts, started it, and swapped it onto the slice, leaving the construction dispatcher orphaned; starting that orphan would open sinks nothing routes through and that on-shutdown (which closes the live slice dispatcher) never tears down. On a hot config-apply, _apply_notification_dispatcher_config starts the new dispatcher BEFORE swapping it in (a failed start aborts the swap and keeps the running dispatcher, and a CancelledError or non-critical failure mid-start closes the partially-started one), so no built-but-unstarted dispatcher silently drops events.
  • The JWT secret is resolved (resolve_jwt_secret) and the dev-auth-bypass flag (resolve_dev_auth_bypass, env SYNTHORG_DEV_AUTH_BYPASS) BEFORE persistence.migrate(), so a missing secret fails fast. SYNTHORG_DEV_AUTH_BYPASS is DEV-ONLY and defaults False: when on it stamps AuthConfig.dev_auth_bypass=True, adds POST /auth/dev-login to the auth-exclude set, and emits a prominent API_APP_STARTUP security warning. That endpoint mints a real admin session for the existing CEO with no password (404s when the flag is off, 401s when no admin exists, never creates one). It MUST never be enabled in production. The frontend dev bootstrap (web/.env's VITE_DEV_AUTH_BYPASS, only honoured under import.meta.env.DEV) calls it on load so the local Vite dashboard auto-logs-in.

Late-startup gated wiring hooks

Every hook below is declared in api/subsystems/registry.py and run by the reconciler, not sequenced here; the notes record what each one needs and what it degrades to. The requires line in the declaration is the authority. See subsystem-reconciliation.md.

  • wire_quota_poller: gated budget.quota_poller_enabled + connected persistence + wired quota_tracker; appended AFTER _wire_features; idempotent; stopped in lifecycle_runner_shutdown.
  • wire_risk_override_service: gated on a TieredTimeoutConfig + persistence + scheduler; rebuilds the tiered policy's classifier wrapped in SecOpsRiskClassifier seeded from the durable override repo and swaps it via ApprovalTimeoutScheduler.set_timeout_policy; publishes RiskOverrideService on SecurityStateSlice; controllers + MCP 503 when unwired.
  • wire_org_memory_backend: declared as the org_memory_backend subsystem (requires persistence); failure-tolerant connect(); publishes MemoryStateSlice.org_memory_backend consumed by offboarding + ontology sync, degrading to None when absent.
  • _wire_task_activity_observer: registers the TaskActivityObserver on the TaskEngine in the same try block as _wire_workflow_observer, gated on task engine + a live ChannelsPlugin (threaded via MessageBusBridge.plugin) + a wired HrStateSlice.performance_tracker; idempotent (checks _observers for an existing instance). Publishes every persisted task transition to CHANNEL_TASKS (dashboard live-activity feed) and records a TaskMetricRecord (org health + completions sparkline) only for a classifiable, attributable executed terminal run (fresh into a TERMINAL_RUN_STATES member, with a resolved RunOutcome and an assignee); a run whose outcome cannot be resolved (artifact-count fault) or that carries no assignee, and FSM-terminal-but-never-ran states (CANCELLED/REJECTED), are published but not recorded, so unclassifiable runs never skew health. Logs API_SERVICE_AUTO_WIRED on success and API_APP_STARTUP on each skip; a missing prerequisite disables the live feed rather than failing boot.
  • wire_project_rollup_service: declared as the project_rollup_service subsystem (requires persistence + task engine); idempotent via EngineStateSlice.project_rollup_service. Registers ProjectRollupService.on_task_state_changed as a TaskEngine observer so a plan and its project advance from the tasks implementing them. TaskEngine re-reads _observers per mutation, so registering after start() is honoured. Failure-tolerant: a missing prerequisite leaves initiative status static rather than failing boot. The slice field is written only to serve the idempotency guard; the service runs off the observer, not slice resolution.
  • attach_integration_stage / attach_evaluation_stage / attach_replan_trigger / attach_stall_escalation / attach_ship_retro_capture: the initiative tail, declared as five subsystems (initiative_integrate, initiative_evaluate, initiative_replan, initiative_stall_escalation, initiative_retro_capture), each separate from the rollup above and each requiring only what that one collaborator needs: the work pipeline, the provider + agent registries, the coordinator, the approval store, and both memory layers respectively. Each attaches onto the already-registered rollup, so the rollup keeps its early activation and its useful tailless work while the tail waits. Liveness is read one probe per collaborator (has_integration() / has_evaluation() / has_replan_trigger() / has_stall_escalation() / has_retro_capture()). The escalation is separate from the replan trigger rather than folded into it because it is what runs when the trigger is absent or refusing, so a boot with one must not read as covering the other. Splitting them off the rollup is what makes the tail reachable at all: the stages all decline during the rollup's own activation (the provider registry is built ~190 ms later), and the reconciler short-circuits on an already-active subsystem. Splitting them from each other is what makes the degradation table in initiative-tail.md true: one spec over the union of their requirements left a coordinator-less boot with no integrate stage either. The retro capture alone declares a teardown and rebuild_on_change, because the memory backends it holds are replaceable while the process runs.
  • _maybe_start_health_prober: the FIRST background service started in _start_background_services, gated on a wired health_tracker + config_resolver (the provider-management service is optional and only supplies the SSRF discovery policy). It builds the ProviderHealthProber, start()s it, and only then hands it back to ProviderManagementService.set_probe_requester so a provider created after boot is probed on write instead of sitting unknown until the next cycle (providers.health_probe_interval_seconds, re-read per cycle). The hand-off is deliberately after start(): a prober that never started must not become the probe requester. Failure-tolerant (critical errors re-raise, everything else logs an API_APP_STARTUP warning and returns None), so a prober fault leaves health derived from real call outcomes rather than refusing to boot; lifecycle_runner_shutdown stops it.
  • HealthMonitoringPipeline (judge + triage + dispatcher) is built in build_runtime_services and threaded into AgentEngineExecutionService (health_pipeline= / health_enabled=), re-reading engine.health_monitoring_enabled per run; absent dispatcher means no pipeline.

Controller-facing read services

  • ConversationalResumeService lives in meta/chief_of_staff/ (NOT api/services/, so the early api_core_state import chain never pulls the communication enum modules and trips the cold-import cycle gate), is published on MetaStateSlice by _wire_conversational_repositories_and_reconcile, and is deliberately UNGATED (wraps only the persistence repos, never the toggle-gated Chief-of-Staff feature services) so a decided conversational approval still resolves after its feature switches off.
  • The single WorkflowExecutionService (_wire_workflow_observer) caches the config_resolver and resolves max_subworkflow_depth per call so settings hot-reload is preserved (constructing with a resolved int would freeze the seed default; a None resolver logs a warning and falls back to EngineBridgeConfig().max_subworkflow_depth).
  • _wire_webhook_request_services wires webhook_replay_protector unconditionally and webhook_activity_service only on connected persistence. Idempotency keys on mutating endpoints MUST bind the resource id (f"{resource_id}:{idempotency_key}") so a reused key cannot collide across resources.

Self-extending toolkit (toolsmith)

Autonomous detection wires in wire_toolsmith AFTER install_dynamic_tool_layer (gated on self_improvement.tool_creation_enabled + provider + connected persistence): install_capability_gap_sink(runtime.service) routes every capability_gap MCP envelope into the gap store (fire-and-forget; write failure logs via safe_error_description, never an unhandled task-exception traceback), and a ToolsmithCycleScheduler (cadence toolsmith.cycle_interval_seconds, floor 60s) drives run_cycle periodically with a meta.toolsmith_cycle_paused kill-switch (re-read each tick, fail-safe to enabled). The ToolsmithStateSlice carries service + cycle_scheduler as a both-or-neither paired invariant; lifecycle_runner_shutdown stops the scheduler. The cycle only PROPOSES (human approval still gates apply).

Periodic model-refresh

Wires in wire_model_refresh (_wire_model_refresh, AFTER wire_toolsmith), gated on providers.model_refresh_mode != off AND a built provider-management service AND connected persistence (off-by-default, so a normal boot skips it). The cadence modes (detect_only / reconcile_recommend) build + start() a ModelRefreshScheduler BEFORE publishing ModelRefreshStateSlice and roll back (no publish) if start fails; manual_only wires the on-demand service with no scheduler. The slice's both-or-neither invariants: a scheduler implies a service, and a service implies a recommendation_repo. lifecycle_runner_shutdown stops the scheduler. The scheduler re-reads the live mode + auto-apply flag each tick (fail-safe to off), and stop() does NOT reset its lifecycle lock/event to None (it passes reset_primitives_on_stop=False to the shared AsyncCycleScheduler base: clearing them under the held lock opens a rebind race).

Runtime tool-call failure feedback

Wires in wire_tool_call_feedback (AFTER _wire_model_refresh), gated on a built provider-management service AND connected persistence (NOT on providers.tool_call_feedback_enabled: the ToolCallFeedbackTracker re-reads that setting live per observation so the feature toggles without a restart while the cheap sink stays installed). Failure-tolerant + idempotent: a missing resolver / management / persistence logs an API_APP_STARTUP warning and returns (never an invisible skip). The tracker is published on ToolCallFeedbackStateSlice, then install_tool_call_signal_sink(tracker) runs LAST so a partial build leaves no dangling sink; lifecycle_runner_shutdown uninstalls the sink + clears the slice. The tracker keeps its instance lock OFF the capability-writer call (the writer takes ProviderManagementService._lock), so the two locks never nest. The matcher hard-fails every agent on tool_calls_verified is False (authoritative over the optimistic supports_tools path, and unconditional because tool calling is a floor rather than a per-role requirement); a genuine tool-call success only re-enables a truly-downgraded (False) model, never promotes an untested (None) one.

Stakes-gated red-team completion

ReviewGateService reads the red-team threshold off the shared CapabilityPolicy per decision (set_capability_policy, wired in startup_steps.py right after the deliverable-input builder), so a write to engine.red_team_min_stakes takes effect on the next completion rather than at the next restart; with no policy wired it falls back to the declared default (HIGH). The gate fires only for task.stakes >= red_team_min_stakes; a below-threshold completion logs RED_TEAM_GATE_SKIPPED and dispatch_completion runs it INLINE (a missing task also runs inline so complete_review raises TaskNotFoundError synchronously) rather than deferring a no-op to the background.

Self-improvement rollback executor

Wires in api/lifecycle_helpers/meta_apply_wiring.py::_wire (failure-tolerant + idempotent, off by default with the self-improvement feature): _build_rollback_executor constructs the six mutators (config/prompt/principle-removal/architecture/code unconditional, branch gated) and threads the executor into SelfImprovementService, which dispatches the applier-MATERIALISED inverse operations (carried on RolloutResult.applied_rollback_operations, NOT the proposal's static plan) on a post-rollout regression, flipping REGRESSED to ROLLED_BACK. The branch/revert_branch handler is gated on code_modification.github_token + github_repo (absent means branch revert omitted, matching the applier's credential gating); github_api_url is https-validated (token rides the Authorization header). A materialised op with no registered handler logs ERROR reason="unregistered_operation_type" and keeps REGRESSED (never a silent swallow).

Runtime services selection

synthorg.workers.runtime_builder.build_runtime_services selects behind ONE provider-present switch and returns a RuntimeServices pair (worker execution service + multi-agent coordinator) built from a SINGLE shared boot AgentEngine: AgentEngineExecutionService + a build_coordinator(...) coordinator with a provider, or NoProviderExecutionService + None coordinator as the empty-company backstop. The provider-present arm validates coordination.decomposition_model is non-blank before building the coordinator and raises CoordinationConfigError at boot when it is empty (the setting ships blank, so an operator must set a model id from their catalogue). The _install_runtime_services boot hook installs both via the AppState.worker_execution_service and AppState.coordinator seams; it is appended FIRST after the persistence/SettingsService hooks so the once-only set_worker_execution_service and if-absent set_coordinator_if_absent seams cannot lose the race with the worker property's lazy LifecycleAdvancingExecutionService default. Empty-company rejects task creation at the controller (AgentRuntimeNotConfiguredError, 4014) and /coordinate honestly 503s (no coordinator). swap_worker_execution_service / swap_coordinator / swap_provider_registry hold a lock (synchronised against lazy reads).

Setup completion

post_setup_reinit() (provider reload, agent bootstrap, AND runtime-services rebuild + dual hot-swap of the worker execution service and coordinator, defined in src/synthorg/api/controllers/setup/_runtime_wiring.py) propagates failures, and settings_svc.set("api", "setup_complete", "true") only runs if reinit returns clean. The whole check/validate/reinit/persist sequence is serialised under COMPLETE_LOCK (also in _runtime_wiring.py) so two concurrent /setup/complete requests cannot race on the flag write. A half-configured runtime presenting itself as "complete" is worse than a clear error the operator can retry after fixing the underlying provider config.