16 — Operations

L4 on a live plant is a pinned release, a dual-family model slot, and a short list of rates somebody watches. Humans still decide and execute. Ops owns availability and promotion discipline; it does not own money, hard stops, or card authority.

---

Purpose#

Say how L4 is deployed, what a release pins, how shadow and canary work, which rates on-call watches, and how the plant path stays separate from the offline improvement path.

---

Decisions#

Rejected: floating “latest model” deploys; council models on the live path; plant-scoped silent registry edits without lockfile; treating withhold rate as a model-quality KPI alone.

Would change this: a certified air-gap profile that requires a different pin ceremony; measured evidence that canary windows are too short or too long for Pilot plants.

---

Deploy profiles#

Ports (ModelSlot, MemoryPort, L3MethodsPort, CardSink, L4Store) stay stable across profiles. Providers and residency change; stage graph and kernel do not.

v1 slice: Pilot plants run cloud. Local modes stay design-portable; no Pilot claim that air-gap L4 is proven until a plant runs it.

---

Release lockfile#

One hash pins, at minimum:

• kernel version;
• stage-graph version;
• every registry entry version in scope (domains, families, workflows, stages, analyses, patterns, constraint kinds, tools, prompts, memory missions, ranking policy, soft-gate thresholds);
• model pins (provider, family id, revision or weight digest);
• L3 methods port / shared registry pack versions consumed at the L3–L4 seam.

Every DecisionTraceAlways-on record: observed, context, action, policy, approval, outcome (and seam decisions) carries this hash (12-trace-and-eval.md). Replay of an old run uses the lockfile that was live then, not today’s tip.

Plant overrides layer on a global entry with explicit precedence and appear in the same lockfile digest for that plant.

---

Model pins#

Pins are data-class changes (17-change-guide.md). Hosted vs self-hosted is a deploy-profile choice behind ModelSlot, not a kernel change.

---

Shadow and canary#

In-flight runs stick to the lockfile they started under. Open cards keep their proposal lockfile reference; supersede after a pin uses the new lockfile and re-checks kernel gates.

Rollback never invents a third “hotfix” pin outside the lockfile ceremony.

---

Monitoring (plant path)#

On-call watches rates and the production-hardness signals. Full SLO intents: 29-software-quality-and-release.md. Queue: 25-work-queue-and-concurrency.md. Lifecycle: 26-decision-case-lifecycle.md. Ports: 27-ports-and-reliability.md.

On-call watches rates, not narrative quality scores.

Hard-gate block volume is audit and constraint-data ownership, not a soft-gate tuning input.

Correlation: every work item and DecisionCaseOne run unit: intake + snapshot + obligations + candidates + terminal carries correlation_id through ports, traces, and CardSink.

---

On-call ownership#

Pager pages on availability and rate anomalies. Semantic “should this gate be 0.7 or 0.6?” is an improvement-loop ticket with an owner pack, not a 3 a.m. edit.

---

Plant path vs offline path#

Plant request path
  L2 / L3 / site pack → PSM snapshot → stage graph (pinned lockfile)
  → dual-family plant models → kernel checkpoints → L5 or withhold/abstain
  → DecisionTrace + opportunity ledger row

Offline path (never on the request hot path)
  Traces + ledger + closures → Opus 5.5 + Sol council
  → proposals (prompt, registry, soft-gate, stage graph, Jev)
  → replay on holdouts → owner pack → shadow → canary → pin

Exported traces for offline work strip or tokenize per DPDP and plant contract. Offline labels do not write plant source-of-truth. Case libraryEpisodic store of traces joined with L5 outcomes; authority when it disagrees with Hindsight remains outcome authority when memory disagrees (ADR-021).

---

Degraded modes (summary)#

Normative port map and breakers: 27-ports-and-reliability.md. Control plane: 28-commissioning-and-controls.md.

Infra failures are not soft-gate learning signals (26). Humans on shift still run the plant. L4 being quiet is preferable to L4 guessing.

---

L4 store (ops note)#

The L4 operational store holds derived snapshots, traces, case library rows, registries/lockfile copies, the opportunity ledgerStore of every blocked candidate with gate id and later outcome if known, work-queue state, and control-plane audit. It does not hold plant source-of-truth (ADR-021).

Durability: backup + declared RPO/RTO; restore drill before first emit_enabled (28, 29). Retention and DPDP follow the plant contract; keep traces/ledger long enough for calibration and replay.

Offline export: allow-listed fields only; strip secrets/PII per 28.

---

Safe-start (ops)#

Do not set emit_enabled=true until the commissioning checklist in 28-commissioning-and-controls.md is green and the plant owner accepts. New plants start shadow_only=true.

---

v1 slice vs later#

---

Change class#

Operations procedures and dashboards are data / plug-in around stable ports. Changing hard gates, terminals, money ownership, or the write ban is kernel and needs an ADR (17-change-guide.md). Production-hardness surfaces: ADR-027.