Operating rules
For the first three plants the targets include a fast-loop step under 1 second at p99, a card within 2 minutes of its prescription and zero acknowledged readings lost. They are design targets, not results, and each names the check that proves it. Security uses row-level tenancy, per-device certificates on the Plant Box and no inbound ports. Every degraded mode has an owner, a detector and a fallback, and writes stop first in all of them. Observability traces a card by episode_id from reading to ValueRecord, and deployment profiles run from one pilot box to managed HA.
11. Non-functional requirements#
Targets for the first three plants. They are design targets, not measured results; each is checked by the test or signal named.
| Requirement | Target | Checked by |
|---|---|---|
| Fast-loop step time | p99 under 1 s; heartbeat under 2 s | 72-hour soak on the Plant Box (D3) |
| Writer read-back | Under 1 s after the write | Writer test against plant-sim, then shadow logs |
| Slow-loop freshness | Reading in L2 within 5 minutes, p95 | Source watermark (migration 017) |
| Card delivery | Within 2 minutes of the prescription, p95 | L5 card timeline |
| Acknowledged readings lost | Zero | Sequence numbers and replay (section 7.3) |
| Plant Box buffer on WAN loss | 30 days of readings | Local store sizing (section 15) |
| Cloud availability | 99.5% a month on one box; 99.9% after the D20 move to managed HA | Uptime check (section 14) |
| Recovery point and time | RPO 15 minutes (WAL archive), RTO 4 hours | Quarterly restore drill |
| Retention | Raw readings 90 days hot, 1-minute aggregates 3 years; write log, Action and ValueRecord 7 years | Timescale retention policies |
| Tenant isolation | No cross-tenant read | RLS tests in L2 CI |
| Running cost | Inside the ADR-002 ceiling (₹15–25k a month) up to 5 plants; the D20 move to managed Timescale with HA breaks it by design, so the ceiling is revisited when D20 fires | stamped_running_cost.py (section 16) |
12. Security and tenancy#
| Area | Build now | Later (trigger) |
|---|---|---|
| Tenancy | One shared database with row-level security by tenant_id (migration 006); RLS tests in L2 CI | Dedicated deployment per customer (a customer requires it, priced per deal) |
| People | Staff sign in with SSO and MFA; customer users by email one-time code; roles: viewer, owner, plant approver | Customer SSO (first customer that asks) |
| Services | Short-lived service tokens; least-privilege database roles per service | — |
| Plant Box | Per-device certificate (mutual TLS) issued at commissioning and revocable; OT and IT network ports with no routing between them; read-only root; signed A/B updates; no inbound ports | TPM-backed keys where the hardware has a TPM |
| Writes | OPC UA SignAndEncrypt with a per-plant certificate (D10); writer accepts only WriteRequest (section 5.8) inside the operating envelope; safety systems and interlocks are never on the allow-list | — |
| Secrets | Cloud secret store; none in git | Rotation automation (Growth tier) |
| Agents | Read-only, tenant-scoped tools; tool output never grants an action; step cap per question; provider allow-list per customer (D21) | — |
| Data | Encryption at rest (EBS, S3, managed database defaults) and TLS in transit; personal data limited to the names and phone numbers cards need; retention per section 11 | — |
| Audit | Hash-chained write_log; admin and policy changes logged with who and when | — |
The master document hard stops must be enforced in code, not only in policy: no AutonomyPolicySigned grants, envelopes, expiry (direction; L5 owns engine) can name a safety or critical-equipment tag, a quality hold, a maintenance authorisation, or a dispatch sequence; those action classes exist only as AL1Autonomy levels (direction; fast-loop stages 1–3 = AL1–AL3) advice that a named person acts on. Build gate: the AutonomyPolicy and WriteRequestPlant Box write path request (direction; closure/action-intent.json retired) 0.x schemas, with these action classes deny-listed and a CI test that fails if a policy names one, ship before any write leaves shadow.
13. Degraded modes#
Every mode has an owner, a detector and a safe fallback. Writes are the first thing to stop in every mode.
| Mode | Detected by | Owner | Fallback |
|---|---|---|---|
| WAN loss at the plant | Plant Box sync heartbeat | EDGE / Plant Box | Buffer locally; local display carries cards (D7); standing policies continue only if their grant has not expired; nothing new is granted |
| Clock skew | NTP offset over 2 s | Plant Box | Readings flagged; writes blocked until the clock is back in band |
| Ingest backlog | Sequence gap or lag over threshold in CLOUD ingest | CLOUD ingest | Back-pressure to the edge; replay in order by sequence number; analytics marked stale |
| Model stale or uncalibrated | Coverage or NIS outside D14 band; model age over limit | L3 registry | Router abstains; AL2 and above revert to AL1 messages |
| Coverage drop in data | Data-quality gate (C03) | L2 and Plant Box | Evidence label falls to Unknown; no action gated on it |
| Operator stop or touch | Writer watchdog, operator input | Plant Box writer | Stop writes, restore the pre-write value where the policy says so, card to the owner |
| Cloud write switch off | L5 autonomy | L5 | Plant Box drops to AL1; switching back on needs a person at the plant |
14. Observability#
| Signal group | Signals | Alert (pages someone) |
|---|---|---|
| Plant Box | Heartbeat, buffer depth, clock offset, fast-loop step time p99, writer rejections by reason | Heartbeat lost over 5 minutes at a plant with an active grant; rejection spike |
| Pipeline | Ingest lag per plant, sequence gaps, watermark age (migration 017) | Lag over 15 minutes |
| Models | Coverage and NIS per model class (D14), PSI drift, router abstain rate | Coverage outside band for a model behind an active grant |
| Product | Cards sent, budget suppressions, time to close | — (daily digest) |
| Cost | LLM tokens per tenant against budget; cloud spend against the section 16 forecast | Tenant at 100% of budget |
| Backups | Snapshot and WAL archive success, last restore drill | Any failure |
- Services emit structured JSON logs and OpenTelemetry traces keyed by
episode_id, so a card can be traced from reading to ValueRecord. The backend is CloudWatch for the first tiers; the instrumentation does not depend on it. - An external uptime check measures section 11 availability.
- Application logs are kept 3 months; write_log, Action and ValueRecord 7 years (section 11).
15. Deployment profiles and capacity#
| Profile | Plants | Cloud | Plant side | Trigger to the next |
|---|---|---|---|---|
| Pilot | 1 | One EC2 t4g.large in ap-south-1 runs Timescale and the services in Docker Compose; Fargate for nightly jobs; S3 for backups and documents; Vercel for the front end | Plant Box (D4) | Memory above 70%, or a second plant |
| Early | 2–10 | Database and services on separate instances; Timescale still self-managed (D20) | Plant Box per plant | Any D20 trigger |
| Growth | 11+ | Tiger Cloud (Mumbai) with an HA replica; services on two or more instances behind a load balancer | Same | Load |
| Air-gapped | per deal | None; an on-site server runs the slow loop; LLM features off or self-hosted | Plant Box plus on-site server | Later: first customer that forbids cloud; priced per deal, not in section 16 |
Capacity notes (base case, planning numbers, not measurements):
- Plant BoxPlant-side computer for the fast loop (direction; D4). 40 fast tags at 1 Hz is about 3.5 million readings a day; 30 days fits easily in 256 GB. The twin step and safety filter must pass the D3 soak on the same box.
- L2. About 5 GB per plant after 12 months with compression, from the sizing in
scripts/pricing/stamped_running_cost_assumptions.json. - Jobs. Nightly analytics and CP-SAT repairs run as Fargate tasks of 1 vCPU and 2 GB, about 30 minutes per plant per day. The MHE benchmark runs offline (D19).
- Delivery. CI builds images; the Plant Box pulls signed images (A/B); L2 migrations run through the existing migration runner. Infrastructure as code from the Early tier.
Page history: last 5 changes
- docs(research): retire stale research to archive/research-2026-10 with a register
ab84821 - docs(technical): rewrite fast-loop/; all architecture diagrams in house style
7330f47 - docs(technical): archive archify; add SYSTEM_VIEWS.md house diagrams; check_docs --min
1e190b6 - docs(technical): carry product sections; rewrite README and pointers
ee1e818 - docs(technical): split decision board into DECISIONS.md
b4db9d4