L1–L2 data plane

How plant systems become closed records in one TimescaleDB, and how later layers read them only over HTTP.

Paths above resolve from the workspace docs/plans/ tree; in the contract repo alone, see the consumer READMEs that link here.

What you can do#

• Configure meters, CNC/SCADA, EMS exports, DISCOM bills, and ERP/WMS/CMMS/QMS/shift sheets into a fixed set of records.
• Keep collecting while the WAN is down (SQLite buffer; prune removes flushed rows older than retention_hours, default 72; MQTT QoS 1). Broker persistence plus ingest clean_session=False / MQTT_CLIENT_ID hold the queue while cloud ingest is down.
• Refuse bad payloads before they become facts (schema DLQ; bills need the rupee gate — missing printed total fails; cloud refuses extraction.validated != true; unmapped tags/columns do not become zeros).
• Store once in L2 with RLS. L3–L6 call query HTTP only — never L2_DATABASE_URL.
• Prove a typical forging-shop site with files when the live ERP is unknown.

What this does not do: write to ERP/QMS/CMMS or CNC programs; become those systems; invent asset_id or timestamps. The ingest path never writes to a PLC. The only PLC write path is the separate Plant BoxPlant-side computer for the fast loop (direction; D4) stamped-writer for allow-listed, plant-approved process setpoints (direction, ADR-035, D10).

Pieces and how they connect#

flowchart TB
    %% house-style: l1-l2-data-plane
    subgraph otNet["Plant OT network"]
        direction LR
        meters["Meters and EMS"]
        cnc["CNC SCADA Kepware"]
        edge["edge-agent Go"]
        meters --> edge
        cnc --> edge
    end
    subgraph itNet["Plant IT network"]
        direction LR
        files["CSV and XLSX drops"]
        erp["ERP WMS CMMS QMS"]
        ctx["context-agent Go"]
        files --> ctx
        erp --> ctx
    end
    subgraph people["People"]
        pwa["connectors-doc PWA"]
    end
    mqtt["MQTT stamped/v1"]
    saas["SaaS ERP optional"]
    subgraph cloudPath["Cloud ingest"]
        direction LR
        cloud["connectors-cloud"]
        outbox["Cloud outbox"]
        gate{{"schema and quality gate"}}
        l2ing["L2 ingest"]
    end
    ts["Timescale seven schemas"]
    query["L2 query API"]
    later["L3 L4 L5 L6"]
    dlq(["↩ DLQ and refuse before facts"])
    pwa --> mqtt
    edge --> mqtt
    ctx --> mqtt
    mqtt --> cloud
    saas -.-> cloud
    cloud --> outbox --> l2ing
    l2ing --> gate --> ts
    gate -.-> dlq
    query --> ts
    later --> query

    classDef govc fill:#fff4d6,stroke:#c99a2e,color:#000
    classDef agentc fill:#e8f0ff,stroke:#5b7bd5,color:#000
    classDef loopc fill:#eef7ee,stroke:#4f9a4f,color:#000
    class gate govc
    class pwa agentc
    class dlq loopc

Protocols by program#

edge-agent (OT): Modbus TCP/RTU, plant MQTT / Sparkplug B, OPC UA, MTConnect (simulator-proven field path), BACnet/IP read-only (simulator-proven), DLMS read-only LN Low + WRAPPER (simulator-proven; not HLS), EMS file watch, single-number REST poller, historian SQL (sqlite, Postgres, MySQL; session read-only). No FOCAS / native S7 / EtherNet/IP — reach those via OPC UA, Modbus, Kepware, or plant CSV.

context-agent (IT): CSV/XLSX file drops, HTTPS REST/OData, read-only SQL (SELECT + watermark; Postgres / MySQL / sqlite).

connectors-doc: human upload → extract → review → explicit publish → MQTT. No subscribe. No L2 open. Local profile stores files on a volume (STORAGE_BACKEND=local), not MinIO.

connectors-cloud: MQTT subscribe (persistent session, MQTT_CLIENT_ID, clean_session=False) + HTTP backfill (POST /v1/measurements, /v1/production-orders, /v1/context — no /v1/bills). Schema validate → quality (measurement late/range; bill_line must be extraction.validated=true) → dedupe → outbox → L2. SaaS REST poller exists but CONTEXT_POLL_ENABLED defaults off; empty poll list polls nothing.

How to load a source#

1. Pick a pack under profiles/systems/ (or bill profile).
2. Add an instance to site config: pack id + folder / base URL / DSN reference. Secrets are secret_ref names only.
3. Override columns where the plant export differs. Resolve machine names via asset_aliases.
4. Start edge-agent or context-agent. Restart for new ERP instances.

What happens to one row#

Pack maps fields → timezone (default Asia/Kolkata) → UTC; lakh grouping; asset alias. Unresolved → unmapped_tag event. Context change-detection uses last-hash in SQLite; asset_state also heartbeats every 60s. Buffer → MQTT → cloud validate → outbox → L2 inbox (201 new / 200 duplicate) → route_record.

Context dedupe is sha256(org|plant|record_type|business_id|observed_at) (see TOPICS.md). Latest reads order by observed_at. topology_record keys on kind|topology_id|site_pack_version instead: a correction or retirement comes with a new site packVersioned, owner-reviewed plant configuration including topology version.

Fast loop topics and fast-batch upload (direction)#

Not as-built. On a hybrid site the Plant Box runs a plant-only broker beside the normal uplink. All topics sit under stamped/v1/{org}/{plant}/; publisher and reader per topic are fixed in fast-loop/07 section 2, and broker ACLs follow them.

Fast-batch upload. The Plant Box keeps a rolling fast archive and uploads compressed batches on the existing durable path into telemetry.fast_reading (kept at least 13 months). Per-minute measurements continue unchanged. Config comes the other way only by outbound pull: GET /v1/plants/{plant_id}/plant-box-config?since=.

Record types added in contracts 0.16.0#

All travel on …/context in the existing wrapper and are read-only plant truth.

Additive 1.1.0 bumps: shift_roster assignments gain escalation_role, channel_preference, contact_available (no phone field); operating_rate gains approved_by, approved_at. 1.0.0 payloads stay valid.

How L2 stores it#

One TimescaleDB, seven schemas:

What each record is for#

• Energy: measurement, bill_line, tariff, asset_state
• Cost: + operating_rate (calculator still owns rupees)
• Time: asset_state, production_record, production_order, stop_event, changeover_matrix, tool_life
• Continuity: process_batch, flow_position, topology_record
• Exception: event, maintenance_context, quality_status, material_availability, plant_constraint

Two clocks#

Every context-style read has two times. as_of filters on observed_at (when the plant says it was true). known_as_of filters on known_at (when L2 stored it; created_at, or coalesce(created_at, ts) for asset_state). Each row returns known_at. Replaying with known_as_of shows what Stamped knew at that moment, so a late backfill never rewrites a past decision.

Builder reads vs agent tools#

Two catalogs, both read-only, sharing no path:

• Agent tools — contracts/tools/l2-query-tools.json (1.1.0), served at GET /v1/agent-tools. Allowlisted zoom reads during a decision. No SQL, no org_id argument, no graph/traverse. Org comes from X-Org-Id on the service credential. 0.16.0 adds get_stop_events, get_changeover_matrix, get_tool_life, get_data_health.
• Builder reads — contracts/tools/l2-builder-reads.json, served at GET /v1/builder-reads. builder_topology and builder_changes (incremental feed with an opaque since cursor) under /v1/builder/. For the L4 plant-state-model builder only; never on an LLM tool allowlist.

Data health#

GET /v1/plants/{plant_id}/data-health reports each source (record type + connector or tag) as fresh, stale or silent from ingest watermarks, plus open data_gap / unmapped_tag / connector_death events from the last 24 hours. A late backfill never moves a watermark backwards. Agents abstain when the source they need is not fresh.

Contact route (L5 only)#

Phones live only in graph.person_contact, loaded through admin-api and never streamed; the shift roster carries channel_preference and contact_available, never a number. GET /v1/plants/{plant_id}/people/{person_id}/contact needs a second key (X-Contact-Key, env L2_CONTACT_READ_KEY; unset means always 401) on top of the service key. Every read that returns 200 or 404 writes an append-only audit row. It is not in either catalog; only L5 calls it.

Boot (proven)#

L1-only (this run). Folder is Connector-L1 (no spaces). From the workspace:

cd "D:\Startups\Stamped_Energy\L1-L6"
bash scripts/l1-only-e2e.sh

R1 (2026-09-26): exit 0 in 104 s. Mock-L2 matched outbox: measurement 86, event 50, asset_state 4, production_order 3, bill_line 1 (after explicit publish), and one each of process_batch, production_record, flow_position, material_availability, operating_rate, quality_status, maintenance_context, shift_roster. Field: MTConnect, BACnet, DLMS all present. DLQ 0.

Older L2-coupled boot (still valid as a prior proof; path uses Connector-L1):

cd "D:\Startups\Stamped_Energy\L1-L6\Connector-L1\connectors-edge"; uv run --python 3.12 --with requests --with psycopg[binary] python scripts/e2e_context_l2.py --workspace "D:\Startups\Stamped_Energy\L1-L6" --report "D:\Startups\Stamped_Energy\L1-L6\docs\plans\l1-l2-context-records\planning\R1_BOOT.md"

That run’s fixture counts: measurement 9, asset_state 1, production_order 1, production_record 1, process_batch 1, shift_roster 1, bill_line 1.

Trials#

L1-only T1: bash scripts/l1-trials.sh — 24/24 PASS (2026-09-27). Includes WAN store-and-forward, prune (clock-aged), SQL write refuse, field sims, DISCOM 11, bill gates, photo/CSV, DLQ, L2_DATABASE_URL grep. Weakenings: audit section Honest weakenings.

Older T1: 22/22 PASS against the live-L2 stack (happy path, fail-closed unmapped paths, auth/tenant, tool catalog, A→B→A / late older, replay, WAN and L2-down drain, heartbeat, IST/lakh, SQL write reject, poller off/on, bill gates, lost last-hash).

Honest holes#

• Live ERP credentials not exercised; file packs are the floor.
• DLMS / BACnet / MTConnect are simulator-proven, not plant-proven. DLMS lab is LN Low + WRAPPER, not HLS.
• No POST /v1/bills; MQTTv5 session expiry is not set (MQTT 3.1.1 persistent session).
• T1 prune and “no printed total” trials use a backdated clock and a post-extract SQL edit (see the L1 audit).
• Flat CSV asset_state / shift_roster without full schema objects still need the e2e *.publish.json wrappers (noted in the older R1).
• Two changes inside one poll interval can collapse to one context publish; a wiped last-hash may resend one extra history row per business id.
• No material-genealogy relationship record. process_batch and flow_position describe continuity, but nothing stores an assertion such as billet → part → basket → test → reject with its provenance (nearest_time, register, scan, dpm_code), a confidence score and a correction history. Nearest-time joins (for example the Bhatia 98.5% billet-to-part join) must stay labelled as event joins, not genealogy. Direction: an immutable, correctable material_link record keyed by both identities, with method, confidence, observed_at/known_at, and the confirming role; physical identity (basket scan, Data Matrix) upgrades the method. See ISA-95 material lot and sublot genealogy (ISA).