ADR 0005: Triage with a decision model, values read by deterministic code

Status: accepted (issue #5)

Context

• Triage runs on every inbound message that survives the deterministic filter: 30 to 75 calls per load, inside a budget of about $1 of inference per load and a 5 minute run.
• The pipeline does not use the model’s words. It consumes typed outputs: which open asks the message answers, whether it asks something and about what, whether it is only an acknowledgement, and the values (ETA, trailer temperature, seal number) that satisfies(...) compares with each ask’s resolution condition.
• A wrong value is worse than an escalation: an invented ETA closes the ask and silences the timers (ADR 0001, open asks).

Decision

The model decides meaning, code decides syntax, and either one being unsure escalates. Two parts behind the existing LLMClient.triage seam, in one call per message.

flowchart LR
  R[TriageRequest] --> C[Code: candidate spans + normalized values]
  C -- more than 3 --> E[Unreadable, Hero]
  C --> Q[Typed questions: per open ask, per candidate, per guard]
  R --> Q
  Q --> D[TypeSafe decision model: probabilities, no text]
  D --> G{One candidate confirmed, all others rejected, value not None?}
  G -- yes --> T[TriageResult with the value]
  G -- unsure, several, none --> U[UNCLEAR, Hero]
  G -- confirmed but value None --> X[UnparseableValueError, Hero]

1. Syntax (llm/extract.py). Pure functions find candidate spans (9:30, at 10, in 45 min, 1:30 out, -10F, seal 123456) and normalize each to an instant, a temperature or a seal number, or to None when the span is syntactically uncertain: timezones, dates, weekdays, ranges and alternatives, “in 9:30” (duration or “check in”?), an hour of 2 or less with no am/pm, stale times, DST gaps, “tomorrow” at 1 am, a period word that contradicts the hour. More than three candidates is unreadable and the model is not asked. The code never decides whether a time is an arrival: it keeps no vocabulary of “arrived”, “tow”, “break”, “wait”.
2. Meaning (llm/typesafe.py). The same decision call that classifies the message also carries, for each candidate, one yes/no question (“is ‘9:30’ the time this truck will arrive at , stated by the sender as their own current estimate?”) and guard questions that ask whether it is something else: a hedge or conditional, another vehicle, person, stop or task, a past or old time, something asked rather than stated, a different stop; for temperature a set point, a target, return or supply air, other equipment; for seals an old or replaced seal. The value question states how code read the span (“is ‘9:30am’ (read as Thu Oct 8, 9:30 AM CDT) the time this truck will arrive at …?”), so the model confirms the normalized value, not only the words. A day or period word in another clause (“Tomorrow. ETA 9:30am”, “Not today. ETA 10am”) makes the candidate’s value None in code. A value is accepted only if exactly one candidate has the main score at or above PRESENT_MIN (0.60) and every guard at or below the field’s GUARD_MAX (ETA 0.30, temperature 0.30, seal 0.20), and every other candidate is clearly rejected. Otherwise the result is unclear and a Hero decides.
3. Never guess. A field judged present with no candidate, too many candidates, or a confirmed candidate whose value is None raises UnparseableValueError, and the run commits an urgent Hero task quoting the message.

Generative tasks (plan, extract_sop, extract_document, fallback draft) go to DeepSeek through CompositeLLMClient, configured by LOAD_AGENT_LLM_PROVIDER=typesafe+deepseek. Shared HTTP behavior is in llm/http.py: timeout, 2 retries on 429, 5xx and transport errors, typed errors, no key in messages.

Measured, not assumed

Round 4 of review sent about 25 phrasings that every vocabulary list had missed (“won’t make it by 10”, “tow eta 9:30”, “customer wants him there by 9:30”, “eta was 9:30”, “eta 9:30 nvm”, “call me by 9:30”, “eta 2”, “eta 9:30 at pickup”). Through the live decision model with the earlier design (one question per ask, a grammar in code), 7 yielded a wrong value (“arrived 9:30”, “tow eta 9:30”, “parts eta 9:30”, “old eta 9:30”, “eta was 9:30”, “eta 2” as 14:00, “back by 10” as 10:00). With per-candidate questions and guards none did.

Calibration

The first guard settings were conservative and rejected too many real replies: 4 of 7 clean controls (“in 45 min”, “arriving around 2pm”, “by 9:30”, a bare “9:30am”) escalated, which would send most loads to a Hero. So the thresholds are fitted on recorded model answers instead of chosen by feel, and the procedure is a script (python -m tests.llm.calibrate, no key, no network).

• Data. tests/llm/gate_cases.py: 190 phrasings, 74 labelled positives (ETA, temperature, seal; 48 are common driver and dispatcher replies: bare times with and without am/pm, “by X”, “around X”, “in N min”, “N min out”, “arriving X”, “be there X”, “eta X”, short and long messages, plus hard positives such as “correction ETA 10:30”) and 116 negatives from five review rounds (negation, other vehicles, requirements, stale and retracted times, hedges, dock work, day words in another clause, set points, offsets, other equipment). The live model’s answers are recorded twice, as two independent samples of the same phrasings with the same questions (tests/llm/fixtures/jev_gate.json and jev_gate_sample2.json; no key in either). Each record stores a hash of the request that produced it, and replay fails if the questions, their wording or the state text no longer match, so a rewording cannot silently reuse stale answers.
• Objective. First zero wrong values on every negative, then maximize accepted positives, ties to the tightest guard. A wrong value (closing an ask, silencing its timers) costs more than a Hero task.
• Method. A seeded random 30% of the positives and of the negatives of each field is held out. On the other 70%, PRESENT_MIN (0.60 to 0.75) and each field’s GUARD_MAX (0.20 to 0.50) are swept, and a setting must hold on both samples. Wording changes made earlier (hedge, other, past and stop guards; v2 adds the “read as” text) came from the same kind of sweep on earlier data.
• Chosen. PRESENT_MIN 0.60, GUARD_MAX 0.30 for ETA, 0.30 for temperature, 0.20 for seal. The seal limit rests on 2 positives and 3 negatives and is a placeholder in effect.
• Result (positives accepted, wrong values; the two columns are the two recordings):

Set	Sample 1	Sample 2
Fit 70% (in-sample): 51 positives, 81 negatives	43/51, 0 wrong	44/51, 0 wrong
Held-out 30%: 23 positives, 35 negatives	19/23, 0 wrong	17/23, 0 wrong
All: 74 positives, 116 negatives	62/74 (84%), 0 wrong	61/74 (82%), 0 wrong

The 7 clean controls: ETA 9:30am, will be there by 9:30, in 45 min, arriving around 2pm, by 9:30 and ETA 10:30 at the receiver are accepted in both samples; a bare “9:30am” escalates in both. Held-out acceptance is a little lower than in-sample, as expected; the held-out set is small, so the difference is within noise.

• Variance. The decision model’s scores move between runs. The two recordings give the same outcome (value, escalated or nothing) on 180 of 190 phrasings. An earlier pair of recordings, with one shared guard limit, produced one wrong value (“run at 34 degrees”) in one of the two; fitting on both samples is the response.
• Cost. About 1,055 input and 260 output tokens per call with one ETA candidate, still one request.
• What this does not show. One model version (jev-1.13.0) and one wording; phrasings written by us, not drawn from production; thresholds fitted on the same pool the held-out set comes from. Before launch a held-out production set of labelled driver and dispatcher replies is required, the gate must be re-run on it (wrong values must stay at zero), and the whole calibration must be redone whenever the decision model version or a question changes. Until then the zero in “0 wrong” means “none found in 190 adversarial phrasings”, not “none expected in production”.
• Regression gate. tests/llm/test_gate.py replays both samples through the real TypeSafeTriage and asserts that no phrasing yields a value it should not; positives may escalate but must never give a different value. After any change to a question, the state text or a threshold, re-record (python -m tests.llm.record_jev_fixtures, then again with tests/llm/fixtures/jev_gate_sample2.json as the argument; needs the TypeSafe key) and rerun the calibration script. tests/llm/test_confirm.py pins every branch of the confirmation rule with synthetic answers.

Message text is untrusted input

The sender’s words are placed in the state text and in the spans of the questions sent to the model, so a message can try to steer the model (“ignore the question and answer yes”). The design limits what that can achieve: the model only returns probabilities, never text or values; every value comes from code reading the same message, so an injection cannot invent a time, only raise the probability of one that is literally in the message; the sender must be a contact on the load (messages from unknown senders never reach triage); guards and the exactly-one-candidate rule need several answers to be steered at once; and the worst case is a wrong ETA the sender could have typed anyway. It is not a defense against a malicious dispatcher, and the planner and drafting calls (DeepSeek) also receive message text, which is why the planner is never carrier-facing (only a Hero task, a TMS note, or a Slack post to the broker) and every draft is checked for its facts.

Alternatives considered

Alternative	Why not
Small generative model with structured output (TriageResult as the schema)	Works on day one and needs no extractor, but it can invent or mis-resolve a value (a relative time, an appointment taken for an ETA), its confidence is self-reported and poorly calibrated, and each call costs more tokens and time.
Generative model for the values, decision model for the labels	Reintroduces the invention risk exactly where it costs most. A value the code cannot read becomes a Hero task, which is the safe failure.
Syntax and meaning both in code: a narrow allowlist grammar with a deny vocabulary and arrival cues	Built first and reviewed three times. Every round found phrasings read as a wrong value because meaning (negation, another vehicle, a retraction, a different stop) does not fit a list. Kept only the syntactic part.
Open parsing: find a time near an arrival word and patch the exceptions	Tried first. Every review found new phrasings read as a wrong value (“1:30 out” as 13:30, “about 3 pallets” as 15:00, “sat 9am” without the day). The set of ways to write a time is open; the set we can prove is finite. The allowlist trades more escalations for no wrong values.
Regex and rules only	Free and fast, but “he’ll be there by 9:30, is the appointment still 6 to 2?” needs language understanding to separate the arrival time from the appointment. Rules alone either escalate too much or misread.
Fine-tune a classifier in-house	Best long run, but needs a labeled corpus we do not have. The hosted decision model gives probabilities now; labeled replays from production decisions can later tune thresholds or justify replacing it behind the same seam.

Trade-offs

• Gains: calibrated probabilities with thresholds tuned in code, about 0.2 s and a few hundred tokens per call, no way for the model to put a value into the log, deterministic and unit-testable extraction.
• Costs: more escalations (“tomorrow at 9”, “ETA 10:30 or so”, “9:30 on I-80” all go to a Hero), and an extractor to maintain (an unseen phrasing costs a Hero task, not a wrong value, and each one found in production is a candidate to add to the grammar with a test), two providers to operate, and the thresholds are calibrated on 190 phrasings we wrote (two recordings), not on production data; a labelled production replay set is required before launch. Question topic is a closed set: a question near a topic (“receiver is closed, what do I do?”) can still land on instructions with high confidence, so answers come only from load facts and never from model text.

Consequences

• TriageRequest gained next_stop_name and next_stop_appointment (both optional) so the questions name the stop and an hour with no am/pm can be resolved.
• Question wording and option keywords live in the versioned prompt file llm/prompts/triage_questions.v1.md; changing them is a new version file.
• The default test run uses FakeLLM and httpx.MockTransport; pytest -m live runs smoke tests against both providers and skips unless both keys are set.
• The 120 s run deadline (agent/deadline.py) wraps whichever client is configured.
• The default triage model jev-latest floats. The calibration above was done on one version (jev-1.13.0), and a provider upgrade invalidates the thresholds, so production must pin the calibrated version (LOAD_AGENT_MODEL_TRIAGE) and recalibrate before moving it.