Load Agent for Freight Hero

Freight Hero take-home · AI Engineer

An agent that follows each client's SOP, acts when safe, escalates when not, and does nothing when nothing is needed.

Every input and every action goes into a per-load event log. Deterministic code owns state, timers and escalation chains compiled from plain-English SOPs. A model is used only to read language and for the rare situation no SOP covers.

Executive summary Scenario explorer Presentation

30

scenario evals

Each replays a timeline step by step and asserts the exact actions, or an explicit noop.

1,455

tests

1,354 run offline with no API key; 101 Postgres contract tests run in CI.

$0.11

LLM cost per load, budget $1.00

Measured on a synthetic day of 87 inputs with the real providers; the decision model at an assumed per-call price.

21 s

slowest run, limit 5 min

p50 0.02 s, p95 1.56 s across the day. Agent time only, network to the providers included.

Load #481207 in four decisions

Client A, delay risk high. One model call in the whole timeline: classifying the PO question.

1. 7:01 AM Ping turns delay risk high SMS the driver for the ETA, set the 7:31 AM follow-up
2. 7:13 AM Driver asks for the PO Answer it from load data; the ETA ask stays open
3. 7:31 AM Timer fires, no ETA yet Ask again, set the 8:01 AM follow-up
4. 8:01 AM Two requests unanswered Email the dispatcher, create an urgent Hero task

Step through it in the explorer →

Run it locally

Python 3.12+ and uv. No API key, no database, no network.

# unit tests and all scenario evals, offline
uv sync && uv run pytest

# watch the agent handle load #481207
uv run load-agent run \
  evals/scenarios/client_a_load_481207_full_timeline.yaml

Full instructions →

ArchitecturePipeline, memory, timers, SOPs, safety, scale and cost. Assignment answersEvery question, in the assignment's order. DecisionsFive ADRs with alternatives and trade-offs. SOP previewsWhat the agent understood from each layer.