Photo-to-Purchase-Order

TL;DR

The crew says "the smoked-lens Memphis ones." The supplier needs a SKU and a count. This build puts a vision LLM between them: photo or slang in, confirmed catalog match out, purchase order drafted, approval requested, vendor emailed — all in one chat thread, with a state machine making sure nothing gets lost or sent twice.

The problem this solves

In any business that buys physical stuff — construction, trades, cleaning, food service — ordering runs on translation. Between crew language and vendor SKUs sits a human re-keying paper requisitions, deciphering texts, and chasing half-described requests. Orders get delayed, wrong items show up, and nobody can say where any order currently sits.

This workflow collapses the whole chain into one chat thread. The human stays in the loop — but only at the decision, never at the data entry.

The full picture

                          THE END-TO-END FLOW
═══════════════════════════════════════════════════════════════════════

 ┌──────────────┐  photo or  ┌──────────────────┐
 │ FIELD WORKER │  slang txt │   CHAT INTAKE    │
 │ "need more of│ ─────────▶ │ (Telegram bot or │
 │  these" 📸   │            │  Twilio MMS)     │
 └──────────────┘            └────────┬─────────┘
        ▲                             ▼
        │              ┌──────────────────────────────┐
        │              │       VISION LLM             │
        │              │ sees photo / reads slang     │
        │              │ searches YOUR catalog        │
        │              │ returns top matches +        │
        │              │ confidence                   │
        │              └──────────────┬───────────────┘
        │  "DEWALT DPG55,             │
        │   smoke lens — how many?"   ▼
        └──────────────── CONFIRM IN THE SAME THREAD
                                      │ confirmed qty
                                      ▼
                       ┌──────────────────────────────┐
                       │      ORDER STATE MACHINE     │
                       │                              │
                       │ draft → pending_approval →   │
                       │ approved → sent → confirmed  │
                       │ → received                   │
                       │                              │
                       │ (DB triggers enforce every   │
                       │  arrow — no skipping states) │
                       └──────────────┬───────────────┘
                                      │
              ┌───────────────────────┼────────────────────────┐
              ▼                       ▼                        ▼
   ┌────────────────────┐ ┌────────────────────┐  ┌────────────────────┐
   │ ONE-TAP APPROVAL   │ │  VENDOR EMAIL (PO) │  │  AUDIT LOG         │
   │ magic link to the  │ │ formatted, line-   │  │ every state change,│
   │ budget holder's    │ │ item snapshot,     │  │ who, when, what —  │
   │ phone (SBW-005)    │ │ sent after approval│  │ written by trigger │
   └────────────────────┘ └────────────────────┘  └────────────────────┘
═══════════════════════════════════════════════════════════════════════

What you need

Phase 1 — The catalog (everything depends on this)

The LLM can only match what exists here. One table, one row per SKU:

 products
 ┌──────────────┬─────────────────────────────────────────────┐
 │ sku          │ DPG55-2D                                    │
 │ name         │ DEWALT Protector Safety Glasses, Smoke Lens │
 │ description  │ anti-fog, ANSI Z87.1, smoke gray lens       │
 │ vendor       │ Norwood Supply                              │
 │ unit         │ pair                                        │
 │ price        │ 4.85                                        │
 │ image_url    │ (photo if you have one)                     │
 │ aliases      │ ["smoked lens", "the gray dewalts",         │
 │              │  "memphis ones"]  ← grows every week        │
 └──────────────┴─────────────────────────────────────────────┘

The aliases column is the secret. Every time the LLM mismatches crew slang, you add the slang as an alias. The system gets smarter weekly — this is the cheapest machine learning you'll ever do.

Phase 2 — The chat intake + vision matching

  WORKER SENDS          LLM DOES                    BOT REPLIES
  ───────────           ────────                    ───────────
  📸 photo         ──▶  identify product       ──▶  "Safety glasses, smoke
                        match against catalog       lens — DEWALT DPG55?
                        w/ confidence                How many?"

  "more of the     ──▶  parse slang            ──▶  "Cut-resistant gloves
   cut gloves,          search name + aliases       size L — Superior S13?
   large"               w/ confidence               How many?"

  something        ──▶  confidence < 0.7       ──▶  "Not sure — is it one
  ambiguous                                         of these 3? [list]"

The matcher prompt:

You match field-worker requests to a product catalog for {{business_name}}. Input: a photo and/or a short text. Below is the catalog (sku, name, description, aliases).

Return ONLY JSON:

• matches: up to 3 of [{sku, name, confidence 0–1, reason (under 8 words)}]
• clarifying_question: one short question if top confidence < 0.7, else null

Rules: match against names, descriptions, AND aliases. Never invent a SKU that is not in the catalog. A photo of a brand label beats a guess from shape. If the text mentions size or color, treat it as a hard filter.

CATALOG: {{catalog_json}} REQUEST: {{text}} {{photo}}

Confirm, don't assume. The bot proposes; the worker confirms the match and gives quantity in the same thread. A wrong guess costs one chat message instead of one wrong delivery.

Phase 3 — The order state machine

Every order is born a draft and moves through explicit states. The database enforces the arrows with triggers — application code cannot skip a state, and neither can a bug.

                        THE ORDER BOARD
 ┌─ DRAFT ──┐ ┌─ PENDING ──┐ ┌─ APPROVED ┐ ┌─ SENT ───┐ ┌─ CONFIRMED ┐ ┌─ RECEIVED ┐
 │ items    │ │ APPROVAL   │ │           │ │          │ │            │ │           │
 │ adding   │▶│ magic link │▶│ waiting   │▶│ emailed  │▶│ vendor     │▶│ crew got  │
 │ up       │ │ with budget│ │ on email  │ │ to vendor│ │ replied    │ │ the goods │
 │          │ │ holder     │ │ send      │ │          │ │            │ │           │
 └──────────┘ └────────────┘ └───────────┘ └──────────┘ └────────────┘ └───────────┘
      │              │
      ▼              ▼
  cancelled      rejected (back to draft with a note)

Two disciplines make this bulletproof:

• Snapshot line items. When an item is added, copy the price, name, and description onto the order row. Never JOIN back to the live catalog for a sent PO — historical orders must be immutable even when prices change.
• Kill switch. An auto_send_enabled flag per vendor, default false. Until you flip it, approved POs queue instead of emailing. You watch the system run for two weeks before it talks to real vendors.

Phase 4 — Approval and the vendor email

  draft complete
       │
       ▼
  budget holder's phone:  "PO-118 — 24 pr cut gloves, $312, Norwood. [Review & Approve]"
       │ one tap (see SBW-005 for the magic-link build)
       ▼
  approved ──▶ formatted PO email ──▶ vendor ──▶ reply-to goes to your office
                (PDF or clean HTML       │
                 with PO number,         └─ subject: "PO-118 from {{business_name}}"
                 line items, ship-to)

Folder structure

 photo-to-po/
 ├── CLAUDE.md            ← the AI-builder brief (ships in this kit)
 ├── bot/
 │   ├── telegram.ts      ← chat intake, photo download, thread state
 │   └── conversation.ts  ← match → confirm → quantity flow
 ├── lib/
 │   ├── match.ts         ← vision LLM call + JSON validation
 │   ├── orders.ts        ← state machine transitions (DB-enforced)
 │   └── po-email.ts      ← formatted vendor email
 ├── db/
 │   ├── schema.sql       ← products, orders, order_items, order_events
 │   └── triggers.sql     ← state transition guards + audit log
 └── prompts/
     └── matcher.md

Compliance & safety rules

• Drafts are free, sends are guarded. The auto_send_enabled kill switch stays off until two clean weeks.
• Snapshot discipline: sent POs never recompute from live catalog data.
• Audit everything: a DB trigger writes every state change (who, when, old → new) to order_events. When a vendor says "we never got PO-118," you have the receipt.
• Vendor email needs SPF/DKIM on your domain or POs land in spam.
• No secrets in the bot: LLM keys, DB keys, email keys all live server-side in env vars.

Numbers to watch

In the source build: intake dropped from a 10–15 minute office task per order to a 30-second chat for the worker and one tap for the manager.

Week-one checklist

• ☐ Photo of a catalog item → correct match proposed with confidence > 0.8
• ☐ Slang request ("the smoked-lens ones") → correct match via aliases
• ☐ Ambiguous request → bot asks ONE clarifying question, not three
• ☐ Confirmed item lands on a draft with snapshotted price
• ☐ Try to mark a draft as "sent" directly in the DB → trigger blocks it
• ☐ Approval tap moves pending → approved, audit row written
• ☐ Kill switch off: approved PO queues, does NOT email the vendor
• ☐ Kill switch on (test vendor): clean PO email arrives, reply-to correct

Troubleshooting

• LLM matches the wrong product: catalog descriptions are too thin. Add aliases from real crew language — that column is your accuracy dial.
• Workers won't adopt it: the bot asks too many questions. Two messages max: match + quantity.
• Vendor conflicts in one request: split drafts per vendor automatically; the worker should never think about vendors.
• Photos too dark/blurry to match: the bot should say so and ask for the label: "snap the brand tag if you can."
• State machine fights you during dev: good — that's it working. Add the missing transition deliberately instead of bypassing the trigger.