Turning a Contractor's Walkthrough Video Into a Parts Quote

A contractor walks a job site with his phone out, talking the whole time. "Sixteen by twenty, two-level, composite decking." By the time he's back in the truck, he should have a draft parts quote priced from his own inventory catalog.

That's BidWalk. Walkthrough video in, draft parts quote out. Everything downstream of that — the app, the PDF, the polish — is worthless if that one sentence isn't true, so I built the system in the order the risk actually sits rather than the order that demos well.

The load-bearing word is "draft"

The output is never a final quote. He reads it, corrects it, approves it. Tap a line item and the video seeks to the moment he said the thing that produced it.

That sounds like a UI nicety. It isn't. It's the only reason a contractor would trust a number a machine wrote. Every line carries provenance back to a timestamp, and provenance is one of two things in this project that genuinely cannot be added later — you either capture it while the pipeline is running or you never have it.

The AI is not allowed near any number that matters

This is the part I'd defend hardest.

The model never sees a price. It never does arithmetic. It never approves anything. It never emits a SKU that hasn't been validated against the catalog.

Deck work is what makes this tractable. Nobody narrates a deck part by part — they narrate dimensions. "Sixteen by twenty, two-level" is not a materials list, it's a geometry problem, and a geometry problem is something you solve in Python with the AWC DCA 6 span tables in front of you. Joists, beams, posts, hangers, ledger fastener spacing. Deterministic, testable, and wrong in ways a test can catch.

So the division is clean: the model reads the site and the narration, and plain code turns dimensions into a takeoff, the takeoff into catalog matches, and the matches into money. There is no language model anywhere in the pricing path.

The quantity rule follows from the same instinct. Quantities come from narration, from stated dimensions, or from clearly countable objects. Vision never eyeballs "forty feet of railing." If the quantity isn't genuinely there, the line becomes a suggestion with no quantity and a blank for him to fill.

A blank the contractor fills in is honest. A confident wrong number is how you lose him permanently.

Then the unglamorous half that decides whether it ships

A 720p walkthrough is a few hundred megabytes leaving a job site over whatever cellular exists there. That reality dictates most of the architecture.

The video never passes through the API. The server issues a GCS resumable upload session and the phone talks straight to storage. Processing is triggered by the object landing, not by the client announcing it finished — a client call can be lost, and a walkthrough that silently never processes is the worst possible failure for something a contractor is standing on a job site waiting for.

Delivery is at-least-once and unordered, so the worker can't assume it runs once. Each of the six stages is its own row, claimed before it runs. A redelivery resumes at the first incomplete stage instead of re-paying for finished ones — and those stages cost real money, because two of them are a speech-to-text call and a sixty-thousand-token multimodal request.

The retry semantics are sharper than they look. Cloud Tasks treats any non-2xx as "try again", so the HTTP status the worker returns is the retry decision. Return 500 on a permanent failure and you burn five attempts on an input that will never improve. Return 200 on a transient one and you silently drop a walkthrough. Getting that classification wrong costs money in both directions, so it's an explicit ordered table with an invariant a test checks — not a stack of except clauses nobody re-reads.

One constraint I enjoyed: there is no disk on Cloud Run. The working directory is RAM, so ffmpeg's scratch space competes with the process itself. That's not something to reason about — it's something to measure. A full-length walkthrough peaks at 1,032 MB against a 2.40 GiB budget, about a quarter of the container limit.

What's deliberately still failing

The test suite currently sits at 2,882 passing, and five tests that are supposed to fail.

Four of those five are data gates. They stay red until the contractor's own material lists, his real catalog, adversarial job-site audio, and at least ten real walkthroughs land. Each one names the missing data in its failure message.

That's a design decision, not a backlog. I have a golden-set harness and a scoring pipeline, and a synthetic walkthrough run through them proves the harness works. It proves nothing whatsoever about whether extraction works on a job site — his narration, his slang, his part names, a camera walking past real rot. Marking those green with synthetic data would convert "we don't know yet" into "we're fine," which is the single most expensive lie you can tell yourself on a project like this.

So the mechanism ships complete and the gate stays open, in public, in the test output.

Where it stands

The deterministic core came first — catalog, takeoff, matching, pricing — because it's provable against numbers the contractor already trusts. Then the extraction pipeline as a local command, so quality became a measured number instead of a vibe. Right now I'm on the durable cloud pipeline. After that: the review screen the whole product is judged by, resumable capture on the phone, and finally approve-freeze-PDF and a real field trial.

The interesting work has consistently not been the AI part. It's been deciding exactly what the AI is not allowed to touch, and then making that boundary structural instead of aspirational.