Counting cars for prediction markets
A prediction market needs something to resolve on. We wanted a signal that was local, continuous, public and hard to argue with. Cars crossing a line on a road.
Every five minutes a market opens: will the count be above N? People bet up or down, watch the live stream, and the market settles on chain when the cycle closes. At peak, around 5,000 people are watching the same count at the same time. If the number is wrong, real money resolves wrong.
I built the pipeline that produces that number. Most of the code, the audits, the fixes and the runbooks were written by agents. This post is about how the detection works and how the system around the agents kept it alive.
The pipeline
public camera (HLS) ──▶ ffmpeg ──▶ freshest-frame queue
│
▼
YOLO (car, bus, truck) + ByteTrack
│
▼
counter: line crossing per track
│
┌────────────────────────┼─────────────────┐
▼ ▼ ▼
overlay on frame 5-minute cycles WebSocket
(OpenCV, ROIs only) (SQLite, clock- (partial
│ aligned) count)
▼ │ │
ffmpeg ──▶ MediaMTX ──▶ WebRTC ▼ ▼
public stream keeper resolves market maker
~5,000 viewers the market quotes odds
Python, OpenCV, Ultralytics, ByteTrack, FFmpeg, MediaMTX, one 8-core VPS with no GPU. That last part shaped every decision.
How a car becomes a count
Detection alone does not count. A car is in the frame for a hundred frames, and a model that says "car" a hundred times has counted nothing. So the model runs with a tracker, and the count is a property of a track, not a detection.
track 4127 (car)
frame 1..8 seen, not yet eligible ← minimum frames
frame 9 side A confirmed ← signed distance to line
frame 23 side B ← crossing
count += 1 track 4127 marked counted, never again
frame 60 track gone, state expires ← TTL
Each track carries its own state: how many frames it has been seen, which side of the line it was confirmed on, whether it has already been counted. A crossing only counts when a track was confirmed on one side and then confirmed on the other. A track that flickers at the line does not count. A track that is lost behind a bus and comes back with a new id is the failure mode that is left, and it is the one the sample checks are for.
The line is a polygon in the frame, editable from the admin, per camera. Trucks and buses count as vehicles. People and fish have their own profiles with their own models, because the pipeline does not care what crosses the line.
Fast enough on a CPU
The camera sends 1080p at 25 frames per second. YOLO looks at 640 pixels. The stream goes out at 854 by 480. Nothing in between needs the full resolution, so the pipeline does not carry it. Queues keep only the freshest frame and drop the rest. The encoder runs in its own thread with its own pacing.
The overlay is where this was learned the hard way. A redesign moved the drawing to PIL with full-frame layers. Production went from smooth to 5.6 frames per second, 188% CPU on the Python process, and an 11-second delay, while the GPU sat at 39%. The bottleneck was composition, not detection. The fix was OpenCV primitives drawn only inside regions of interest. The effects came back one at a time, each measured at three resolutions before going out.
That incident became a document, and the document became a rule: measure before and after at 720p, 1024p and 1080p, or it does not ship.
Public by design
The count is the market. If people cannot see it, they will not trust it, and if they do not trust it they will not bet.
So the annotated stream is public over WebRTC, sub-second, straight from the media gateway. Snapshots are public. The partial count is pushed over WebSocket to anyone listening, which includes the market maker quoting odds while the cycle is still running. Only the admin and every write endpoint sit behind auth.
Staying alive
A market that resolves on a wrong number is worse than a market that pauses. So most of the work after the first month was not detection. It was making the number survive the real world.
camera dark ──▶ supervisor restart with backoff
──▶ 120s offline ──▶ failover to the next camera
from a catalog of 155,
health-checked, with a
pre-configured line each
──▶ Telegram alert, deduplicated
cycle broken ──▶ marked partial, never settled as complete
process dies ──▶ cycle state recovered from SQLite on boot
An offline blip in the middle of a cycle used to erase the counts before it. A crashed snapshot loop used to die silently on its first exception. The supervisor's backoff used to reset itself and restart every ten seconds forever. Each of these is now a test.
Built with agents
The repo has 24 test files and about as many documents. That ratio is the method.
audit ──▶ plan ──▶ fix ──▶ measure ──▶ record ──▶ next audit
│ │
│ 2 critical, 7 high, 14 medium, │ docs/ + memory:
│ 19 low, each with file:line, │ what broke, why,
│ severity and attack order │ the rule that follows
└───────────────────────────────────────┘
An agent audited the whole codebase and produced a ranked list: two critical, seven high, fourteen medium, nineteen low, each with the file and line, a reproduction and a suggested order of attack. The critical ones were a deploy that overwrote production camera config and background loops that died silently. Both were fixed the same week, both have tests, and the audit document stayed as the map for the next pass.
Every investigation ends in a file. The overlay incident. The performance analysis that found resolution leaking through the whole pipeline. The map of how three downstream services consume the API, which defines the contract nothing can break. The plan for the camera catalog, in five phases, each shipped and checked off.
And a memory that survives the chat. When restreaming YouTube cameras from a datacenter IP kept failing, the agent found the combination that works, then found the wall that does not: YouTube invalidates the session after an hour on that IP no matter what, while a residential IP holds indefinitely. That is recorded as a fact with a date, so no future session spends a day rediscovering it.
Why this is not about cars
traffic camera any live signal
───────────────────── ──────────────────────────────
track, not detection count the entity, not the event
side A then side B two confirmations before a fact
overlay measured at 3 res performance is a test, not a feel
public stream the number people can verify
partial cycle a bad reading pauses, never lies
failover catalog the next source, already configured
audit with attack order agents that rank their own work
memory with dates what was true, and when
The hard part was never the model. Detection was solved in a week. Producing a number that thousands of people can bet on, every five minutes, from a public camera on a box with no GPU, for months, is a systems problem. Agents did the systems work, because the loop gave them a way to audit, fix, measure and remember.
What I did
I picked the signal, drew the first line, set the rule that the number is public and the rule that a broken cycle never settles. I built the first pipeline, then handed the audits, the fixes, the catalog and the failover to agents, and read every document they left behind. When one of those documents said "do not ship without measuring at three resolutions", that was the system improving itself.