The $0 Specialist and the Load-Bearing Bug
That clip up top: the same Federer rally, tracked two different ways. Left, the old general-purpose tracker's trail of yellow dots — a comet's tail — thins out and dies partway through; it wrote down only six of twelve shots. Right, a tiny tennis-only specialist's trail stays solid to the last shot. Same clip, same notation software underneath both — the only thing that changed is who's watching the ball. The specialist charted all twelve shots, and got every one of the seven shot symbols it committed to right.
Before spending another dollar chasing a tennis ball across the screen, I stopped and asked a simpler question: has somebody already solved this?
Here's where things stood. The pipeline had just been graded for the first time against a real match a human had already hand-charted — an out-of-sample test, rules frozen before anyone saw the answers, on a night match between Nadal and Shapovalov. That test caught a mirror-image bug — forehand and backhand swapped, because the software assumed a right-handed player against two lefties — fixed with a tennis fact, not new vision code. A second test was already promised: a sunlit day match, Federer against Haase. Every bit of ball-tracking so far had been done by SAM 3, a general-purpose "find that object" AI that needed a box drawn around the ball by hand, and that charged a small fee per clip.
The specialist and the generalist
Picture two people you could hire to watch the ball for you. One is a touring pro: capable of almost anything, but before every point you have to walk over and tell them exactly what to watch. The other has had one job their whole career — they've stood on the same line, watching where the ball lands relative to it, thousands of times a day — and never once needed a pep talk first.
SAM 3 is the touring pro. WASB, the program introduced in this post, is the line judge: about the size of a phone photo, and it has only ever looked at tennis balls. Show it a photograph and it lights up how likely the ball is to be at each spot on screen, the way a heat map does, and picks the brightest spot, fresh, thirty times a second, no memory of the last one. That's a tiny specialist versus a huge, promptable generalist — more at Court Vision in Plain English — and it's what this post turns on. One side effect matters later: with no memory, WASB can't lose the ball the way SAM can. SAM carries the ball forward frame to frame — tracking, in the technical sense — and a tracker that loses its grip stays lost. WASB never had a grip to lose.
Somebody already solved this
An evening of careful research, checking every claim against more than one source, settled two things. First: finding the ball in a tennis broadcast is already a solved, free, open problem — a specialist almost exactly like WASB is out there, pretrained and downloadable today. Our own SAM-based tracking had been reinventing that wheel, worse, for a small fee on every clip.
Second, and this matters more: every system like that stops before the notation. None of them read a broadcast and write out who hit what, where, in the shorthand a human charter would recognize. That gap — not the pixels, the grammar — is the real target here. Somebody wants it filled, too: the volunteer project supplying our human answer keys runs today on just 32 unpaid, part-time people, and its founder has publicly asked for more help.
One more gap we could close ourselves: nobody had published a head-to-head test of a generalist against a specialist on a ball this small and fast. So we ran one.
What happened when we swapped them in
We wired the specialist's ready-made weights into the pipeline and left the notation-writing software untouched, so any difference in the output could only be the tracker's doing. It won almost everywhere it mattered — numbers below. It also did something SAM structurally couldn't: thirteen night-match clips never got a usable starting box for SAM, so SAM tracked zero frames of them. WASB needs no starting box at all — it tracked all thirteen anyway, 73–97% of frames each, no clicking, no cost.
Then something broke.
The bug that was secretly the whistle
On the night match, the number this project cares about most — shots counted in a point, checked against the human's count — got worse with the better tracker. Before the swap it had seven of eleven points right; after, only five. Every miss counted too many shots, never too few.
I went back to the photographs. Match point: the ball resting at the frame's bottom edge, the crowd already on its feet, mid-standing-ovation. A dead ball bouncing quietly in the corner while both players stood around between points. A ball drifting through the crowd above the back fence, long after the point ended. In every case, the point was over. The ball-finder was not.
Here's the mechanism. SAM tracks — it locks onto one ball and remembers it. When a rally ended and the ball sailed out of frame or vanished into the crowd, SAM's memory broke: its trail of dots died, right there, at the end of the point. Nobody designed that as a signal, but the software downstream — the part that turns ball positions into a chart — had quietly been leaning on it anyway, never told in so many words when a point was over; it had just been reading SAM's dying trail as the whistle. WASB never gets lost, because it never had hold of the ball to begin with — it just keeps finding it, dead ball and standing crowd and all, and the chart-writer kept faithfully logging "shots" out of a point already finished. A bug had secretly been doing somebody else's job.
The first fix I tried didn't work: a ball still in play after a real hit moves fast, a dead ball bouncing after a point should be slow, and that gap looked clean — until the day match's real serves turned out just as fast as the fake bounces, and the line I'd drawn cut through real shots too.
What worked was tennis, not more tuning: a ball still part of the point never leaves the court's playable area — the same screen-to-court conversion, the homography, used elsewhere in this project. Every phantom shot sat outside that area; every real shot, across both matches, stayed inside it, serves excepted (a legal toss goes up and out on purpose). One rule, one exception, and the phantom shots disappeared. Checked again against the human chart, the night match's shot-count came back to ten of eleven — the best this project has scored — while the day match's numbers didn't move by a single shot: proof the fix cleaned up the one problem it targeted and touched nothing else.
The scoreboard
| What we measured | SAM, the generalist | WASB, the specialist |
|---|---|---|
| Ball found on screen — night match | found it 82% of the time | found it 87% of the time |
| Ball found on screen — day match, bright glare | found it 74% of the time | found it 88% of the time |
| Points counted correctly — day match, 5 points checked | 1 of 5 | 4 of 5 |
| Cost to track a match's worth of clips | a small fee per clip | $0 |
Good, here, is the specialist matching or beating the generalist everywhere that counts, at zero cost — which is what happened, except on a handful of easy, brightly lit clips where the generalist still edged ahead by a few points.
Sometimes the scariest bugs in a pipeline aren't the ones that break something — they're the ones that had quietly been holding something else together.
For the technical reader
Landscape review: 23 sources fetched, top 25 claims adversarially verified, 25 survived. Specialist = WASB-SBDT (BMVC 2023, NTT, MIT license), pretrained tennis weights, 6 MB, F1 94–96 on the broadcast-tennis benchmark. TennisProject replicates our classical stack (ball, court keypoints, homography, players, bounces). No product/paper/repo found ingests broadcast video and emits a symbolic shot-by-shot chart: SwingVision wants your own phone footage, TennisViz consumes Hawk-Eye's tracking feed, Hawk-Eye runs ~EUR 60–70k/court/week. Match Charting Project: ~17,800 matches, ~a quarter of tour matches, Sackmann's January post pleads for more charters.
A/B, project shorthand: night match = t1, day match = t2. ~200-line driver (hard-asserts CUDA; ours runs on MPS, 15–20 fps), frozen chart loop untouched. SAM's disaster clips evaporate under the swap (55%→93%, 55%→87%); its only wins are 3–7 points on easy floodlit clips.
Regression: t1 rally length 7/11 → 5/11, all over-counts. Failed fix: velocity magnitude — coda bounces swing 109–210 px/frame vs real hits' 8–33, a beautiful gap until t2's serves landed inside it. Fix that held: one positional gate, serve-window exemption, three new constants, frozen-era constants untouched — t1 lands at 10/11, t2 byte-identical before/after. All 60 dev-reel clips retracked at $0, including 37 SAM never touched.
Session cost: $0.00. Project total: ~$4.15 of $9 — and nothing ahead of us charges by the frame anymore.