Today the software stops seeing players as smudges and starts seeing swings. The question: does that finally move the forehand-or-backhand call that's been stuck for two weeks?

Previously. The machine finds players by comparing each frame against an empty-court photo (background subtraction) — which yields a moving smudge per player, no arms, no legs. The forehand/backhand letter has been read from which side of the smudge the ball arrives on, and that idea has formally hit its ceiling: careful experiments showed no amount of re-anchoring the smudge changes the calls. Meanwhile this same week the court learned to fit itself and the machine grew ears. The players are the last part still made of fog.

Pose estimation: drawing the swing​

The upgrade is a technology every fitness app now uses: pose estimation — a model that marks seventeen body landmarks (wrists, elbows, shoulders, hips, knees, ankles) on any person in an image, turning a smudge into a stick figure. Instead of guessing a stroke from where a blob bulges, you can read the actual tennis: where the racquet wrist is, which side of the body it crossed to, how the shoulders line up at contact. A free, openly-licensed model does this at better than real-time speed on this laptop — no cloud, no cost.

The open question was never the near player, who fills a fifth of the screen. It's the far player, who stands about ninety pixels tall — thumbnail-sized. Research on this exact problem predicted: shoulders and hips will hold up, wrists will get shaky. The audit stills in the clip say the prediction was right on both counts — the far player's skeleton is real and readable, and the model's own confidence in the far wrists hovers around fifty-fifty, versus a solid two-thirds for shoulders.

The head-to-head​

Two matches with human answer keys, the two where the letter call is benchmarked: the Paris clay final and the Wimbledon final — grass being the surface where the smudges smear worst. Every method graded on the same shots against the same human charts:

On clay: a tie, two points better — exactly what the earlier counterfactual experiments predicted, since on clean footage the smudge and the skeleton mostly agree about where a body is. On grass, where the boxes fall apart: thirteen points better. The skeletons help most precisely where the old method fails hardest — the same pattern as the ears last post, and it's not a coincidence: every new witness earns its keep in the old witnesses' blind spots.

The blooper, kept​

The first version had a bug that one audit image exposed instantly. It matched each skeleton to the player by picking the detected person nearest the smudge's center — and on one clay point, a rogue smudge box (an old known disease) sat so far from the actual player that the nearest human was a ball kid at the back fence. The machine carefully drew a skeleton on the ball kid and graded his forehand.

The fix is pure tennis sense: a candidate skeleton only counts if its feet are standing in the striker's half of the court — the screen-to-court conversion makes that a one-line check. Ball kids kneel at the fences and net posts; umpires sit outside the lines; players play inside them. The fix costs coverage — with strict standing-room rules and shaky far-player ankles, the matcher now declines about half the shots rather than risk grading the wrong human — and recovering that coverage by tracking skeletons across frames, instead of judging each frame alone, is the obvious next version. Fourth time this week that a rendered picture caught what the accuracy numbers politely hid.

The takeaway: given actual arms to read instead of a fog bank, the machine's forehand-or-backhand call improved most exactly where it was worst — and the day's best evidence was one frame of a very well-postured ball kid.

Session cost: $0.00 — openly licensed weights, laptop inference. Project total: ~$16.

Next question: in one day the machine learned to call lines by physics, hear impacts, fit its own court, and read swings — four new witnesses, each stronger where the others are weak. None of them are wired into the shipped pipeline yet; they're experiments with receipts. The next build is the courtroom: the fusion decoder that hears every witness, weighs each one's track record, and writes the chart no single witness could — with the grammar of tennis notation making illegal testimony impossible to write down.

Today the software tries to take over the last job a human still does for every single match: teaching it where the court is.

Previously. Everything this machine calls — serves, bounces, wide-or-deep, who's standing where — runs through the homography, the screen-to-court-diagram conversion (primer) built from the court's painted lines. Since the very first fit, getting that conversion has ended one of two ways: an automatic fitter finds the lines, or — on stubborn footage — a human clicks the four corners by hand. The automatic fitter has a famous scar: at Roland Garros, faint brushed-over baselines and reddish stands fooled it so badly its fit missed the court by 74 pixels, and a human had to step in. Clicking corners takes a minute. But a minute per match, times the thousands of matches this project wants to chart, is the difference between a tool and a hobby.

Borrowing a scout who's seen 9,000 courts​

The fix wasn't invented here, and that's the point. Someone on the internet already trained a small network to find tennis courts in broadcast frames — not by hunting for four corners, but by recognizing fourteen landmarks at once: every corner, every T, every place two painted lines meet. It learned from 8,841 broadcast images across hard courts, clay, and grass. It's free, it's small, and it runs on this laptop in under a second per image.

Fourteen landmarks instead of four is the difference between asking a tourist for the two street corners they remember and asking a letter carrier who knows every house on the block. If a couple of landmarks are hidden or faint, twelve others still pin the court down — and the fitter can even check its own work, by fitting on some landmarks and measuring how well it predicted the ones it didn't use.

The scoreboard​

Each court's automatic fit, measured against the human's hand fit at six landmarks (the four court corners and the two net posts):

Seven of eight courts, fit to within a few pixels of a careful human, by a network that had never seen any of them. And the eighth result might be the most important one on the board: on the dim night match, the network found only 2 of its 14 landmarks — so instead of guessing from starvation-level evidence, the fitter abstained and left the job to a human. That's the behavior this whole project is organized around: an automatic call when the evidence is strong, a raised hand when it isn't — the same contract as the confidence flag that grades every drafted point.

The twist: the student corrected the teacher​

One court disagreed with the human by 16 pixels, and I went in expecting to document the network's failure. The overlay says otherwise: on that clay plate, the orange lines sit squarely on the painted court, and the green — the human hand fit that all the benchmarks treat as truth — floats visibly inside the paint on the far side. The "error" may be the teacher's. It's not settled yet — the neutral referee will be the pipeline's existing line-scorer, which measures any fit directly against the paint itself — but it's a healthy reminder this series keeps getting taught: hand-made truth is still hand-made.

The bug, kept on the record​

First run, every single fit came back wrong by the same enormous amount — about 750 pixels, all eight courts. A consistent error is a convention error, not a model error: the borrowed code, it turns out, already doubles its coordinates internally (it assumes 720p footage), and my wrapper doubled them again. One rendered overlay exposed it instantly; the numbers alone never would have named the cause. Third time this week: eyes beat derivatives.

The takeaway: the last per-match chore a human always did — teaching the machine where the court is — just went automatic for seven courts out of eight, and on the eighth the machine did the next best thing to being right: it knew it didn't know.

Session cost: $0.00 — pretrained weights, laptop inference. Project total: ~$16.

Next question: the court now fits itself and the machine hears every impact, but the players are still smudges — blobs from background subtraction, which is why forehand-versus-backhand has been stuck at a ceiling for two weeks. The research says the upgrade is stick figures: wrists, elbows, hips, feet, drawn on every frame by a free model that runs at 87 frames a second on this laptop. Time to give the smudges skeletons.

Today the software gets a second sense: when its eyes blink, can its ears carry the point?

Previously. The ball tracker — the tiny specialist network that finds the ball in every frame (the zero-dollar specialist) — has one chronic weakness: it blinks. On clay especially, its track is full of holes, and every downstream call (hits, bounces, rally length, wide-or-deep) inherits the blindness. Last post, physics recovered the landing spots the eyes never saw. This post goes after the holes themselves, with a witness that doesn't have eyes at all.

The witness that never blinks​

Think about how you experience tennis from the cheap seats, or the next room. You can follow the rhythm of a rally without seeing it: pock — pock — pock. The strike of strings on a ball is one of the most distinctive sounds in sport — sharp, short, and unlike anything else on a tennis court. Every broadcast microphone picks it up, no matter where the ball is, how small it looks on screen, or whether it just left the frame. Crowd roar rises and falls slowly; commentary murmurs along; the pock is a spike.

That's the whole idea: onset detection — scan the soundtrack for sudden bursts of energy in the frequencies where racquet impacts live, and mark each one's exact moment. No AI, no training: the software slices the sound into 11-millisecond snippets, watches how much new energy each snippet brings, and flags the spikes that tower over their neighborhood. Notice the unit: 11 milliseconds. A video frame is 33. The ears, when they're right, are three times more precise than the eyes can ever be.

First problem: the tapes were silent​

One discovery before any listening could happen: every video this project has ever downloaded is silent. The staging pipeline grabbed video-only streams from day one — it never needed sound until today. So the soundtracks had to be fetched fresh, audio-only, from the same uploads. For the Roland Garros match the source was known. For the US Open match it had to be re-found — and verified the way this project verifies everything: by receipt. The candidate video's length matched the match reel to a tenth of a second, and then the real proof: after lining up the two recordings, a single fixed time-shift (95 milliseconds on one match, 310 on the other) snapped hundreds of audio spikes onto video-detected hits at once. A wrong recording can't fake that — there is no single shift that aligns someone else's match.

What the ears heard​

Two matches, both already graded against human charts: the US Open final (hard court, where the eyes work well) and the Roland Garros final (clay, where they don't).

Read that last row again, because it's the headline. Across the clay match, the ball tracker spent 78 seconds of rally time seeing nothing — and the microphone logged 196 impacts inside exactly those seconds. The witness that never blinks works best precisely where the eyes fail worst. That's what the yellow-spikes-in-red-zones moments in the clip up top are showing.

And a small piece of physics showed up in the data, unasked. When the detector is set strict — only the loudest spikes — the number of impacts per shot settles at almost exactly two. Two impacts per shot is precisely what a court hears: the racquet's pock, then the bounce's thud. The microphone is telling us it hears both, cleanly enough to count.

The honest limit​

Set looser, the ears hear too much. A tennis broadcast is full of things that go pock: line calls, grunts, the umpire's microphone thump, a ball kid's bounce, applause spikes. At the settings that catch the most real hits, the detector logs about three impacts per shot — the extra one is chatter. Asked to count rally lengths by ear alone (spikes divided by two), it lands within one shot of the human chart only 3 times in 10 on hard court, worse on clay.

So the verdict, stated precisely: audio v1 is a witness, not a counter. It confirms moments and fills blind time with candidate events; it can't yet tell a forehand from a line call. Teaching it the difference — pock versus thud versus everything else — is a small classification job, and the training data is free: thousands of audio moments already labeled by the video hits they align with. That's the same flywheel this project keeps returning to — one sense teaching another, no human labeling anything.

The takeaway: the machine's eyes blink, but the court always makes the same two sounds per shot — and now the machine can hear them, timed finer than its own eyes can see.

Session cost: $0.00 — the audio tracks were free, and everything runs on the laptop. Project total: ~$16.

Next question: the ears confirm the eyes and cover their blinks, but both witnesses still hand their evidence to a court fit that a human clicked into place. Four corners, every match, by hand — the last big manual knob on the machine. The soccer world already builds courts that fit themselves and, more importantly, know when they've failed. Time to steal that too.

Today the software learns to answer a question it has been failing, completely, since grading began: when a ball goes out, did it go out wide or deep?

Previously. This machine watches broadcast tennis and drafts a shot-by-shot chart of every point, which then gets graded, correction by correction, against charts written by human volunteers (how the grading works). Most parts of the draft have been inching upward for weeks. One part has not moved at all: when a rally ends with a ball landing out, the machine scored zero percent at saying which way it went out. Not "poor." Zero — across 159 tries, on eight matches, twice confirmed by different methods. The reason is simple and physical: the ball tracker loses sight of the ball a split second before an out-ball lands. Fuzzy yellow blur, busy background, and the one moment that decides the call is exactly the moment the machine goes blind.

The obvious fix — make the tracker hold on longer — is the one I'd been planning. Then a research deep-dive turned up something better, hiding in plain sight.

Hawk-Eye never sees the ball land either​

Think about what Hawk-Eye actually shows you. That smooth animated arc, the ball flattening onto the paint, the "OUT" verdict by two millimeters — none of that is a photograph. No camera on Centre Court captures the ball touching the ground; at that instant the ball is a smear of pixels, half-hidden, moving at highway speed. What the system photographs is the flight. The landing is calculated — physics, fitted to the part of the flight the cameras saw clearly, finishes the sentence.

A veteran line judge does the same thing. She doesn't wait to study the skid mark. She reads the arc coming off the racket and knows the ball is sailing long before it lands. The information was in the flight the whole time.

That's the whole idea I stole this week: the boundary race — stop trying to see the landing, and instead extend the flight the machine did see until it crosses a line. If the extended path crosses the sideline first, the ball was going wide. If it crosses the baseline first, it was going deep. Whichever finish line the flight reaches first names the error. (One more entry in the plain-English primer, where the recurring ideas of this series live.)

No new AI. No new camera. Straight-line physics, fitted to the last stretch of dots the tracker recorded before it blinked — and two papers at this year's top computer-vision conference validate the family of idea, one of them placing landings within about 0.6 meters from a single broadcast camera.

What happened when I raced it​

I ran the race on every point in the benchmark — the eight matches where human charters already wrote down the truth — and looked only at the points that genuinely ended with a ball landing out: 169 of them, of which 159 were plainly wide or deep (the rest were charted as both at once).

From nothing to half, with arithmetic. And when the race actually picks a side — wide or deep, rather than hedging "both" or declining — it separates the two at about five in six, on flights the tracker gave up on.

The misses are honest and mostly one story: the TV camera looks down the court, so a flying ball always appears deeper than it really is — the same distortion that makes every approach shot look like it's sailing long from your couch. That bias makes the baseline finish line fire early, so some wide balls get called deep or "both." Fixing that lean is a real project for later, not a knob to fudge today.

What the video caught that the numbers didn't​

Rendering that clip up top wasn't decoration — it caught two real bugs the scoreboard had quietly absorbed.

First: in one point, the machine's "last sighting" marker drew itself at the top of the screen while the ball's trail exited bottom-right. Impossible. The track data showed why: the ball supposedly moved 37 meters in three frames. Balls don't teleport — the tracker had lost the real ball and latched onto something else, and those junk points were poisoning the physics fit. The fix is a rule of nature, not a tuned setting: end the flight at the first physically impossible jump.

Second: the race originally asked an earlier stage of the pipeline which end of the court the final shot was heading toward — and that stage was sometimes wrong. One point claimed the far player hit the last shot, while the ball demonstrably died two meters beyond the far baseline. Now the race reads the direction from the flight itself and trusts nobody.

Both bugs were invisible in the accuracy tables and obvious in one rendered frame. This series keeps re-learning the same lesson, so it's earned a bolding: eyes beat derivatives — when a number and a picture disagree, believe the picture.

The dead end, kept​

One more result, because this devlog keeps its failures on the record: I also tried the lazy version of "integration" — feeding the race's outputs as extra hints into the existing ending classifier, the one that learned from 11,646 human-charted matches. It did nothing. Zero percent stayed zero.

The reason is instructive. Winners — balls that land in and end the point — are nearly half of all endings, and thanks to that same looks-deeper-than-it-is distortion, the race happily extends many in-balls out too. A single flat classifier drowns the new signal in the old base rates. The race's real gift is conditional: given that a ball went out, it names the flavor. Deciding whether the ball went out at all needs different witnesses — a net-death, an observed landing, and soon (next on the roadmap) the sound of the ball itself. The architecture lesson: don't ask one judge to answer two different questions.

The takeaway: the machine stopped trying to see the ball land, read the flight instead — the way Hawk-Eye and every good line judge always have — and half of a stone-blind category came back at the cost of zero dollars and some algebra.

        w     d     x     ?
  w    20    11    27    11   n=69
  d     4    55    18    13   n=90
  x     1     0     8     1   n=10

Session cost: $0.00. Project total: ~$16.

Next question: the race can name an out-ball's flavor, but something still has to say "that ball went out" in the first place — and the tracker's eyes have proven they blink at exactly the wrong moment. The next witness doesn't have eyes at all. Every broadcast microphone courtside hears the same thing you do: the pock of strings on ball. Time to give the machine ears.

For sixteen posts, this project has graded its own homework — checking drafts against charts it already had the answers to. Today, for the first time, a draft goes to a real judge: someone who charts tennis for a living, whose only job is to say whether it helped.

Previously: cv-01 set the destination — the Match Charting Project (charters call it the MCP), run by Jeff Sackmann of Tennis Abstract, is volunteers watching tennis and writing down every shot in a compact code; nearly 18,000 matches charted so far. cv-14 gave the project a way to grade a machine-written line against a human one: how many corrections an editor would need, the token edit distance (more here). cv-15 turned that grade into a stoplight — HIGH ("start from this draft") or LOW ("re-chart it") — tuned so the flag never grades its own homework (more on confidence). cv-16 grew the test to seven matches across five broadcast feeds, 491 human-charted points, fixed a blind spot on the Wimbledon match, and landed the number this post leans on: HIGH is right about 94% of the time.

None of it has been true for long. In 2025 the MCP ran on 32 active charters, covering about a quarter of tour matches — and Sackmann published a plea for more. That's the door this post walks through.

The human loop​

Being graded correctly isn't the same as being useful. Groove a stroke against a ball machine for months and you prove your technique reproduces — never that it survives a live point, because a machine can't return your shot and tell you how it felt. Every number this project has published so far graded the pipeline's aim, not its use: does the drafted string match a human's, was a flag right when checked later. Nobody who charts a match for a living has ever opened one of these files and tried to work from it.

That's the gap this post closes. The step where a tool's output leaves the lab, lands on a real person's desk, and the only score that counts is what they report back, is the human loop: grading against a fixed answer stops, and getting judged by whether you saved someone real time starts. Nothing in the pipeline changed this session — who's on the other side of the net did.

In tennis terms: the ask is what a doubles player makes of a prospective partner before ever stepping on court — play one set with me, then tell me if I made you faster or slower.

What ships today​

Seven files went public today, one per benchmark match, in the exact column layout the MCP already uses for its own point-by-point files. The familiar columns are all there — match, point number, set score, game score, point score, server — a real charter could read those blind. Then come the columns Court Vision added:

A real row, from the US Open final, flagged HIGH at 0.902: s5b2f1f2f2x@. Read left to right: s5 — serve into the body. b2 — backhand return, up the middle. f1, f2, f2 — three forehands, one into an outer third, then twice down the middle. x@ — unforced error, wide and deep. Six tokens, one point.

Every match's file, with how many points it drafted and how many got the green flag:

99 of 508 — about one point in five — got the green flag; cv-16 found it right about 94% of the time on matches the grading model never trained on. Start with the US Open final: best hit rate in the batch, and the model called who served right 95% of the time there too — a camera that holds still pays off.

A drill doesn't prove anything until it's played against a real opponent. That's the whole reason this post exists.

The honest label​

A draft that hides its weak spots wastes an hour, so the weak spots come first.

The drafts are drafts. Across all 491 human-charted points behind the whole benchmark, only 5.7% — roughly 1 in 18 — match the human's line closely enough to need just one fix. The real value sits in the green rows: HIGH means within five corrections, holding up 94% of the time on matches the grading model never trained on (96% on matches it did study — the friendlier, less trustworthy number, included anyway). LOW, 409 of 508 rows, means exactly what it sounds like: re-chart from the video, draft as hint, serve_s as the jump-to.

Every missed serve is invisible. The pipeline can't yet see a fault, so every point is written as if won on the first serve, and the second-serve column is empty on all 508 rows. On a real scoresheet, roughly a third of points start with a miss. Fixing that isn't a footnote of the charter's pass — it's most of it.

The shorthand isn't official yet. The s before a serve number is the pipeline's own invention, not real MCP notation, and a ? shows up when the machine can't tell which way a shot went. The draft also skips most of what a full MCP chart captures — no depth, no court position, no volley vs. lob vs. drop shot; endings and rough directions only. Whether to write strict, official strings with blanks instead, or keep the uncertainty visible, is undecided — question three, below.

Seventeen rows have no point number. The machine caught the point on video but couldn't match it to a spot on the scoreboard, so that column is blank. The point happened; a person still has to say where it belongs.

One answer key has problems of its own. cv-16's autopsy caught the Wimbledon final's human chart running at half speed on some rallies — strokes written down that the video contradicts. That file still ships, 18 of its 49 rows green. Open that one first if you're hunting for the model being too sure of itself.

The ask​

If you chart for the MCP — or ever have — three questions matter here, in the order they'd change what gets built next.

1. The stopwatch. Chart ten points cold, straight off a clip. Then correct ten different green-flagged drafts against their clips instead. Which pass is faster, and by how much? The whole project is a bet on that ratio, never yet measured by anyone who actually holds the pencil.

2. The triage. Is a draft right 94% of the time, within five corrections, the right bar for "start from this"? And with only about one point in five earning that flag, does a pre-filled file even help mid-session — or does switching between "check this" and "chart from nothing" cost more than it saves?

3. The schema. What would this file need before you'd reach for it mid-session — official MCP strings with blanks, a real fault column, a charting-sheet layout instead of a spreadsheet, something else entirely?

The channel is an issue on court-vision — a one-line "this wastes my time because X" is signal, not rudeness. It would be the single most valuable sentence anyone has contributed to this project.

Lineage and license​

Worth knowing before opening these files: the score and server columns exist because these drafts were lined up against the MCP's own published charts — the same charts the 94% number is graded against. So these files carry the MCP's license, CC BY-NC-SA 4.0: credit the source, don't sell it, share what's built from it the same way. The draft strings and confidence columns are this project's own addition, released under that license to keep the files one whole thing.

Session cost: $0.00 — nothing ran but a video renderer. Project total: ~$16. The next number that matters isn't the pipeline's to produce: it's one charter's minutes-per-point, with and without the draft.

Would you trust a player's world ranking more if it came from one dazzling week at one tournament, or from a whole season played across different surfaces and different opponents?

Two posts back, cv-14 gave the project its scorecard: count how many fixes — changes, insertions, deletions — a human editor would need to turn the machine's chart into the real one — an acceptance at one fix or fewer. cv-15 then built a trust flag on top of that scorecard: every drafted point gets stamped HIGH or LOW, and HIGH means "start correcting from this draft." But that flag had only ever been tested against 135 points from four matches — barely 1 in 9 of which passed the one-fix bar to begin with — and cv-15 signed off with HIGH at 93% reliable, plus a warning: 135 points wasn't enough to trust yet, and three more matches were already on the way.

Here's why that matters. A hot week can flatter a ranking — a small field, a favorable draw, a home crowd — and it isn't wrong exactly, it's just thin. A full season is the harder test: more matches, tougher and more varied opponents, and the ranking is allowed to go down afterward even though the player didn't get worse. It's finally being asked to hold up over more evidence, and a ranking that survives a full season is worth more than a flashier one built on a single great week, precisely because it was harder to earn.

That's recalibration: redoing the trust-score math against a bigger, tougher set of examples, and trusting the number wherever it lands — even down. (More on how these trust scores get built.) The confidence flag lives or dies on two numbers: how often it's even willing to say HIGH at all — called coverage — and how often HIGH is actually right when it does — called precision. It's the same trade a line judge faces: call everything close and you're helpful but wrong a lot; call only the obvious ones and you're always right but useless. The cv-15 flag had only faced one hot tournament's worth of evidence. This session gave it a full season.

Three more finals, same machine​

The pool of candidates was every match the Match Charting Project's volunteers have hand-charted — this project's answer key, explained here — cross-checked against what's findable on YouTube. The winning format was the "condensed match" upload: a 20-to-40-minute re-cut that keeps nearly every point and skips only the between-point filler, which lines up with the human chart far better than a highlights reel. Three finals made the cut: the Australian Open final at night, the U.S. Open final, and the ATP Finals indoors in Turin — 575 more charted points to draw from.

All three ran through the pipeline's existing machinery untouched — no new code, just settings. Each broadcast had one quirk: a court's blue paint too bright for the usual color threshold, a green sideline apron too close to the court's own color, a hazy light-blue apron needing a brighter ceiling — all one-line settings fixes. One old lesson got relearned, too: the recipe that converts a screen position into a court position — the homography — has to be built from footage where the camera is actually holding still. The first attempt smeared because the camera was still panning; a static five-second window during the serve toss fixed it.

New broadcasts also surfaced a new headache and caught an old bug. The headache: all three hide the scoreboard graphic at 0-0, briefly confusing the tool that uses it to tell points apart — solved by cross-checking the human charter's own serve notes. The bug: the method for tracking which end of the court a player serves from had been quietly miscounting tiebreak games as a full 13th game instead of the shorter, differently-scored game it actually is — a one-line fix that flipped a whole match's worth of serve-end guesses from mostly wrong to mostly right. And on the record: an overnight background job died silently, and two matches' worth of finished work sat untouched until someone checked by hand.

None of that dented the results. Scored against the pipeline's unchanged rules, the three new matches set new highs: the U.S. Open match got the serving end right 121 times out of 128 (95%), and Turin got rally length right, give or take one shot, 131 times out of 157 (83%). On the strict pass-fail test, the bigger set held essentially flat: 28 of 491 points needed no correction at all (5.7%), close to the earlier 7 of 135 (5.2%). The pipeline didn't get worse by testing it harder.

The honest number goes down, on purpose​

On the small set — 135 points, four matches — the confidence flag's HIGH rating was right 93% of the time, willing to call HIGH on 32.6% of points. Recalculated on the full season — 491 points, seven matches, five broadcasts, same math, no changes — the honest numbers came in lower: 88% right, called on 21.2% of points. That's not the tool getting worse. That's small-sample flattery washing out.

One match makes the point almost too neatly. In the small test, the Wimbledon match looked flawless: 11 for 11, 100% right, the best result in the table. In the big test, with more than double the Wimbledon points to check, that same match fell to 69% right while claiming to flag more than half the match. The small test wasn't rigged — it just didn't have enough Wimbledon points yet to catch what was wrong. More data didn't break Wimbledon's score. It let the project finally see it, and go looking for why.

The Wimbledon blind spot​

The why turned out to be a single broken check.

cv-15's best idea was a serve check: a real toss lands the ball's launch point more than 20 meters beyond the baseline, so a "serve" that starts from inside the court is really a broadcast cutting into the middle of an already-running point — and the draft gets marked down for it. On its own, it had been the confidence project's single biggest improvement.

But it only works when the system identified the serve by watching the ball's own flight. When the ball-tracking data is too spotty to trust, it falls back to guessing the serve from the players' stance and position instead — and on the Wimbledon broadcast, every single serve was identified that second way. So the serve check had zero real serves of its kind to examine, for an entire tournament's worth of points. It wasn't passing every point because it checked them and they held up; it had nothing to check, and defaulted to "trust it" silently, every time.

Under that free pass, three failures went unnoticed. One point's drafted chart recorded 8 shots, evenly spaced about a second and a half apart — but the real footage runs twice as fast, a shot every 0.9 seconds, meaning the shot-detector had missed every other stroke. It still looked trustworthy because the who's-hitting-next check happened to add up anyway — the same ball-tracking blind spots hit both checks the same way, so the errors canceled out instead of standing out. A second clip opens on a slow crossfade into a rally already underway — five real seconds of tennis before the stroke the stance method called "the serve." cv-15 had already caught this trick once, on the clay-court match, where the broadcast cut hard into the middle of a point; Wimbledon's editor does the same thing with a fade instead of a cut — exactly what the (silently broken) serve check exists to catch. And a third case: a four-shot rally in the human's chart where the ball-tracking data never shows a single net crossing — the ball's path across the net is a rally's skeleton, and this rally had none.

The fix wasn't to patch the broken check. It was to add two checks that look at the whole point instead of just the serve. First: does the ball cross the net, at rally speed, before the moment the chart claims the serve happened? If so, the clip started mid-rally already — catching both the hard-cut and the fade. Second: does the charted shot sequence show any net crossings in the tracking data at all? No crossings, no trust, however many shots the chart claims. Swapping the one broken check for these two took Wimbledon's held-out score from 69% to 85%, and the project's overall number from 88% to 94%, while cutting the worst-case disaster flags roughly in half.

The project already learned this lesson once, in a different costume. Back in cv-14, the machine's shot-direction calls came out backwards for a large share of points, because the code assumed a fixed side of the court instead of checking which end the receiver was standing at — nobody noticed until the scorecard itself gave the bug away. The Wimbledon check is the same mistake in a different jersey: a trust score is only as honest as its least-examined assumption. The project's best idea had quietly been a rubber stamp on one entire broadcast. The audit question was never "is this check smart" — it's "can this check even fire, here, on this footage."

Where the honest number landed​

The trade for that last jump is real: 1.6 fewer points of coverage, for a genuinely more honest number — the right trade every time. See Court Vision in Plain English for why coverage and precision have to be read together.

A let-cord sensor that never buzzes isn't proof of a clean serve. It might just be unplugged — and the only way anyone finds out is by checking whether it could have buzzed at all.

metric              t5 AO night    t6 USO WTA      t7 Turin indoor
server end          53/71  (75%)   121/128 (95%)   133/157 (85%)
rally length ±1     47/71  (66%)    99/128 (77%)   131/157 (83%)
letters (aligned)   93/117 (79%)   120/148 (81%)   112/157 (71%)
acceptance ≤1 edit   2/71          10/128            9/157

LOMO (held out)   high prec (≤5 edits)   coverage
t1 night              10/10 (100%)         45.5%
t2 ctrl                 3/3 (100%)         60.0%
t3 clay               11/12  (92%)         20.3%
t4 grass              18/26  (69%)         53.1%
t5 AO                   3/4  (75%)          5.6%
t6 USO                34/35  (97%)         27.3%
t7 Turin              13/14  (93%)          8.9%
pooled                92/104 (88%)         21.2%

LOMO (held out)      before                after
t4 grass         18/26 (69%) @ 53.1%   17/20  (85%) @ 40.8%
pooled           92/104 (88%) @ 21.2%  90/96  (94%) @ 19.6%

Where things stand now: seven matches, five broadcasts, 491 human-charted points on the books, and a HIGH flag that means "start from this draft" and is right about 94% of the time on the honest, held-out test — for roughly 1 in 5 points confident enough to earn the flag at all. The benchmark finally has the size its own warnings had been asking for.

Session cost: $0.00 — three tournament finals' worth of brand-new answer-key data, added for zero marginal dollars. Project total so far: about $16.

Say a machine hands a volunteer chart-corrector a stack of drafted points — how's she supposed to know, before she's watched a single one, which drafts are worth trusting and which she should chart from scratch?

Previously: cv-14 gave the project its report card — the token edit distance, how many fixes an editor needs to turn the machine's chart into a human's, a point "accepted" at one fix or fewer. cv-10 set the rule for grading any of it: score only against matches a human already charted, never a match the pipeline saw while being tuned. And since cv-05, players get found for free by background subtraction — photograph an empty court, call anything different in a later frame a person. Today tests whether that free method is good enough, then teaches the draft to flag its own reliability.

A confidence flag is a line call​

Picture two line judges. The first only calls a ball "out" when it lands a clear foot past the line. She's almost never wrong, but rarely speaks, leaving a player who trusts only her guessing on everything else. The second judge calls anything that looks close — she catches nearly everything worth catching, but plenty of her calls are wrong on replay.

A confidence flag sits on that same spectrum, and two numbers say where. Coverage is how often it's willing to call HIGH at all — how often the strict judge speaks. Precision is how often a HIGH call is actually right. A flag that's always right but rarely fires isn't useful, and neither is one that fires constantly and is often wrong — you need both numbers together. (More in the primer.)

Making those numbers honest takes a third idea: calibration. Take points a human volunteer has already charted — the answer key — and check signals the pipeline keeps about its own effort: did the serve call commit cleanly? does the ball's path have gaps? did two vote-counting methods for "who hit that" agree? Fit those signals to the real outcomes and the fit becomes a small scoring formula.

The trap: a formula that peeks at a match's answers while being built will always look brilliant tested on that same match — an open-book quiz, not a test. So fit it on three matches, then grade it, cold, on the fourth it's never seen, rotating until every match gets a turn held back. Tennis translation: the system never grades its own homework — the same rule cv-10 set for the pipeline, now applied to the flag that grades its work.

Is the pricier player-finder worth $12?​

Today's first question: how much of the draft's remaining mistakes trace back to the free player-finder misjudging where people stand?

Every shot call — forehand or backhand, roughly which third of the court — comes from one rule: whichever side of the player's outline the ball arrives at, that's the shot. A bad outline poisons the call, already the biggest single source of remaining edits, about 1.7 corrections per point. Sorting every scored shot by outline quality at contact:

Nearly half of every shot call — 45% — was riding on a shaky or missing outline; accuracy was 71% on clean outlines, 39% on shaky ones. Ceiling: about 26 more correct calls.

Free cleanup — rejecting teleporting outlines, checking court-half, filling short gaps, widening for legs-only silhouettes — bought back 3 of those 26: about an eighth of the ceiling. The rest was missing data, not noisy data: the far player usually isn't in the free tracker's output near contact. No cleanup invents a player the tracker never saw.

That earned the right to spend money: SAM 3 — the promptable AI that tracked this project's first ball, back in cv-02 — re-found both players in the Roland-Garros match only, prompted from the free tracker's own boxes, for about $12. Shot-call accuracy went from 59% to 67%, mostly outlines that had been missing — 21 calls flipped right, one point gaining seven alone.

It still didn't move the number that matters — full points a volunteer would sign off with one fix or fewer, stuck at 3 of 59. A better shot call wasn't holding those points back, and the paid tracker inherited the blind spot it was hired to fix: where the free tracker drew no usable box, an automatic prompt had nothing to point at either. Verdict: the $0 tracker stays default — what every single-piece upgrade has said since cv-14: improving one ingredient nudges the score without changing which points a human would accept.

One tool, not twenty scripts​

Before any of that shipped, this project's first fourteen posts' worth of scattered experiment scripts — twenty-odd files, four near-identical chart-builders kept in sync by hand — got folded into one package: one chart-builder, one grading tool, one command-line tool, broadcaster differences moved into a settings file per match. Condition for merging: it had to reproduce every number this series has published, byte for byte, before anything new shipped on top. It did, down to 138 chart files and four scorecards.

Building a flag that isn't lying to you​

With the free player-finder settled and the tool consolidated, the actual flag got built — the scoring formula above, trained on the pipeline's self-reported effort and graded the never-grades-its-own-homework way.

Two surprises turned up. Aces — serves the returner never touches — were scored as if the machine had refused every shot in the point, since there are no rally shots to refuse. Backwards: an ace is the cleanest point in the match, punished for being short. Fixed by teaching the formula that no shots after the serve means the point ended cleanly.

Stranger: on the Roland-Garros match, the broadcast editor sometimes cuts into a rally already in progress. The serve-detector, seeing the ball cross the net for the first time in the clip, calls that crossing "the serve" — even though a real serve toss lands 20+ meters past the baseline, and this one launched from inside the court. That's a clip that joined mid-rally, not a serve. Teaching the formula that one tell took held-out reliability from 65% to 84%.

One more result belongs here because it failed: a stricter flag — "sign off without watching the point at all," 85% accurate or better. Tested the same honest way, it landed at 50% precision, a coin flip, willing to call only 1.5% of points that confident. There aren't enough graded points to support it: of 135 hand-charted points, only about 1 in 9 already lands within two corrections. That tier is dead, reported dead rather than dropped quietly — a confidence system that hides its own failures isn't one anyone should trust.

The scorecard​

What did the shipped flag earn, tested the honest way? Read the pooled number the way you'd read a strong first-serve percentage — not a guarantee, but a number worth building on:

HIGH means "start from this draft" — 93% of the time, held out, it needs five corrections or fewer. It doesn't mean the draft is already right: only about a quarter of HIGH points need two corrections or fewer. Missing HIGH means "expect heavy correction" — more than half of those need six or more, so treat it as a hint, not a finished chart. The Roland-Garros match stays the shakiest, not because its math is worse — its disease is missing footage, shots the editor cut before the camera ever saw them, which no signal can detect from film that doesn't exist.

Said the way a tennis fan would at dinner: a confidence flag only means something once it's been tested on points it never got to see the answers to first.

outcome         sane   implaus  absent
right             70      31       0
wrong             17      18       0
refused           11      23       8
accuracy          71%     39%      --

t3 letters              bgsub+hygiene   SAM-3
strict positional        67/114 (59%)   76/114 (67%)

LOMO (held out)   high prec (≤5 edits)   coverage   low ≤5 rate
t1 night              11/11 (100%)         50.0%        27%
t2 ctrl                 3/3 (100%)         60.0%        50%
t3 clay               16/19  (84%)         32.2%        43%
t4 grass              11/11 (100%)         22.4%        50%
pooled                41/44  (93%)         32.6%        44%

None of this moves the project's real ceiling, unchanged since cv-14: only 5.2% of drafted points — about 1 in 20 — are an exact-or-one-fix match to a human chart, and even counting anything within five corrections as close enough, that's still just 57%. The ceiling isn't stuck on any piece of the pipeline anymore — it's stuck on the answer key, only 135 hand-charted points, barely 1 in 9 of them already within two corrections. More matches are already being lined up, and the moment they land, every number here gets recalculated against a bigger, harder answer key.

Session cost: about $12 — the SAM test, under a one-time $30 limit — then $0.00 for everything built after it. Project total: about $16.

Here's the question this post is finally built to answer: if you handed the machine's chart of a point to the volunteer who charted that same point by hand, how many things would she have to fix before she'd sign her name to it?

Previously: last time (cv-13), the software learned to read the small scoreboard graphic in the corner of the screen. That graphic stays pixel-for-pixel identical for the whole point and changes only the instant a point ends, so reading it let the pipeline do two things: glue video clips back together into whole points, and look up the exact matching point in a human volunteer's hand-charted record of the same match — the ground truth this whole project has been grading itself against since cv-10. That match-up is what makes this post possible: for the first time, every point the machine drafts can be checked, one for one, against what a real person wrote down for that exact point. Before today, the software also had a specific way of finding hits and bounces — watching for the moment the ball's path bends backward on itself, the court doesn't return serve, from cv-04 — a rule that mostly worked but quietly missed shots hit far from the camera.

How do you grade a chart?​

Think about how a coach would grade a rookie scout's notes on a match. The scout writes down what happened, point by point, in shorthand. The coach already has her own notes on the same match. She doesn't grade shot by shot — she reads the scout's line for a point next to her own line for that same point and asks one blunt question: how many words would I have to cross out, change, or add to turn the scout's version into mine?

That's exactly what this post's headline number measures, except the "shorthand" is real tennis charting notation and the "coach" is a volunteer from the Match Charting Project — the same ground truth this project has been grading itself against. Write a chart of one point as a list of pieces, or tokens: the serve and where it landed, then each shot's stroke and direction, then how the point ended. Do that for the machine's version and the human's version of the same point, then count how many of those pieces would need to be changed, added, or removed to turn one list into the other. That count is called token edit distance, and it's now the north-star number for the whole project — the honest answer to "would a human sign off on this."

A point counts as accepted when that count is one or less — the kind of chart a charter would wave through after fixing at most one thing. And there's a rule that keeps the machine honest: if it isn't sure and writes a "no idea" placeholder instead of guessing, that placeholder still has to be fixed later, so it costs exactly as much as getting the answer wrong. Shrugging is not a way to dodge a mistake.

Rebuilding how hits get found​

Before any of that grading could happen, the event detector needed work. The old approach hunted for a kink in the ball's flight path — the moment its direction snaps backward, which is how a hit differs from a bounce — but a kink from a shot on the far side of the court, seen from a raised broadcast camera, can shrink to a single wobbling pixel that's easy to miss entirely. So the detector got rebuilt around a blunter, more reliable piece of tennis sense: a rally in progress sends the ball across the net every single shot. Instead of squinting at every wobble, the software now counts how many times the ball crosses the net and adds one for the serve — and that crude count alone, before any fine-tuning at all, already beat the entire old wobble-hunting detector at guessing how long a rally ran. The rebuilt detector went on to sharply improve how often it read stroke letters correctly, though one older test clip actually got worse in the process — a small, honestly reported step backward, not swept under the rug (exact numbers in the appendix).

The first reading: 2.2%​

With the rebuilt detector in place and every point matched to its human original, the project could finally run the grading question for real — across four matches, three court surfaces, both the men's and women's tours: how many of 135 points would a volunteer charter sign off on with at most one fix?

The answer: 3 points out of 135 — 2.2%. Before the detector rebuild, using the old event logic, that number was zero. Two percent isn't a good score. But it's the first honest score — a real distance to the top of the mountain, measured in a unit a volunteer charter actually cares about, instead of a pile of separate stats that could all be quietly improving while the finished chart is still wrong.

Where the fixes were coming from​

A 2.2% pass rate means the typical point needed about seven fixes, on average, to become the human's version — so the project broke that average down by fix type to see where the effort was going. Missing shots — a shot the human charted that the machine's chart skipped entirely — were the single biggest source. But close behind, and the most fixable-looking, was shot direction: which way a shot traveled, not just what stroke it was. Getting the direction digit wrong, whether alone or combined with the wrong stroke letter, accounted for 39% of every fix an editor would have to make — the largest chunk of the whole edit budget. And the direction guesses weren't just wrong often: the machine only ventured a direction guess on 70% of shots, and when it did, it was right just 48% of the time — barely above the 33% you'd expect from picking one of three directions at random.

A compass pointed the wrong way​

That's a big enough number to be worth chasing, and the cause turned out to have nothing to do with cameras or vision at all — it was a definition problem. The Match Charting Project's own instructions define one shot direction as "a right-hander's forehand side" and add, in parentheses, "a lefty's backhand." Read carelessly, that sounds like direction depends on which hand hit the ball. It doesn't: the parenthetical is the tell. Direction is defined relative to a fixed side of the court — specifically, the half that's receiving the shot — and it only happens to line up with "forehand side" for one particular kind of player. The pipeline's code, though, had been computing direction using fixed court thirds, as if the same physical side of the court always meant the same direction: correct by pure coincidence for a shot going to the far end, backwards for a shot going to the near end. It was a compass mirrored for half the court the whole time. Once the definition got fixed — direction read relative to which half was receiving, instead of a fixed physical side — direction accuracy jumped from 48% to 75%, the machine started venturing a guess on 97% of shots instead of 70%, and overall acceptance nearly doubled: from 3 points out of 135 to 7, from 2.2% to 5.2%.

The scoreboard​

Four matches, three surfaces, both tours — here's how many of each match's points a human charter would sign off on with at most one fix, after the compass fix went in:

That's up from 3 of 135 (2.2%) before the direction fix. The honest low point on the board is Wimbledon: zero points accepted, because its problem isn't a wrong direction digit — it's shots that shouldn't be in the chart at all, or shots missing from it (more in the appendix).

The takeaway a tennis fan could repeat at dinner: what matters isn't how many little stats are climbing, it's how many marks a red pen still needs — and fixing one wrong assumption about which way "direction" points nearly doubled how many points needed no more than one.

edit category (mean/point, n=135)      overall
deletion (shot never charted)          1.95
sub: letter AND direction wrong        1.61
sub: direction digit only              1.20
sub: ending token                      0.70
insertion (phantom shot)               0.54

counterfactual                        accept ≤1      mean dist
baseline (v5 as charted)              3/135 ( 2.2%)  7.18
directions perfect (all shots)        9/135 ( 6.7%)  5.88
endings perfect                       7/135 ( 5.2%)  6.47
letters + directions                 21/135 (15.6%)  3.93
letters + dirs + endings             53/135 (39.3%)  3.23
all components (structure residual)  59/135 (43.7%)  2.93

metric              t1 night   t2 ctrl   t3 clay      t4 grass
server end          10/22      2/5       48/59 (81%)  37/49 (76%)
rally length ±1     13/22      5/5       36/59 (61%)  28/49 (57%)
serve zone          11/12      1/3        8/17        15/32
letters (aligned)    9/11      12/12     65/85 (76%)  15/28
ending type          9/16      2/3       19/42         9/33
acceptance ≤1 edit   3/22      1/5        3/59         0/49

acceptance at ≤ k token edits (n=135):
≤1: 2.2 → 5.2%    ≤2: 6.7 → 11.1%    ≤3: 14.8 → 23.7%    ≤5: 41.5 → 57.0%

Two percent was the first honest look at how far this project has to climb. Five percent, after fixing the compass, is the second look. The real change isn't the number itself — it's that the climb now has arithmetic behind it instead of a guess.

Session cost: $0.00. Project total so far: about $4.15 of a $9 budget.

Every TV director cuts away sometimes — a reaction shot, a replay, a different angle. So how do you prove, from footage alone, that two clips are really the same point? Two test matches weren't enough, so this session added harder ones: the 2023 Roland Garros final on clay, and the 2024 Wimbledon final on grass. Four matches now, three surfaces, both tours, about 108 new points checked against a human's chart, for zero dollars. What broke wasn't one narrow piece — it broke the assumption under every number on the board, and the fix had been on screen all along.

Previously: cv-10 gave the project its first honest report card, grading its draft charts against a match a human had already charted by hand. cv-12 then swapped in cheaper, more reliable ball-tracking software — and uncovered an accident: the old tracker's habit of losing the ball when a rally ended had secretly been flagging "this point is over." The new tracker never loses the ball, so that free signal vanished, leaving the project needing a real point-boundary detector. cv-06 had already learned a broadcast is mostly not rally footage — finding the stretches that even show the court took its own detective work.

The one graphic that never lies​

Picture a broadcast with the sound off. Almost everything keeps changing — players move, the camera pans, the crowd shifts — except the scoreboard graphic in the corner. It isn't part of the court; it's stamped on top of the picture like a watermark, so a pan can't nudge it. And it obeys a rule nothing else on screen does: frozen, pixel for pixel, for the whole length of a point, changing at exactly one moment — when the point ends.

That discipline is what this series calls the score bug as metadata (more in the primer): the graphic, just by sitting there frozen, quietly carries the match's structure. Two things fall out of it for free. It's a point ID — two chunks of video showing the identical frozen scoreboard are fragments of the same point, however the editor cut between them, and can be safely stitched together. And it's a join key — a shared label matching this clip to its point in the human charter's own record.

Clay and grass fight back​

The clay final (Djokovic vs. Ruud) broke the court-mapping recipe that worked fine on hard courts. Turning the TV picture into a court diagram needs the homography — the screen-to-court conversion recipe — and its usual tricks assume a blue hard court with crisp white lines. Clay has neither: the "find the blue court" trick found nothing, and the trick that erases players by blending frames together smeared the faint lines into the dirt instead. A clay-specific rebuild (appendix has the method) got it working again, accurate to a fraction of a pixel.

The grass final (Krejcikova vs. Paolini) broke the check on the recipe instead: the software found a court mapping whose lines lined up neatly with each other — and it was wrong, like an umpire confidently announcing a plausible score that isn't the real one. That's an impostor fit. The fix: instead of "do the few lines this mapping used line up?", ask "do all nine of the court's real lines land where this mapping says they should?" A wrong mapping explains a few lines and leaves the rest floating; the real one explains all nine. (One more check earned its keep on this match too, following cv-11 — the appendix has it.)

A "segment" is not a "point"​

Both matches mapped, the draft charts still came back mediocre — for a reason underneath every number, not inside any one of them. The pipeline assumed one unbroken stretch of court-view video (a segment) equals one point. That breaks in opposite directions, depending on who's holding the editing scissors.

Roland Garros cuts away mid-rally, chopping one point across several segments: the draft charted just 3.9 shots per point here against the human charter's real 8.9. The rally was there; the video just told it in pieces. Wimbledon does the opposite, almost never cutting between points, so one fifteen-second stretch can hold two or three back to back — the draft overcounted there, 8.1 shots per point against the real 7.2, stringing separate points into one fake rally.

The two earlier test matches had simply gotten lucky: their broadcaster cut almost exactly once per point, splitting the match for free — the same kind of accident as the old tracker's habit of losing the ball, doing a job nobody built it to do.

Reading the scoreboard settles it​

This is the gap the score bug closes. Two chunks of video with the identical frozen scoreboard, however the editor cut between them, are the same point — merge them. One chunk where it changes partway through is secretly two points — split it there. Read the sequence of changes and you get the real point-by-point structure, regardless of where the editor cut. It held with a wide, safe margin between "same point" and "different point" readings, and a handful of broadcaster quirks that tried to break it — a scoreboard that grows after every set, a tiebreak banner that shoves it sideways, a fade-in that could fake a change that never happened — all got caught by looking closely at the pixels (appendix has the numbers).

One theory got tested and killed outright, worth saying since it contradicts what I'd guessed going in: I suspected slow-motion replays were sneaking duplicate points into the count. They aren't. Every suspicious duplicate score turned out to be fragments of one rally, or a genuinely different point that passed through the same score — deuce recurs, and the scoreboard alone can't tell those visits apart, only their order can. The last suspect was cleared by physics: the ball's own falling motion matched real-time gravity, and a slow-motion replay would fall roughly four times slower.

With segments stitched into points correctly, the clips lined up against the human record almost perfectly.

The scorecard​

Matched row for row against the human-charted record — the independent answer key this project grades itself against:

The two rows that went down are real, worth naming rather than hiding: merged clips can start mid-rally, so the software falls back to a weaker clue (simple alternation) for who served and how it ended. A fair trade for getting the boundaries right, but a real cost. Where the merges had footage to work with, the payoff showed: the 16-shot rally up top went from four broken pieces to one point charted 12 of 16 correctly, and another clay point came back a perfect 12 of 12.

Two problems stay open, neither a scoreboard problem anymore. On clay, the editor sometimes cuts into a rally already in progress, so some early shots were never filmed — the scoreboard proves the clip is one point, but can't invent missing footage. On grass, 19 clips still count as two points when they're really one, because Wimbledon's dead time sits inside that same frozen reading (mechanism in the appendix). Both are chart-reading problems now, not video-editing ones — and they're next.

The least glamorous graphic on the screen — a tiny scoreboard nobody watches on purpose — is the one witness every broadcaster in the world agrees to keep rolling at all times, which makes it the most trustworthy thing in the picture.

Session cost: $0.00. Project total: about $4.15 of a $9 budget. The benchmark is now four matches, three surfaces, both tours — and every point ID in it comes from the one witness every broadcaster agrees to keep on screen.

Before spending another dollar chasing a tennis ball across the screen, I stopped and asked a simpler question: has somebody already solved this?

Here's where things stood. The pipeline had just been graded for the first time against a real match a human had already hand-charted — an out-of-sample test, rules frozen before anyone saw the answers, on a night match between Nadal and Shapovalov. That test caught a mirror-image bug — forehand and backhand swapped, because the software assumed a right-handed player against two lefties — fixed with a tennis fact, not new vision code. A second test was already promised: a sunlit day match, Federer against Haase. Every bit of ball-tracking so far had been done by SAM 3, a general-purpose "find that object" AI that needed a box drawn around the ball by hand, and that charged a small fee per clip.

The specialist and the generalist​

Picture two people you could hire to watch the ball for you. One is a touring pro: capable of almost anything, but before every point you have to walk over and tell them exactly what to watch. The other has had one job their whole career — they've stood on the same line, watching where the ball lands relative to it, thousands of times a day — and never once needed a pep talk first.

SAM 3 is the touring pro. WASB, the program introduced in this post, is the line judge: about the size of a phone photo, and it has only ever looked at tennis balls. Show it a photograph and it lights up how likely the ball is to be at each spot on screen, the way a heat map does, and picks the brightest spot, fresh, thirty times a second, no memory of the last one. That's a tiny specialist versus a huge, promptable generalist — more at Court Vision in Plain English — and it's what this post turns on. One side effect matters later: with no memory, WASB can't lose the ball the way SAM can. SAM carries the ball forward frame to frame — tracking, in the technical sense — and a tracker that loses its grip stays lost. WASB never had a grip to lose.

Somebody already solved this​

An evening of careful research, checking every claim against more than one source, settled two things. First: finding the ball in a tennis broadcast is already a solved, free, open problem — a specialist almost exactly like WASB is out there, pretrained and downloadable today. Our own SAM-based tracking had been reinventing that wheel, worse, for a small fee on every clip.

Second, and this matters more: every system like that stops before the notation. None of them read a broadcast and write out who hit what, where, in the shorthand a human charter would recognize. That gap — not the pixels, the grammar — is the real target here. Somebody wants it filled, too: the volunteer project supplying our human answer keys runs today on just 32 unpaid, part-time people, and its founder has publicly asked for more help.

One more gap we could close ourselves: nobody had published a head-to-head test of a generalist against a specialist on a ball this small and fast. So we ran one.

What happened when we swapped them in​

We wired the specialist's ready-made weights into the pipeline and left the notation-writing software untouched, so any difference in the output could only be the tracker's doing. It won almost everywhere it mattered — numbers below. It also did something SAM structurally couldn't: thirteen night-match clips never got a usable starting box for SAM, so SAM tracked zero frames of them. WASB needs no starting box at all — it tracked all thirteen anyway, 73–97% of frames each, no clicking, no cost.

Then something broke.

The bug that was secretly the whistle​

On the night match, the number this project cares about most — shots counted in a point, checked against the human's count — got worse with the better tracker. Before the swap it had seven of eleven points right; after, only five. Every miss counted too many shots, never too few.

I went back to the photographs. Match point: the ball resting at the frame's bottom edge, the crowd already on its feet, mid-standing-ovation. A dead ball bouncing quietly in the corner while both players stood around between points. A ball drifting through the crowd above the back fence, long after the point ended. In every case, the point was over. The ball-finder was not.

Here's the mechanism. SAM tracks — it locks onto one ball and remembers it. When a rally ended and the ball sailed out of frame or vanished into the crowd, SAM's memory broke: its trail of dots died, right there, at the end of the point. Nobody designed that as a signal, but the software downstream — the part that turns ball positions into a chart — had quietly been leaning on it anyway, never told in so many words when a point was over; it had just been reading SAM's dying trail as the whistle. WASB never gets lost, because it never had hold of the ball to begin with — it just keeps finding it, dead ball and standing crowd and all, and the chart-writer kept faithfully logging "shots" out of a point already finished. A bug had secretly been doing somebody else's job.

The first fix I tried didn't work: a ball still in play after a real hit moves fast, a dead ball bouncing after a point should be slow, and that gap looked clean — until the day match's real serves turned out just as fast as the fake bounces, and the line I'd drawn cut through real shots too.

What worked was tennis, not more tuning: a ball still part of the point never leaves the court's playable area — the same screen-to-court conversion, the homography, used elsewhere in this project. Every phantom shot sat outside that area; every real shot, across both matches, stayed inside it, serves excepted (a legal toss goes up and out on purpose). One rule, one exception, and the phantom shots disappeared. Checked again against the human chart, the night match's shot-count came back to ten of eleven — the best this project has scored — while the day match's numbers didn't move by a single shot: proof the fix cleaned up the one problem it targeted and touched nothing else.

The scoreboard​

Good, here, is the specialist matching or beating the generalist everywhere that counts, at zero cost — which is what happened, except on a handful of easy, brightly lit clips where the generalist still edged ahead by a few points.

Sometimes the scariest bugs in a pipeline aren't the ones that break something — they're the ones that had quietly been holding something else together.

Session cost: $0.00. Project total: ~$4.15 of $9 — and nothing ahead of us charges by the frame anymore.

Could software, rules locked exactly as last tuned, watch a match it had never seen and write a chart a tennis fan would call the same match?

Previously​

Two posts back, the software learned to guess who hit each shot by weighing three signals — did the ball reach that side, the serve call, where the shot landed. Last post, a human volunteer's chart of a real match — Nadal against Shapovalov, at night, in Montréal — became the project's first answer key, ending a stretch of calling its own lines: fine for practice, worthless as proof. (Why that matters: Court Vision in Plain English.) Today's rule, set before a single clip played: nothing changes before the scorecard.

Ends swap. The software had to learn that too.​

Every tennis fan carries this rule without thinking: players swap ends after the first game, then after every odd-numbered game total, and every six points in a tiebreak. Tell a fan the score is four games to three and they'll say, without looking, which end each player stands at — arithmetic, not a guess. The software had to learn that same arithmetic: no coin toss to watch, just a ball arriving from the near or far side of the screen, so it computes which end a player stands at from the score alone.

That arithmetic matters beyond serve calls. When software is wrong in a way that flips on one fact it never checked, the fault usually isn't what it's measuring — it's a label it assumed instead of computed. Call this the mirrored error: every miss points the same direction, so the tool's eyes work fine and only a naming convention flipped — a line judge with perfect eyesight calling "out" every time she means "in." You wouldn't trust her calls, but you wouldn't send her for glasses.

What carried over from the practice match​

Stabilizing the footage, building the empty-court reference photo (the median-frame trick that erases players and ball so anything left over stands out), and tracking players ran on the Montréal footage unchanged. Far-side coverage improved: 100% here versus 96% on the practice reel, because floodlights kill the shadows that had tricked the daytime footage.

A new problem showed up — the fourth time something that isn't a player has fooled the player-finder. This court's ball kids crouch at the net posts, and the code assuming "the biggest moving blob is the player" grabbed them, opening serve-toss windows above a ball kid's head, once above a chair umpire's. The fix is geometry: a server stands at the baseline, so the bootstrap checks each candidate's foot position in real court coordinates — the screen-to-court conversion the project runs on — and rejects anyone standing where a server never would.

Who served? Ask the score, not just the video​

The serve detector called a direction confidently on 11 of 25 clips here, down from 40 of 60 on the practice match — a harder test: a different match, at night.

The human chart already knows who served every point; changeover parity turns that into "which end that server stood at," once you know where the match started. Run it both ways: "Nadal started far" explains 11 of 14 confident calls, "near" explains only 3. Nadal started far — a real accuracy figure of 11 of 14, about 79%, up from 6 of 8 on the practice match, checked by eye instead of a chart.

Paying only for tracks worth paying for​

Ball tracking still runs behind a spending gate: a human eyeballs a proposed box before the project pays to send it to the paid, prompt-it-yourself tracker. Of 25 proposed boxes, 13 were approved — nine on the first look, three more after the ball-kid fix, one via a backup rule that just follows whatever's moving fastest. Twelve were sent, $0.85 total, eleven usable; the twelfth, a shaky approval, died twelve frames in. That check is 6 for 6 across both matches at picking winners. It stays.

The scorecard​

Eleven points, chart locked, graded against the human's:

That last pair is the headline — the best bad result this project has produced. Nadal and Shapovalov are both left-handed; the software still assumes every player is right-handed, a risk flagged and frozen on purpose so a lefty-heavy match would expose it. A lefty's forehand lands where a righty's backhand does, and vice versa — assume the wrong hand and the label flips, though the geometry stays right. Nine mirrored calls out of eleven means the software found the right side of the body at contact almost every time — just the wrong name for it. One line of match configuration — telling it who's left-handed — turns today's 2 of 11 into roughly 9 of 11 at the next freeze: the cheapest seven-point improvement this project will ever buy.

The rest of the misses hold up too. A tie-break rule letting a strong serve call override a shaky forehand/backhand vote had rescued one confusing point on the practice match; here it pushed two calls wrong instead. And rally length runs long because some points include a fault — a missed first serve — before the point that counts; the human chart logs both separately, but the software counted the fault as part of the rally. Starting the count from the first confident serve fixes both.

When every miss points the same direction, you haven't found a broken tool — you've found a fact about the match it never bothered to check.

The verdict​

An automated system charted eleven points of a tennis match from broadcast video and got graded against a human's chart of those points — the actual thing this project set out to test, now with a number instead of an impression. The number is bad in the most useful way a number can be: the biggest error has a direction, a known cause, and a one-line fix.

Test-set discipline held — nothing tuned mid-grade. Session cost $0.85; project total about $3.65 of the $9 budget. Next: handedness goes into the match configuration, then a control match — Federer against Haase, two right-handers — answers whether this was ever really about left-handedness at all.

Every setting this project has tuned so far was checked by the one person who tuned it. That's a chair umpire overruling his own call and then congratulating himself for getting it right — the call might be correct, but nobody should trust it just because he says so.

Previously: the plan is to feed broadcast video in and get the same shot-by-shot notation out that Match Charting Project volunteers hand-code by watching real matches (cv-01). Every dial since then — how confidently motion counts as a swing, how the serve toss gets spotted, whose shot each ball is credited to — was tuned on one practice match, Zverev–Gasquet, and checked with my own eyes (cv-09). So was the homography — the rule, built from the court's painted lines, that converts a screen position into a court position (cv-03).

Somebody else has to hold the answer key​

A line judge who grades his own overrules isn't cheating, exactly — he's just human, and human eyes trust their own first read. Tennis's fix is a second set of eyes that never made the call. This project needs the same fix: an answer key built by someone who never saw its output. This devlog calls that ground truth (more: Court Vision in Plain English) — and Jeff Sackmann's Match Charting Project supplies it: volunteers who hand-code real matches, shot by shot, in the exact notation this project is trying to produce.

There's a second half to the rule, the half that makes a test count instead of a rehearsal: the moment a match gets used to fix a bug, it becomes practice, not proof. So every setting gets locked first — written down, untouched — before the software meets a match it never influenced. Freeze first, grade second: that's a benchmark.

Finding a stranger's homework​

The Match Charting Project has hand-charted more than 7,500 men's matches; the practice match isn't one of them. A match from the same tournament, court, and broadcast package is: Canada Masters 2017, Nadal vs. Shapovalov, round of 16 — 225 points, charted point by point, at night under floodlights, both players left-handed. Neither fact was tested on the practice match: its shot calls assume a right-hander, its motion settings were tuned in daylight. Same court and graphics package, so any failure here traces to what actually changed.

Held in reserve: Federer vs. Haase, the semifinal — day, both right-handed, the closest match here to practice conditions. If the software struggles there too, lefties and floodlights aren't why.

The test's rules were locked before any of it ran: no setting changes once grading starts (one part stayed adjustable — appendix). For every clip that lines up with a human record, the check compares who served, where the serve landed, how long the rally ran, and how often the shot-by-shot calls match — counting only what the software committed to, not its shrugs. That 225-point answer key is public, free with attribution, and cost nothing to get.

Matching clips to the human chart​

Each clip first has to be found, then matched to a point. Finding them gave the practice match the most trouble; here it barely put up a fight — point the software at the file paths and the new camera math, and 25 clips of real play fell out, no setting touched. Matching them means reading a score bug — the little scoreboard graphic in the corner, frozen pixel-for-pixel during a point, changing only when the point ends (more: Court Vision in Plain English) — and finding the line in the human's chart with that same score.

One convention nearly wrecked it: the human record lists the server's score first; this match's bug always lists Nadal's first, serving or not. Every Shapovalov service game disagreed on which number came first, costing six wrongly-matched points before I caught the pattern. Before the fix, 17 of 25 clips lined up; after it, 23 of 25 did.

The remaining two aren't errors, they're genuine ties: a 40–Advantage score that happened twice in the same game, and a deuce from a game that reached deuce ten times — indistinguishable by score alone. The process even caught a mistake of mine: I'd misread one clip's serve indicator by eye, and the human chart showed Shapovalov had actually served.

The camera math doesn't carry over​

The camera math built on the practice match missed by about 20 pixels on this new footage — different filming day, camera remounted. Refitting it took three failed attempts before a fourth worked.

Attempt one reused the first homography experiment's approach, unchanged. Under floodlights, the paint boundary where the court's blue surface meets its green surround reflects light almost exactly like a real white line. The fit grabbed that paint edge and drew a court 15 to 20 pixels too big.

Attempt two made the software pickier about what counts as a white line — which killed the paint problem, and the far baseline and both sidelines along with it. Not enough real lines left to build a court from.

Attempt three searched narrow bands near where the practice match's lines had been. It failed more quietly: one band locked onto ground paint 70 pixels off; another mixed in doubles pixels while hunting for a singles line. The fit then reported a great accuracy number for itself — but that number only measures agreement with the pixels the fit chose to trust, and some of those were the wrong paint. Drawn onto the actual frame and eyeballed, the center line was obviously off, by 32 pixels: ten times what the fit had reported.

What worked: a human — me — zoomed the frame in six times and picked the court's four corners by eye, once. The same kind of one-time nudge the very first tracking experiment needed, when someone had to draw a box around the ball before it would track (cv-02). Even that failed twice at first: the corners read near the baseline were the singles corners, not the doubles corners the court is actually measured to, further out and blurred into paint at the frame's edge. Two pieces of hand-worked geometry confirmed which was which. The final fit mixed far doubles corners with near singles corners, holding attempt two's lines back as an independent check.

Where things landed​

This is the third time this project has hit the same wall, so it's a standing rule now: a number can vouch for a wrong answer, but a drawn line on the actual picture can't lie the same way. Every camera-math fit gets rendered onto the video and eyeballed before it's trusted — which is exactly what the two colored outlines in the clip up top are for.

Twenty-five clips found, twenty-three tied to a line from a human's chart, a court that measures accurately, every setting still frozen. Cost for both sessions: $0.00 — public data and an afternoon staring at pixels. Next: turn this into an actual scorecard, machine against human, point by point.

Freeze a rally at the instant of contact, and sometimes you genuinely can't tell who just hit the ball — it's caught in the air, equally far from both players.

This project is teaching software to watch broadcast tennis and write out the same shot-by-shot notation a human volunteer hand-writes for the Match Charting Project (cv-01). The last entry solved the pipeline's final manual step — the serve toss rises alone above the server's head, so the software now finds and boxes the game ball itself — and produced the first complete points, start to finish. But the column recording who struck each shot broke tennis's own rule that hitters alternate, whenever the ball was caught mid-air at contact, equally far from both players.

A vote, not a guess​

Tennis already has a model for resolving exactly this kind of disagreement: a chair umpire weighing a disputed call. No single witness gets trusted blindly — an electronic call, a line judge's shout, the mark on the court can each say something different — and the umpire decides which to believe, not just which spoke first.

That's the idea that replaced "whoever's closest." It leans on the sport's strongest fact: hitters alternate. A rally has exactly two possible stories — near player struck first, or far player did — so instead of deciding shot by shot, the software decides once per rally, scoring both stories against independent evidence and keeping whichever one wins. That's a weighted vote: several independent signals, each trusted by a different amount, added together into one decision.

The alternating pattern itself is allowed to bend in exactly one place: if the ball vanishes from the tracker for a stretch of frames, a hit could be hiding inside that gap, undetected — and only there can the near-far-near-far pattern skip a beat.

Proximity fails, twice​

The first fix tried was the obvious one: whoever's closest to the ball at contact hit it. It failed immediately — right at the instant a hit is detected, the ball is often out in open court, closer to neither player than the other.

The next idea reached for physics instead of position: after a real hit, the ball should move away from whoever struck it. Converted into court coordinates, that direction came back backwards after hits from both ends. The culprit is the homography — the recipe that turns a spot on the TV screen into a spot on the court diagram, more on it in Court Vision in Plain English — which only tells the truth for things touching the ground. It's been lying about anything airborne since the very first post that built it; bounces happen on the ground, so they were always honest, and nobody had noticed the lie until this rule leaned on it.

The evidence, weighted​

Three witnesses get called per rally, in order of trust. Touch counts most: did the ball reach exactly one player's own patch of court during the contact window? The serve call counts less — a starting hunch, not gospel. Which half the ball landed in counts least: a bounce detector — the "speed suddenly collapses" signature from Court Vision in Plain English — only watches the far half of the court right now, so it can vote "the near player hit it" but not the other way round yet. Half a witness, but an honest one. Every vote that loses stays on the record instead of getting erased — a built-in honesty check, so a witness wrong more often than it should be can't quietly hide its bad calls.

Making the serve call outvotable paid off the first time it mattered. In one point, the serve detector was confident the point started at the near end; touch and landing evidence together pointed far; the vote won. A frame-by-frame check of the footage sided with the vote.

A letter only counts if the ball got there​

Once a rally's pattern is settled, proximity becomes trustworthy again for a narrower job: pinning down the exact contact frame, now that it's known which player to look near. But the stroke-type letter — forehand or backhand — only gets written down if the ball demonstrably reached that player's own patch of court, not just "somewhere nearby." That patch scales with how big the player looks on screen: the far player is only about 55 pixels tall, so one fixed-size box for both would reject nearly every far-end letter before it had a chance.

Tracking boxes also go rogue at the worst moments — one wandered onto a spectator standing by a Perrier cooler, another landed on a court shadow instead of a person. That's the box that drifted off-court in the clip up top. Whenever that happens, the letter stays a question mark — the same "declined to guess" mark explained in the notation legend — rather than a wrong one.

Read with that legend, the clip's string is: a serve down the T (s6); a forehand toward one side (f1); a forehand where the stroke is certain but the side isn't (f?); a backhand toward the other side (b3); a backhand with the side unresolved (b?); a shot nobody's sure who hit, though the ball's own path still says the other side (?3); a forehand back up the middle (f2); and an ending not written yet, so it's ? too. Spot-checked against video, the committed letters held up.

Winning the reel before spending a cent​

Before any paid tracking ran, the plan got a free review: every candidate starting box went into one grid of still images, eyeballed by hand. That alone killed most of the backup ball-finder's picks — the method that kicks in with no confident serve toss, chasing "the fastest small thing moving," fooled by lens glare, shadow edges, court-line junctions, the net band, a FedEx cooler, even a ballkid. Of 29 backup candidates reviewed, only 3 survived. Three points moving in a straight line isn't proof of a flying ball, especially mid-camera-pan — a background object can fake that chain too.

The same pass fixed the main method for free. The search box for the toss above a server's head used to be a fixed size, so on far-end serves it was searching well above a player barely 55 pixels tall — effectively searching the crowd. Scaling that box to the player's own size, plus a filter dropping racket-shaped blobs, recovered seven far-end tosses the old search had thrown into the stands.

23 clips went to the paid tracker; 18 came back usable. Five were lost outright — SAM, the promptable tracker from Court Vision in Plain English, locking onto a stuck box instead of the ball, including every borderline case that had looked "good enough" on the free review. The eyeball review, not any automatic score, was the real ground truth.

The scoreboard​

Both serve codes finally show up correctly on far-end serves — six of them — s5 (to the body) and s6 (down the T), the zone codes that had existed in the code for a while but were never switched on; serves had been getting a cruder guess instead, the same rough "which third of the court" scale every other shot uses. One string came back fully committed: s5f3b3? — a body serve, a forehand toward one side, a backhand toward the same side, and the same unwritten ending every string gets right now.

Put it the way a tennis fan would: don't trust whoever's standing closest, trust the best evidence — and let it overrule your own serve call if it has to.

point_36:  s6f1f?b3b??3f2?   (loop v2 demo clip — far box drifts onto a spectator)
point_24:  s5f3b3?           (complete string, spot-checked)

Session cost: about $1.50. Project total: about $2.80 of the $9 budget — this low because the free eyeball review caught problems before they became paid API calls. The toss bootstrap carried this session; the backup ball-finder still needs a rethink, not a tune-up.

Every rally tracked so far needed a person to draw a box around the ball by hand first. Fine once. Absurd sixty times. The fix: a fact about tennis itself. Every point begins the same way, and the software can now spot that beginning on its own.

Two posts back, the first experiment taught a promptable tracking AI called SAM 3 to hold a tennis ball through a rally — but only after a human circled that exact ball with a box, by hand: it could follow once pointed at, never find one alone. Last time out ended on a debt: its notation string was a rally's middle, no beginning, because nothing yet knew where a point starts.

Every point starts the same way​

No rally begins in the middle. Every point opens with one unmistakable action: a player stands still, tosses one ball straight up, and swings. Nothing else moves like that — the receiver stands still, every spare ball sits still in a ball kid's hands, and the one thing rising alone is the game ball.

The bootstrap: instead of a person pointing at the ball every clip, the pipeline uses something tennis always does anyway — the toss — to point at the ball for itself. It already compares each frame to a photo of the empty court and flags what's different, the same trick that finds the players for free. The only new blob near a still server, right before contact, is the rising toss — the software boxes it itself (finding, not yet following), then hands the box to the tracking software. Detect once, then track. The click is gone.

The impostors, once more​

The pipeline has been fooled before, by ball kids holding spare balls and a line judge who erased himself standing still. This time it was broadcast graphics. Without a confident serve, the software falls back to chasing the fastest small mover on court — and its first two guesses were the scoreboard graphic (more on that box later) flipping on every score change, and a small net sign vibrating in the breeze.

The fix: a ball in flight keeps one direction and one rough speed, frame after frame — flying ballistically. A flickering graphic doesn't fly anywhere, so requiring three straight consistent frames knocked both out. Two clips still beat it, fooled by lens glare off the court's own lines during a camera pan — unsolved, noted rather than hidden.

The dribble charts itself​

Six clips ran through the paid tracking service for 55 cents total, less than a candy bar. Those tracks met the hit-and-bounce reader on a complete point for the first time, and found something no mid-rally clip could show: servers dribble before the toss. Bouncing to yourself is a real hit and a real bounce, over and over, so the detector — correctly, by its own rules — charted the warm-up as rally. Fix: the software already knows which frame the point starts on. Discard anything before it.

The first point, start to finish​

Two clips produced a full serve-to-final-shot line with nobody typing mid-point. One reads, shot by shot: serve toward the middle (s2), a backhand the software couldn't place (b?), backhand wide (b1), forehand same side (f1), forehand back to the middle (f2), backhand wide the other way (b3), two more backhands, and an ending it couldn't call — eight shots, serve confirmed right. The other was two shots and out: an unclear serve, one unplaced forehand return, an ending it couldn't call.

Checking both by eye: serve call right, shot count rally-shaped — but the column saying who hit each shot broke the serve-return pattern whenever the guess for contact caught the ball mid-air, equally far from both players. Nearest-player-to-the-ball means nothing once the ball is closest to nobody. The letters aren't trustworthy yet; the shape under them is.

Say it the way a tennis fan would: you don't need to point at the ball if you already know every point starts with a toss.

point_16:  s2b?b1f1f2b3b?b3?     (serve ✓ — 8 shots, rally-shaped)
point_31:  s?f??                 (short point: serve, return, over)

This session's six clips cost 55 cents; project total is about $1.30 of a $9 budget. 48 clips stay untouched on purpose — every fix still ahead is free to try on the six already paid for.

The point string that opened this stretch read ??2?1?1?2?2?2???, almost all question marks, no s anywhere: a rally with a middle and no beginning. It now starts with an s.

Sixty broadcast clips are sitting there ready to be charted, and every one starts with the same problem: telling an actual tennis player apart from everyone else standing near a court who isn't one.

Previously: the last post sliced a highlights reel into sixty complete-point clips using screen-to-court math as a camera fingerprint. That math, the homography — matching the court's painted lines to its real dimensions to turn screen position into court position, from Pixels to Meters — carries a free trick: median-blend many frames of the same court and the players, who never stand still for long, vanish into a photo of an empty court. Two posts later, that trick already found both players in one rally well enough to call every forehand and backhand. Doing it across sixty new clips would otherwise mean paying SAM 3 — the pay-per-prompt "find that object" AI that tracked a ball for about sixty cents in the second post — roughly $0.15 a clip, $9 for the batch, every rerun.

Comparing two photos of the court​

This session tried to cash in that free trick everywhere: build the empty-court photo per clip, then flag whatever pixel changed as a player. One catch — both photos need the same camera position. A broadcast camera looks fixed, but any camera operator worth their job nudges the shot to keep the rally centered, the way your own eyes would tracking the ball from courtside. A pixel-by-pixel comparison notices every nudge: the baseline in one photo no longer sits on the baseline in the other. It's like tracing a picture on an Etch A Sketch, then shaking the toy before checking your work — nothing lines up.

The fix: stabilization, nudging every frame back into line with the others before comparing anything, so the same pixel always means the same spot on the court. More on why the empty-court trick needs a still camera: Court Vision in Plain English.

The camera that wasn't sitting still​

That math, first fit two posts back on a sixteen-second rally, found almost no camera movement — so the code kept one screen-to-court conversion for the whole clip. A full point can run a minute or more, and these sixty clips run that long. Aligning every frame of a clip to its own first frame and measuring the gap told a different story:

The broadcast camera, measured, over one point: a sharp jump — as much as 32 pixels — mid-point, on top of slow drift throughout.

The camera operator was following the play — invisible to a viewer, fatal to the comparison. It blurs the empty-court photo, and near the far baseline, thirty-some pixels of drift is worth meters of court-position error. Every frame now gets stabilized first.

The far player who turned into a ghost​

With frames stabilized, the near player — big, always moving — showed up in essentially 100% of frames; the far player, barely 35%. The empty-court photo had swallowed him.

He's small, and barely moves between points, so a photo built from one clip bakes in a faint standing copy of him — the same trick that erased a stationary line judge two posts ago.

The fix: build the photo from five clips instead of one — this clip plus its four nearest neighbors, aligned and blended by taking the middle value at every pixel again. Players stand somewhere different every point, so five ghostly copies never land on the same pixel twice, and the blend votes them all out. Far-player coverage: 35% to 96%. Near player: still ~100%. Cost: nothing.

Everyone near the court looks like a player​

The first serve-detector run flagged several clips with the far player standing 13.5 meters from the court's center, on a court only 10.97 meters wide. Nobody serves from outside their own court.

Those weren't players. They were ballkids, crouched past the doubles sidelines inside the patch of court scanned for the far player, shifting weight just enough to out-move a motionless server. Third case of mistaken identity: the very first session swept spare-ball-holding ballkids into "the tennis ball," and two posts ago a stationary line judge got swept into "the empty court." To a pixel-by-pixel comparison, anyone standing near a court looks like they belong on it. Fix: the far-side scan region is now cut tight to the court; the near side stays generous.

Guessing who served, from posture alone​

With players tracked, the serve should be readable from posture: the server hugs the center mark, the returner waits wide — which side of the mark gives away deuce or ad, for free — and the toss stretches the server's outline to full reach, so blob height should spike right at the toss.

Version one checked only distance to the center mark, and failed fast: some clips open mid-rally with no toss to find, some end on a leftover zoom fragment, and — the costly lesson — returners often stand nearly as far from the mark as a server does, so a return swing can read as a toss.

Right: a clean toss, full stretch, on the flagged frame.

Wrong: the flagged "server" is the returner, off the mark, mid-swing — every check in version one said yes.

Version two clears several checks before it commits — tracked early and reliably, right distance from center mark and baseline, a genuine height spike — or it refuses, with reasons attached, instead of guessing.

Result: 40 of 60 clips get a confident serve call; the other 20 are refused, on the record — clips that open mid-rally or rejoin mid-point, plus two long clips too thin on far-player tracking to trust, including the marathon 49-shot rally from the last post. Spot-checking a sample of the confident calls by eye: 6 of 8 named the right server, 5 of 8 found a usable toss moment — honest, not perfect. Geometry alone tops out here; a serve is, definitionally, the ball leaving the hand and crossing the net, and once the ball itself is tracked over these clips, it settles what posture can only guess at.

The scoreboard​

The takeaway​

A camera that looks perfectly still to a viewer at home can still wreck a pixel-by-pixel comparison — so before comparing two photos of the court, make sure they were taken from exactly the same seat.

What's next​

Session tally: player positions across all sixty clips, serve details on two of three, three wrong assumptions caught — for $0.00. Project total: still about seventy-five cents, all spent in the very first tracked-rally session. SAM 3, the paid tracker, has exactly one job left: finding the ball.

Next: a ball detector on these same sixty clips. Then the serve call, hits, bounces, and shot letters should click into a complete point, written in the Match Charting Project's own notation — the human-made answer key this project is chasing. The one working chart string so far, from two posts ago, was just the middle of a rally. The next one should have a beginning, a middle, and an end.

Watch a tennis broadcast and count how much of it is actually a point being played. Not much. There's a replay after nearly every big rally, a coach's-box shot, a close-up between points — actual tennis is a small slice of the screen. Today's question: can software find, on its own, only the real points inside a whole reel of broadcast footage? It can — and it turned up the best plot twist of the project so far.

The project's goal, set on day one, is software that writes the same shot-by-shot notation Match Charting Project volunteers write by hand — that first post already flagged finding the right clips as its own job, then four posts put it off. An earlier session built the conversion from screen position to court position, matched to the court's own painted lines. The most recent session called every forehand and backhand across a whole rally — but that rally was one 16-second clip that starts and ends mid-point, with no serve, ending, or score anywhere in it.

That clip's match — Zverev versus Gasquet, Montreal 2017 — also has a 24-minute extended highlights reel: same broadcast, same camera, dozens of complete points, cut together with close-ups, crowd shots, and slow-motion replays. So today's job isn't a tennis question. It's an editing-room one: out of the reel's 36,234 individual pictures, which show the real camera, and which are decoration?

The idea that worked: reusing a conversion as a fingerprint​

A tennis broadcast has one main camera — high, behind the baseline, showing the whole court — the only camera a Match Charting Project volunteer could chart from. Everything else the director cuts to is a different camera, and a highlights reel is built almost entirely out of those cutaways.

The first idea, tried and broken below, was to spot the difference by color: the court is blue, the ground around it is green. What actually worked reused something already built: an earlier session's conversion from screen position to court position, the homography, fit once by matching the court's painted lines to its real dimensions (more on it here). That conversion doubles as a camera fingerprint — a test for whether a frame was shot from that one calibrated camera position. Project the true shape of a tennis court through it onto a new frame: on the main camera, the lines land exactly on the paint, because the conversion was built for that camera's position. On any other camera they miss — the projected court floats off into empty space instead of matching anything real.

Checking a frame becomes checking a few dozen spots: does court-colored paint sit where the projection says it should, and does the strip just outside the doubles lines correctly not have court paint in it? Pass both, it's the main camera. Fail either, it's a cutaway — including an actual replay, which would otherwise chart the same point twice.

Attempt one: paint colors, and why noon breaks them​

The first classifier measured blue and green in each frame against a threshold, tuned on frames from late in the match. It looked great — every accepted frame really was the main camera.

Then came the check nobody skips: looking at what it rejected. Six frames were obviously real points. All from the first set. All in blazing daylight.

The blue region found in a daytime frame: the court and the shaded seating bowl behind it, fused into one shape covering half the picture.

Two things were going wrong, both about light. In daylight, the shaded seats read as the same blue as the court and merge into one giant shape. At the same time, the sunlit ground around the court washes out too pale to register as green. The match starts at noon in full sun and ends at night under floodlights, and a color rule tuned for one end breaks at the other.

What actually worked, in numbers​

The projected court outline, laid onto two frames of the new reel — daytime and evening. Pixel-perfect on both: same broadcast, same camera.

Fifty spots that had to be court-blue, seventeen just outside the doubles lines that had to not be, checked frame by frame across the whole reel. Smooth out single-frame flickers, keep only stretches of three seconds or longer, and the reel breaks into 60 segments, 725 seconds of chartable tennis — about twelve minutes out of twenty-four.

The first twenty segment openings. The service toss shows up again and again — highlight editors cut in just before the serve, so it arrives free with every cut.

All sixty openings got checked by eye: every one is a real broadcast view, every one runs to a dead ball. Broadcast editors, it turns out, already hand this project exactly the unit it needs: complete points.

Auditing the gaps, and one honest miss​

Every long stretch between segments got checked too, turning up twenty genuine cutaways — including sideline and net-camera replays, failing the same way.

One gap was a real loss.

Early in set one the broadcast used a tighter zoom. The court still read correctly inside, but the bigger court filling the frame swallowed the outside-the-lines check. 131 seconds of real tennis, rejected.

The "camera never moves" assumption broke for those two minutes — a known hole, on the record. The fix, if needed: re-fit the conversion per stretch instead of once for the whole reel, which the earlier conversion work can already do automatically.

Twenty-four minutes of reel: 60 green segments found, one orange confession.

The twist buried in segment 51​

All sixty candidate clips went to disk. Segment 51 stood out immediately: 59.6 seconds of unbroken main-camera footage, by far the longest of the sixty.

That's the point this match is famous for — a 49-shot rally, the one that shows up in YouTube video titles. The very first clip this project was built on turns out to be a 16-second slice from the middle of that same rally. The segment finder, which knows nothing about tennis and only checks colors and lines, found the entire point on its own — serve included.

Every session so far has been charting an excerpt of a much longer story. It just got handed the whole thing.

The scoreboard​

The takeaway​

A tennis broadcast spends most of its time showing you everything except the tennis, so before software can chart a single point, it first has to find the one camera actually pointed at it.

This session cost nothing — no AI calls, just color math and pixel checks. Project total: still about seventy-five cents. One thing to fix before next time: this reel plays at 25 pictures a second, while the hit-and-bounce detector was built on a clip that ran at 30, and its speed cutoffs quietly assume that number.

Every blank left in the chart — the serve, the last shot's direction, how the point ends — now lives inside these sixty clips, with a 49-shot monster in the middle waiting to stress-test everything downstream. Next: point anatomy — finding the serve inside a segment, and the ending after the last bounce.

Call a shot forehand or backhand and you're answering a geometry question: which side of the hitter's body did the ball come off? You can't answer that without knowing where the player was standing — and until now, this project had tracked the ball but never found the players. Too much for one sitting: session one of several, starting with finding the players.

Here's what came before. This project's whole bet: feed it broadcast tennis video, get back the exact notation Match Charting Project volunteers hand-write for real matches — the kickoff post has the pitch. Session one tracked the game ball through a rally for about sixty cents, once shown a box around it. Next, the homography — read off the court's painted lines — turns a TV-picture spot into a court-diagram spot, except anything airborne, which converts to its shadow's position instead. Then hit-and-bounce detection read events out of kinks in the ball's path — a hit reverses it, a bounce doesn't, the court doesn't return serve — producing the first rough chart: ??2?1?1?2?2?2???.

Spot the player by spotting what never changes​

Here's the trick, older than computer vision — it's just averaging. Stack dozens of photos of the same stretch of court, one per frame. Add up how often each spot is occupied: spots that barely change are the court itself, net and lines and grass; spots that keep changing are something moving through the scene.

Treat the barely-changing version as a stand-in photo of an empty court, compare every real frame to it, and whatever's different is a player. That's background subtraction. (More in the primer.) This project already ran the same trick once, to erase the players and get a clean court photo for measuring its lines — today, in reverse.

The catch: the trick only knows "moves" or "doesn't move." Stand still long enough on camera and you get folded into "the court" — a line judge at parade rest for a whole rally erases himself as completely as the baseline.

Two boxes, one blob​

The plan: draw a box around each player in one frame and let SAM 3 — the promptable "find that object" AI from the primer — track each box through the clip, same trick that caught the game ball in session one.

The two boxes, drawn by eye and checked before spending real money. The line judge, in red, stands just outside the far box — deliberately left out.

Two boxes went in. One came back — a single rectangle per frame, the union of both players, stretched wide enough to swallow the line judge behind Gasquet too. Same lesson as session one: "tennis ball" once matched every ball; now two boxes merged into one.

Useless boxes, but the response also carries the raw shape data behind them — an outline of what the AI thinks belongs to something. Split it apart wherever it stops touching itself, and every frame resolves into three shapes: Zverev, Gasquet, the line judge.

Two of those are players. Finding the third is background subtraction again: across the rally, add up how often each spot is covered by a shape, as a percentage. Anything covered more than 85% of the time belongs to someone who never left — the judge, at parade rest throughout. Subtract it, and two players are left.

The one figure the AI outlined that never moved. Noticing that is the entire trick for telling a line judge from a player.

He erased himself. Nobody taught the software what a line judge looks like. The only rule: if it doesn't move, it isn't playing.

Both players, the whole rally, no gaps​

What's left splits into near player and far player, by which half of the frame each shape's feet land in.

Both players, every frame of the rally. The two short flat patches around frames 390 and 425 are the tracker briefly wandering onto another far-side figure, both after the rally had ended, but on the record.

Result: both players tracked across all 480 frames, zero gaps. The ball itself, tracked in session one, only managed 470 of 480, with ten dropouts — easier for this AI than a small, fast ball.

Small twist on an old caveat: the homography only tells the truth for things touching the ground — an airborne ball converts to its shadow's position, not its own. Feet are always on the ground, so a player's position comes out exact, every frame. The most trustworthy dots in this project belong to the players now, not the ball.

Who hit that?​

With both players placed, deciding who struck a ball has three checks: which player was closest at the hit; which end of the court it was struck from, the same read used last session for hits and bounces; and the simple fact that a rally alternates — whoever didn't hit last, hits now. All three agreed, on all seven hits — three signals nobody tuned for this question, landing on the same answer anyway.

Forehand or backhand? It's just geometry​

Once a player's position is trustworthy, calling the shot needs no more AI — it's geometry. Ball on the right of a right-hander at contact is a forehand, on the left a backhand — the same call from the couch.

One wrinkle: the camera-facing player has left and right flipped, so his measurement gets mirrored. One liability: handedness is assumed, not detected — a left-handed player would get every call flipped, a debt the series collects later.

The verdict: forehand, forehand, forehand, backhand, forehand, forehand, backhand — seven shots, seven calls, checked by pulling the video frame for every swing, the eyes-over-math discipline that saved last session.

Shot seven, the rally's last swing: Gasquet's one-handed backhand, racket coiled high before he sweeps through it. If you're going to eyeball-check a backhand, make it his.

Seven for seven — but one deserves an asterisk. On shot three, the ball crossed so close to Gasquet's body that the measurement came out just 1.5 pixels to the forehand side — a coin-flip margin. The video settles it: a low pickup off his right shoe, unmistakably a forehand. Right, but only because the coin landed the correct way up.

The chart gets its letters​

The two versions of this rally's chart, side by side:

before:  ??2?1?1?2?2?2???
after:   ?f2f1f1b2f2f2b??

Read the finished string left to right:

• The first ? is the serve — unknown, since this clip never shows one.
• f2 f1 f1 b2 f2 f2 b are the seven shots, in order: forehand, forehand, forehand, backhand, forehand, forehand, backhand — this session's addition. The digits (settled last session) mark which third of the court each shot crossed into. Shot seven has no digit — it ends the point, coded differently.
• The last two ? marks are where the ball landed and how the point ended — also missing from this clip.

The chart also gained something the earlier version couldn't show: where each player was standing when they hit.

Seven arrows — who hit it, which wing, from where, to where. Dashed arrows land on the far side, still accurate only to within a meter or so.

Zverev played right around his baseline; Gasquet, more than three meters behind his — court positioning anyone who's watched him play would recognize.

The scoreboard​

The takeaway​

Stand still long enough on a tennis court, and even a computer starts to think you're part of the furniture.

Cost, and the next gap​

Session cost: $0.15 — one API call, everything after it just arithmetic on numbers already in hand. Running total: about seventy-five cents.

The remaining question marks didn't move — the serve, where the ball landed, how the point ended, all still blank, since a mid-rally highlight can't show a point starting or ending. Next session: better source footage — full match video, complete points, camera cuts and all.

The chart still reads ?f2f1f1b2f2f2b?? — seven question marks fewer than it started with, three still to go. Time to earn the rest of the punctuation.

The last post ended with a warning that turned into today's homework: the system can only trust the ball's screen position at the exact moments it's touching the court — everything else is guesswork. So today's question: watching nothing but the ball's path across the screen, can software find every hit, and every bounce?

The detector I built today ended up calling all thirteen of those moments correctly. It got there by being wrong twice first, and both mistakes taught me more about how a tennis ball moves than the successes did.

Previously: last time, the software learned to convert "where the ball is on your TV screen" into "where the ball is on the actual court" — the same trick a fan does by eye, guessing where a shot landed. The catch, carried straight into today: that conversion only tells the truth for things touching the ground. A ball in mid-air converts to the position of its shadow, not the ball itself. So the software needs to know exactly when the ball is on the ground before it can trust anything else it says.

Reading the rally from the shape of the path​

A ball in a rally does exactly two things, over and over, and they look nothing alike in its path.

When a player hits the ball, it reverses direction — back the way it came, sharp and immediate. When the ball bounces, nothing reverses. It keeps traveling the same way, just lower and slower. A racket sends the ball backward; the court just takes some steam off it and lets it keep going. That's the whole idea in one line: the court doesn't return serve. (More on reading events out of a flight path: Court Vision in Plain English.)

What the ball's path actually looks like​

First move: turn the tracked dots into a graph of position and speed over time, and look at it.

The ball's up-down screen position, how fast that's changing, and how fast its shadow moves lengthwise down the court — across the rally's 290 usable frames (each a single still picture — more in Pixels and frames), with 10 short gaps where the tracker briefly lost the ball.

Two traps, both from camera angle. First: a ball's up-down position on screen isn't the same as its real height — a ball at the far baseline still looks "high" simply because it's small and distant on screen — the pretty arcs you'd expect a bounce to draw aren't real; they're height and distance tangled together. Second, and stranger: the shadow's sideways speed spikes to the equivalent of 200 meters per second when the ball is high on the far side. Nothing on court moves that fast — it's the camera exaggerating airborne motion.

First attempt: eight hits, one bounce​

My first version found every cusp in the ball's up-down speed — where it stops falling — then called hit or bounce by whether direction flipped. It found nine such moments and called eight of them hits.

That's not how tennis works — almost every stroke is preceded by a bounce. Something was badly wrong, and the fix wasn't better math. It was doing what a chair umpire does with a disputed call: stop arguing and go look at the replay. I pulled the actual video frames around every flagged moment.

Frame 44, flagged as a hit. The ball is at Zverev's feet; his racket is nowhere near it. That's a bounce.

Watching those frame strips built a hand-verified answer key for the rally, and exposed two ways the detector was fooling itself.

Two failures, two fixes​

Near-side bounces looked like hits. Right after a real bounce, the shadow speed collapses into a small, noisy number whose sign flips often enough to trick the direction test. Fix: stop asking whether direction flipped; ask how fast the ball leaves instead. A real return leaves fast; a bounce's outgoing speed collapses toward zero.

Far-side bounces were invisible, not just hard. After a far bounce, the ball rises and recedes at once — both push its screen position the same way, so there's no cusp to find at all. What does exist: the same shadow-speed exaggeration from before. Airborne, the far shadow races at wildly inflated speed; the instant the ball lands, that exaggeration vanishes and the shadow slows near the truth. That collapse marks the bounce, even where nothing in the picture looks like one.

The false alarm that taught the most​

The first version of that collapse-detector immediately flagged a moment mid-rally as a bounce. The video showed the ball sailing over the net, a meter and a half up — nowhere near bouncing.

It took a minute to understand: the collapse isn't instant at ground contact — it unwinds gradually through the descent, fading as the ball loses height. An early "slowing down" is just the ball still falling. The bounce is the last collapse of a descent, not the first.

Re-checking frames after that fix surfaced one more find, and it wasn't about the ball: my own hand-built answer key had quietly stopped early, before the rally actually ended. The tracked data ran further — one more far bounce, then Gasquet's final swing, both missed on my first pass. Even a hand-checked answer key needs its own second check.

The finished detector: seven hits, six bounces, all thirteen matching the video.

The payoff​

Thirteen events found, checked against video, correctly told apart as hit or bounce — zero wrong calls. Because the screen-to-court conversion already exists, each one drops onto an actual scale drawing of the court:

Thirteen dots and X's on a court diagram. Solid yellow (near-side bounces) are trustworthy to within centimeters. Hollow dots (far-side) are only good to within a few meters — the same camera-angle problem as above.

The near-side bounces landed at 19.4, 19.6, and 20.4 meters down the court — about three and a half meters inside the baseline, exactly where pro groundstrokes land. Nobody told the software that; it just fell out of the math.

Read the sequence out loud: hit, bounce, hit, bounce, hit — a rally, described shot by shot. No shot types or directions yet, but for the first time this reads like the start of a match chart, not a vision demo.

The scoreboard​

The takeaway​

A hit throws the ball back where it came from; a bounce just lets it keep going, a little slower. That one difference — reversed or not — is enough to read a whole rally's worth of events out of a wobbly, camera-distorted flight path. The court doesn't return serve.

Running total across the first three builds — an object tracker, a screen-to-court conversion, a frame-verified event detector — is still 60 cents, all spent on the first one. The other two were just math run over data already collected.

Next: turning this sequence of events into an actual chart — splitting footage into rallies, figuring out shot direction, and writing it all out in the Match Charting Project's own notation. That's the point where this stops being a computer-vision project and starts being a tennis project.

Last time, the software could hold onto a tennis ball for an entire rally on screen. Today's question: can it tell you whether that ball landed in the court, or a foot wide of it?

Previously: the kickoff post set the goal — teach software to watch a broadcast match and produce the point-by-point notes volunteers hand-type for the Match Charting Project, a slow, human-powered dataset the project wants to speed up. Last time, an AI followed one specific ball through a whole rally for about sixty cents. What came back was a list of screen positions — pixel coordinates, the colored dots a video frame is built from (more in Court Vision in Plain English). A screen position alone can't say whether a shot landed in or well wide.

Turning a TV picture into a court diagram​

You already do this without noticing. Watch a match on TV, and when a ball lands, you can point to roughly the right spot on a paper court diagram — even though the camera sits up high behind the baseline, so the far side of the court looks small and squeezed while the near side looms huge. Your brain quietly un-squeezes it. That un-squeezing rule — turning "this spot on the TV screen" into "this spot on the court" — is what computer vision calls a homography, the screen-to-court-diagram conversion (more on it here). Software needs the recipe spelled out, and it gets one the way a chair umpire settles an argument: from the court's own painted lines. Match what the camera sees to the real dimensions — every court on earth is 10.97 by 23.77 meters — and the conversion falls out.

One catch to tuck away first: this recipe only works for things sitting on the ground. It converts the court's flat surface, not the air above it. A player's feet are on the ground, so their position converts honestly. A ball in flight isn't touching the ground, so the same recipe reports where the ball's shadow would be — not the ball. Hold onto that; it's the twist waiting at the end of this post, and it echoes through the rest of the series.

Getting an empty court for free​

Finding painted lines is easier once you clear the players off them. The trick: the broadcast camera holds still for the whole rally (checked in the appendix). Stack every frame of the clip, and for each dot on screen, take the middle value it shows across every frame — its median. A player is only ever briefly at any one dot, so the middle value votes them out of existence. The lines never move, so they survive razor sharp. It's the same free trick the Plain English primer uses to explain how the pipeline later finds players — here it just clears the stage.

A tennis court with nobody playing tennis on it — built by stacking frames that all had somebody on it.

One line of code, and both players vanish, the ball vanishes, the lines stay razor sharp. A couple of line judges, who barely move, leave faint ghosts, and the on-screen graphics stick around too — neither matters for what's next.

Finding the lines, and one impostor​

Court lines are white paint on a blue court. So: isolate the big blue rectangle first, and only look for white dots inside it. That one rule keeps the crowd's white shirts, courtside signage, and the scoreboard's text out of the search entirely; the blue court fills about a quarter of the frame.

Run a standard line-finding routine on what's left, and 48 short segments come back, covering every painted line on the court.

Every line the court owns — plus one it doesn't.

Every real line, except the search also grabs an impostor: the net cord reads as a line too. It's white, roughly horizontal, and stretches most of the way across the frame — but it isn't painted on the ground, and it isn't even straight; a net sags under its own weight. Any scheme that just grabbed the horizontal lines would trip right over it.

Four corners is enough​

The fix is almost embarrassingly simple. Group the found lines into families — side-to-side, toward-the-camera — and keep only the outermost member of each: the top and bottom side-to-side lines are the baselines, the left and right toward-the-camera lines are the outer sidelines. The net sits in the middle by definition, so it can never be an outer line; it's excluded automatically. Cross those four outer lines and you get the court's four corners. Hand those four points, plus the court's real dimensions, to a standard piece of geometry software, and out comes the homography.

Four corners is about as little as a homography can be built from. Is it any good? The honest test: three pairs of lines never went into building the conversion at all — the singles sidelines, both service lines, and the center line. Run the full court diagram back through the conversion and see where those untested lines land.

Yellow: the lines the conversion was built from. Orange: lines it had never seen before this moment — landing right on the paint anyway.

They land almost exactly on the paint, off by less than one screen dot every time. From four corners, on the first try. I had a refinement step planned in case the numbers came back rough, and deleted it.

Where does the ball actually go​

Now the payoff: take last post's ball positions — one screen dot per frame — and run every one through the new conversion, then plot them on a true-to-scale court diagram.

The same rally, seen from directly above for the first time. Color marks time.

Most of it reads like tennis: the ball crossing the net, dots clustering low near the baselines where players make contact. Some of it doesn't — a handful of points land more than 18 meters past the far baseline, well outside any real court. Not a fit error; the untested lines above just proved this conversion is accurate to a fraction of a screen dot. It's the catch from earlier, on schedule: the conversion only knows the ground, and a ball spends most of a rally in the air. What this chart draws, for every airborne moment, is the ball's shadow, not the ball. The shadow is truthful at exactly one instant — when the ball touches down — and drifts further from reality the higher it flies.

Which hands the next post its whole job: find the moments the ball touches the ground — the bounces — because those are the only moments this chart isn't lying.

The scoreboard​

The takeaway​

The ball — tennis's famous hard problem — needed a paid AI and a human's mouse click to follow. The court — the part that sounds like real computer vision — needed four corners and free arithmetic. By the end of the day, the hard problem and the easy problem had traded chairs.

This step cost nothing — no clicks, no paid API calls, just code on my own laptop. Total for the milestone: $0.00.

Next question: the ball's path on screen is smooth curves broken up by sudden kinks. Is every kink a hit or a bounce? Time to find out if that idea survives contact with real data.

Before software can chart a single stroke of tennis, it has to do the simplest thing a ball kid does without thinking: watch the right ball, the whole rally through.

Short answer: yes, it can — with a catch, and getting there took four tries and two dead ends. Here's how.

The kickoff post laid out the plan: teach software to watch broadcast video and produce the same shot-by-shot chart that Match Charting Project volunteers write by hand today — who served where, how the rally went, who won the point. This post is step one: before anything smarter can happen, the software has to prove it can keep its eyes on one ball.

Picture the court between points: the ball actually in play, plus every spare — tucked in ball kids' hands, a player's pocket, by the net post. Shout "watch the ball!" and anyone would reasonably ask which one. That's effectively the instruction the AI got on the first attempt: the words "tennis ball." It's what's called a promptable model — instead of training it on tennis ahead of time, you just tell it, in the moment, what to look for. Told to find "tennis ball," it found all of them, game ball included, and drew one box around the whole group. That's matching a description, fresh, everywhere it fits — not following one ball.

The fix: point instead of describe. Draw one box, once, around the one ball actually in play, in a single picture. That's a different kind of instruction — not "find anything that looks like this" but "follow this exact object" — and the model carries that one ball forward through the video, remembering which ball is the right one from picture to picture. That's called tracking: locking onto one specific object and following it through the clip, rather than spotting fresh matches from scratch each frame. (More on the difference, with more tennis examples: Court Vision in Plain English.) A text prompt matches every instance of a concept in the picture; a box drawn around one specific ball tracks that ball, and only that ball.

The setup​

I grabbed a 16-second broadcast rally — Zverev vs. Gasquet, in Montreal — to test that idea on real footage. Finding a usable clip was its own small project: the first highlight compilation kept cutting away from the rally to close-ups and crowd shots after a few seconds. It took two tries to find one continuous shot of a full rally from a single camera.

Attempt 1: describe the ball and the players together​

I asked the AI to find "tennis ball, tennis player." The picture looked right for the first several seconds — clean outlines around both players. Then, partway through the clip, it abruptly flipped: instead of the players, it was now outlining the entire court. The model had lost its train of thought and locked onto something else entirely. (Why, and exactly when, is in the appendix.)

Attempt 2: describe just the ball​

Dropping the players and asking only for "tennis ball" did return one box for every single picture in the clip — 480 of them, which sounded promising. Then I plotted where that box actually sat, frame by frame:

The box the AI called "the ball," over time. Long flat stretches — a ball in a live rally never sits still like that.

A ball in play is never still for long; this chart was, flat stretch after flat stretch. Drawing the box back onto the video explained why: it wasn't tracking one ball, it was covering all of them. Broadcast courts commonly have something like six balls in view at once — the game ball plus every spare a ball kid is holding — and the AI had drawn one enormous box around the entire group, roughly the size of half the court.

The actual game ball — a smudge maybe eight pixels wide, moving at 200 kilometers an hour, contributing almost nothing to a box that size.

Attempt 3: point at it instead​

New plan: skip the words, point at the object. I found the game ball by eye in one frame, drew a box around it, and let the model carry that ball forward and backward through the rest of the clip on its own. Most of the clip came back with sensible, ball-sized boxes — a real improvement. Then the very first numbers I decoded came out negative, which a box's width can never be. Turned out to be a units mix-up, not a tracking failure (full story in the appendix). Read correctly, the numbers lined up exactly on the ball I'd pointed at.

Attempt 4: the payoff​

With the numbers read correctly, the tracked path finally looked like a real rally:

The game ball's tracked position, after pointing at it in one frame and letting the model follow it both ways through the clip.

Three clean trips of the ball from baseline to baseline, with a sharp reversal in direction at each hit — exactly the shape a ball traces when it's actually being struck back and forth. Those reversals are exactly the signal a later step will use to work out precisely when each hit happened.

Two surprises. I'd only pointed at the ball in one middle frame, yet the model filled in its position both backward and forward on its own. And track death has a clean tell: once the ball leaves play, the tracked box doesn't vanish — it freezes in place and balloons to several times its normal size. A dead track looks like a frozen, oversized box, as readable a signal as a late line call. (Exactly where, and the fix, is in the appendix.)

One frame from the rally, tracked path drawn in — the box sits on the game ball, not a ball kid's spare.

The verdict​

First milestone: done, at least for the ball — which was the part I genuinely doubted going in.

Two things are still open, not quietly skipped. Players aren't tracked as cleanly as the ball yet — the same fix should work, but that's a guess, not a result. And the box is still hand-drawn: a person clicks the ball once per rally to start the pipeline. Workable, but not the automation this project needs. Both, in depth, are in the appendix below.

Tell an AI to watch "the tennis ball" on a broadcast court and it will dutifully watch every ball in the picture at once. Point at the one that's actually in play, and it will follow just that one, the whole rally through.

All in: about sixty cents, four tries, two dead ends. Next question: now that the pipeline can find the ball in a picture, can it work out where that ball sits on the court itself? A different kind of problem — less about winning an AI's cooperation, more plain geometry homework.