The Draft Grades Itself

That clip up top: the machine's actual draft of the Roland-Garros final, point by point — each gets a chart string, a confidence score, and a flag, HIGH rows glowing green. Three flagged points then play back with their draft on screen, from a clean ace (s4*: serve out wide, unreturned, 0.969 confidence) to a messy point full of "not sure" marks (0.933 confidence). The running tally: 59 points drafted, 16 flagged "start from this draft."

Say a machine hands a volunteer chart-corrector a stack of drafted points — how's she supposed to know, before she's watched a single one, which drafts are worth trusting and which she should chart from scratch?

Previously: cv-14 gave the project its report card — the token edit distance, how many fixes an editor needs to turn the machine's chart into a human's, a point "accepted" at one fix or fewer. cv-10 set the rule for grading any of it: score only against matches a human already charted, never a match the pipeline saw while being tuned. And since cv-05, players get found for free by background subtraction — photograph an empty court, call anything different in a later frame a person. Today tests whether that free method is good enough, then teaches the draft to flag its own reliability.

A confidence flag is a line call

Picture two line judges. The first only calls a ball "out" when it lands a clear foot past the line. She's almost never wrong, but rarely speaks, leaving a player who trusts only her guessing on everything else. The second judge calls anything that looks close — she catches nearly everything worth catching, but plenty of her calls are wrong on replay.

A confidence flag sits on that same spectrum, and two numbers say where. Coverage is how often it's willing to call HIGH at all — how often the strict judge speaks. Precision is how often a HIGH call is actually right. A flag that's always right but rarely fires isn't useful, and neither is one that fires constantly and is often wrong — you need both numbers together. (More in the primer.)

Making those numbers honest takes a third idea: calibration. Take points a human volunteer has already charted — the answer key — and check signals the pipeline keeps about its own effort: did the serve call commit cleanly? does the ball's path have gaps? did two vote-counting methods for "who hit that" agree? Fit those signals to the real outcomes and the fit becomes a small scoring formula.

The trap: a formula that peeks at a match's answers while being built will always look brilliant tested on that same match — an open-book quiz, not a test. So fit it on three matches, then grade it, cold, on the fourth it's never seen, rotating until every match gets a turn held back. Tennis translation: the system never grades its own homework — the same rule cv-10 set for the pipeline, now applied to the flag that grades its work.

Is the pricier player-finder worth $12?

Today's first question: how much of the draft's remaining mistakes trace back to the free player-finder misjudging where people stand?

Every shot call — forehand or backhand, roughly which third of the court — comes from one rule: whichever side of the player's outline the ball arrives at, that's the shot. A bad outline poisons the call, already the biggest single source of remaining edits, about 1.7 corrections per point. Sorting every scored shot by outline quality at contact:

player outline at contact	shot called right	shot called wrong	system gave up
clearly fine	70	17	11
shaky	31	18	23
missing entirely	0	0	8

Nearly half of every shot call — 45% — was riding on a shaky or missing outline; accuracy was 71% on clean outlines, 39% on shaky ones. Ceiling: about 26 more correct calls.

Free cleanup — rejecting teleporting outlines, checking court-half, filling short gaps, widening for legs-only silhouettes — bought back 3 of those 26: about an eighth of the ceiling. The rest was missing data, not noisy data: the far player usually isn't in the free tracker's output near contact. No cleanup invents a player the tracker never saw.

That earned the right to spend money: SAM 3 — the promptable AI that tracked this project's first ball, back in cv-02 — re-found both players in the Roland-Garros match only, prompted from the free tracker's own boxes, for about $12. Shot-call accuracy went from 59% to 67%, mostly outlines that had been missing — 21 calls flipped right, one point gaining seven alone.

It still didn't move the number that matters — full points a volunteer would sign off with one fix or fewer, stuck at 3 of 59. A better shot call wasn't holding those points back, and the paid tracker inherited the blind spot it was hired to fix: where the free tracker drew no usable box, an automatic prompt had nothing to point at either. Verdict: the $0 tracker stays default — what every single-piece upgrade has said since cv-14: improving one ingredient nudges the score without changing which points a human would accept.

One tool, not twenty scripts

Before any of that shipped, this project's first fourteen posts' worth of scattered experiment scripts — twenty-odd files, four near-identical chart-builders kept in sync by hand — got folded into one package: one chart-builder, one grading tool, one command-line tool, broadcaster differences moved into a settings file per match. Condition for merging: it had to reproduce every number this series has published, byte for byte, before anything new shipped on top. It did, down to 138 chart files and four scorecards.

Building a flag that isn't lying to you

With the free player-finder settled and the tool consolidated, the actual flag got built — the scoring formula above, trained on the pipeline's self-reported effort and graded the never-grades-its-own-homework way.

Two surprises turned up. Aces — serves the returner never touches — were scored as if the machine had refused every shot in the point, since there are no rally shots to refuse. Backwards: an ace is the cleanest point in the match, punished for being short. Fixed by teaching the formula that no shots after the serve means the point ended cleanly.

Stranger: on the Roland-Garros match, the broadcast editor sometimes cuts into a rally already in progress. The serve-detector, seeing the ball cross the net for the first time in the clip, calls that crossing "the serve" — even though a real serve toss lands 20+ meters past the baseline, and this one launched from inside the court. That's a clip that joined mid-rally, not a serve. Teaching the formula that one tell took held-out reliability from 65% to 84%.

One more result belongs here because it failed: a stricter flag — "sign off without watching the point at all," 85% accurate or better. Tested the same honest way, it landed at 50% precision, a coin flip, willing to call only 1.5% of points that confident. There aren't enough graded points to support it: of 135 hand-charted points, only about 1 in 9 already lands within two corrections. That tier is dead, reported dead rather than dropped quietly — a confidence system that hides its own failures isn't one anyone should trust.

The scorecard

What did the shipped flag earn, tested the honest way? Read the pooled number the way you'd read a strong first-serve percentage — not a guarantee, but a number worth building on:

match	HIGH flag right, held out	how often HIGH gets called
the night match	11 of 11 (100%)	about half the points
the control match	3 of 3 (100%)	about 3 in 5 points
the Roland-Garros match	16 of 19 (84%)	about 1 in 3 points
the Wimbledon match	11 of 11 (100%)	about 1 in 5 points
all four matches together	41 of 44 (93%)	about 1 in 3 points

HIGH means "start from this draft" — 93% of the time, held out, it needs five corrections or fewer. It doesn't mean the draft is already right: only about a quarter of HIGH points need two corrections or fewer. Missing HIGH means "expect heavy correction" — more than half of those need six or more, so treat it as a hint, not a finished chart. The Roland-Garros match stays the shakiest, not because its math is worse — its disease is missing footage, shots the editor cut before the camera ever saw them, which no signal can detect from film that doesn't exist.

Said the way a tennis fan would at dinner: a confidence flag only means something once it's been tested on points it never got to see the answers to first.

For the technical reader

Shot-call sink. Ball-x vs. box-center-x at contact, gated on the ball reaching the box; biggest post-direction edit sink, 1.71 edits/point. Audit (178 aligned letters, 4 matches, pre-fix):

outcome         sane   implaus  absent
right             70      31       0
wrong             17      18       0
refused           11      23       8
accuracy          71%     39%      --

45% of letters read off a bad box; ceiling ~+26 strict letters of 209 scored.

Free hygiene (tuned on t3, t1/t2/t4 held out): court-half plausibility, x-only teleport rejection, gap interpolation, letter gate widened to clip-typical body height. Net: 114/209 → 117/209. Three other fixes were built, measured, and removed (not detailed further here).

SAM-3 buy test. t3 = Roland Garros 2023 final (Djokovic–Ruud, clay), 59 clips; prompts auto-derived from the bgsub boxes, ~$12 all-in. Same eval, same hygiene:

t3 letters              bgsub+hygiene   SAM-3
strict positional        67/114 (59%)   76/114 (67%)

21 gains, 15 of them refusals turned right (t3_point_25 alone +7). Acceptance stays 3/59. Default stays bgsub at $0.

Consolidation. courtvision/ package: one chart assembler, one eval, one CLI; per-broadcaster differences moved into data/matches/<id>.yaml. Gate: 138 chart CSVs byte-for-byte, 4 scorecards byte-identical, acceptance 7/135 untouched. experiments/ frozen as history. One command: courtvision draft t3 → chart CSVs → confidence per point → outputs/t3/export/t3_mcp_draft.csv.

Confidence layer. Per-point signals — serve commit and stance margin, striker-chain conflicts, ball coverage and holes, refusal fractions, the direction signal-tier ladder, crossings-vs-shots consistency — feed a small logistic regression (numpy), calibrated leave-one-match-out (LOMO) against token edit distance: fit on 3 matches, score the 4th, rotate.

Aces bug: a 100%-refusal charge for zero rally letters to refuse. Mid-rally-cut signature named t3/clay in the codebase.

Failed tier target: ≤2 token edits at ≥85% precision; exact base rate 11.1% (≤2-edit) of the 135 calibration points.

LOMO (held out)   high prec (≤5 edits)   coverage   low ≤5 rate
t1 night              11/11 (100%)         50.0%        27%
t2 ctrl                 3/3 (100%)         60.0%        50%
t3 clay               16/19  (84%)         32.2%        43%
t4 grass              11/11 (100%)         22.4%        50%
pooled                41/44  (93%)         32.6%        44%

Export. One CSV, Match Charting Project points schema (match, point number, score state, server), machine string in the 1st-serve column, plus confidence, conf_p, clip name, jump-to timestamp (serve_s). Across 4 matches: 138 draft points, 46 flagged high.

Precisely stated promise: given a match with a fitted homography and transcribed score-bug alignment, a draft for every tracked point at $0 marginal cost, HIGH on ~1/3 at 93% held-out reliability. Not promised: a HIGH point is right, a LOW point is chartable without video, or anything about rallies cut before broadcast. North star: acceptance ≤1 token edit = 5.2% (7/135), ≤5-edit curve = 57.0%.

None of this moves the project's real ceiling, unchanged since cv-14: only 5.2% of drafted points — about 1 in 20 — are an exact-or-one-fix match to a human chart, and even counting anything within five corrections as close enough, that's still just 57%. The ceiling isn't stuck on any piece of the pipeline anymore — it's stuck on the answer key, only 135 hand-charted points, barely 1 in 9 of them already within two corrections. More matches are already being lined up, and the moment they land, every number here gets recalculated against a bigger, harder answer key.

Session cost: about $12 — the SAM test, under a one-time $30 limit — then $0.00 for everything built after it. Project total: about $16.