Within One Token of a Human

That clip up top: an ad-point rally from the Djokovic–Ruud Roland Garros final, just under six seconds of video, with the machine's chart building itself token by token above the line a human volunteer wrote for the same point. Four pieces light up green, one after another, because the machine and the human agree on every one of them: a second serve landing in the body of the box, then three backhands in a row — two to the same spot, then one that breaks the other way. Then the last ball dies in the net. The human's shorthand for that is the letter n; the machine's chart shows a different mark instead, *, so that last piece glows amber — one disagreement. Under the rule this post is about, one disagreement is still close enough to count: the point is accepted. The video's closing screen turns the whole story into four bars.

Here's the question this post is finally built to answer: if you handed the machine's chart of a point to the volunteer who charted that same point by hand, how many things would she have to fix before she'd sign her name to it?

Previously: last time (cv-13), the software learned to read the small scoreboard graphic in the corner of the screen. That graphic stays pixel-for-pixel identical for the whole point and changes only the instant a point ends, so reading it let the pipeline do two things: glue video clips back together into whole points, and look up the exact matching point in a human volunteer's hand-charted record of the same match — the ground truth this whole project has been grading itself against since cv-10. That match-up is what makes this post possible: for the first time, every point the machine drafts can be checked, one for one, against what a real person wrote down for that exact point. Before today, the software also had a specific way of finding hits and bounces — watching for the moment the ball's path bends backward on itself, the court doesn't return serve, from cv-04 — a rule that mostly worked but quietly missed shots hit far from the camera.

How do you grade a chart?

Think about how a coach would grade a rookie scout's notes on a match. The scout writes down what happened, point by point, in shorthand. The coach already has her own notes on the same match. She doesn't grade shot by shot — she reads the scout's line for a point next to her own line for that same point and asks one blunt question: how many words would I have to cross out, change, or add to turn the scout's version into mine?

That's exactly what this post's headline number measures, except the "shorthand" is real tennis charting notation and the "coach" is a volunteer from the Match Charting Project — the same ground truth this project has been grading itself against. Write a chart of one point as a list of pieces, or tokens: the serve and where it landed, then each shot's stroke and direction, then how the point ended. Do that for the machine's version and the human's version of the same point, then count how many of those pieces would need to be changed, added, or removed to turn one list into the other. That count is called token edit distance, and it's now the north-star number for the whole project — the honest answer to "would a human sign off on this."

A point counts as accepted when that count is one or less — the kind of chart a charter would wave through after fixing at most one thing. And there's a rule that keeps the machine honest: if it isn't sure and writes a "no idea" placeholder instead of guessing, that placeholder still has to be fixed later, so it costs exactly as much as getting the answer wrong. Shrugging is not a way to dodge a mistake.

Rebuilding how hits get found

Before any of that grading could happen, the event detector needed work. The old approach hunted for a kink in the ball's flight path — the moment its direction snaps backward, which is how a hit differs from a bounce — but a kink from a shot on the far side of the court, seen from a raised broadcast camera, can shrink to a single wobbling pixel that's easy to miss entirely. So the detector got rebuilt around a blunter, more reliable piece of tennis sense: a rally in progress sends the ball across the net every single shot. Instead of squinting at every wobble, the software now counts how many times the ball crosses the net and adds one for the serve — and that crude count alone, before any fine-tuning at all, already beat the entire old wobble-hunting detector at guessing how long a rally ran. The rebuilt detector went on to sharply improve how often it read stroke letters correctly, though one older test clip actually got worse in the process — a small, honestly reported step backward, not swept under the rug (exact numbers in the appendix).

The first reading: 2.2%

With the rebuilt detector in place and every point matched to its human original, the project could finally run the grading question for real — across four matches, three court surfaces, both the men's and women's tours: how many of 135 points would a volunteer charter sign off on with at most one fix?

The answer: 3 points out of 135 — 2.2%. Before the detector rebuild, using the old event logic, that number was zero. Two percent isn't a good score. But it's the first honest score — a real distance to the top of the mountain, measured in a unit a volunteer charter actually cares about, instead of a pile of separate stats that could all be quietly improving while the finished chart is still wrong.

Where the fixes were coming from

A 2.2% pass rate means the typical point needed about seven fixes, on average, to become the human's version — so the project broke that average down by fix type to see where the effort was going. Missing shots — a shot the human charted that the machine's chart skipped entirely — were the single biggest source. But close behind, and the most fixable-looking, was shot direction: which way a shot traveled, not just what stroke it was. Getting the direction digit wrong, whether alone or combined with the wrong stroke letter, accounted for 39% of every fix an editor would have to make — the largest chunk of the whole edit budget. And the direction guesses weren't just wrong often: the machine only ventured a direction guess on 70% of shots, and when it did, it was right just 48% of the time — barely above the 33% you'd expect from picking one of three directions at random.

A compass pointed the wrong way

That's a big enough number to be worth chasing, and the cause turned out to have nothing to do with cameras or vision at all — it was a definition problem. The Match Charting Project's own instructions define one shot direction as "a right-hander's forehand side" and add, in parentheses, "a lefty's backhand." Read carelessly, that sounds like direction depends on which hand hit the ball. It doesn't: the parenthetical is the tell. Direction is defined relative to a fixed side of the court — specifically, the half that's receiving the shot — and it only happens to line up with "forehand side" for one particular kind of player. The pipeline's code, though, had been computing direction using fixed court thirds, as if the same physical side of the court always meant the same direction: correct by pure coincidence for a shot going to the far end, backwards for a shot going to the near end. It was a compass mirrored for half the court the whole time. Once the definition got fixed — direction read relative to which half was receiving, instead of a fixed physical side — direction accuracy jumped from 48% to 75%, the machine started venturing a guess on 97% of shots instead of 70%, and overall acceptance nearly doubled: from 3 points out of 135 to 7, from 2.2% to 5.2%.

The scoreboard

Four matches, three surfaces, both tours — here's how many of each match's points a human charter would sign off on with at most one fix, after the compass fix went in:

Match	Points compared	Accepted (≤1 fix)
Night match	22	3
Control match	5	1
Clay — Roland Garros final	59	3
Grass — Wimbledon final	49	0
All four, combined	135	7 (5.2%)

That's up from 3 of 135 (2.2%) before the direction fix. The honest low point on the board is Wimbledon: zero points accepted, because its problem isn't a wrong direction digit — it's shots that shouldn't be in the chart at all, or shots missing from it (more in the appendix).

The takeaway a tennis fan could repeat at dinner: what matters isn't how many little stats are climbing, it's how many marks a red pen still needs — and fixing one wrong assumption about which way "direction" points nearly doubled how many points needed no more than one.

For the technical reader

Net-crossing detector rebuild. Two passes. The serve detector went ball-first — the serve is the track's first sustained net crossing, its direction names the server's end — and clay server-end accuracy went 7/58 to 48/59 (12% → 81%), grass 17/49 to 38/49, with the honest asterisk that grass refuted the ball signal outright (white ball, white lines: 26/49, a coin flip), so the grass detector kept the player-based signal and only the search window learned to slide. Then the coda pass went hunting for a per-event tell that separates a live rally from Wimbledon's dead-ball aftermath, and refuted every candidate — landing positions, walking speed, coverage, all byte-similar between the two — until the rule that held was sequence-level tennis: a live rally sends the ball across the net every shot.

That sentence turned out not to be a truncation rule; it was the detector, inverted. The old design — M2-era cusp classification (the wobble-by-wobble hit/bounce guesser from cv-04), tuned on sparse SAM tracks (the ball-tracking AI's own thin, gap-ridden paths) — inspected every wobble in the ball track and asked "hit or bounce?" Diagnosing all 57 rally-count misses (not a sample) found the far-end miss systematic: a far hit at 250+ px is a ~1 px/frame wiggle no near-tuned gate can feel, but its flight crosses the net at full projected speed. The keystone measurement, taken before any design was written: raw net-crossing count

• 1 already beat the entire old detector — rally ±1 of 36/59 vs the chart's 25/59 on clay, 36/49 vs 26/49 on grass, from just counting crossings.

So v5 stops classifying wobbles. Net crossings are the rally skeleton: they partition the point, each partition contains exactly one hit, and the cusp machinery is demoted from counting shots to locating the hit inside its slot. Constants tuned on t3 (the clay match) only, everything else scored untouched — the tuned tree went 42% → 61% on rally length while letters (aligned) tripled its base, 19/24 → 65/85. The named regression: t1 (the night match) fell 16/22 → 13/22. The skeleton needs crossings, and the night reel's thin low-recall tracks amputate rallies the old cusp counter used to pad out. A design that leans on the net pays the price wherever the track can't see the ball cross it.

Edit-distance mechanics. Tokenize both strings as [serve+zone][letter+direction]*[ending] and accept a point iff the machine's list is within one token edit of the human's — strict equality, a ? matches nothing, so refusing to commit costs exactly what being wrong costs. First reading, all four matches: 3/135, 2.2% (before v5 it was 0/135). Mean token edits per point: 7.18 across n=135, backtraced to an alignment and binned:

edit category (mean/point, n=135)      overall
deletion (shot never charted)          1.95
sub: letter AND direction wrong        1.61
sub: direction digit only              1.20
sub: ending token                      0.70
insertion (phantom shot)               0.54

Direction digits sit in two of those bins and add up to 2.81 edits per point — 39% of the entire edit budget — attempted on only 70% of shots and 48% right when attempted, barely above the 33% floor. Meanwhile the effort curve says the skeleton v5 just rebuilt is no longer the wall: structure alone (shot count + letters) would already accept 23.7% of points at the same ≤1 bar, per the original decomposition note (this figure is carried over as originally reported alongside the ≤3-edit curve below; the two are not obviously the same slice of the data, and that ambiguity is preserved rather than resolved here). The headroom table, fixing one component to ground truth at a time:

counterfactual                        accept ≤1      mean dist
baseline (v5 as charted)              3/135 ( 2.2%)  7.18
directions perfect (all shots)        9/135 ( 6.7%)  5.88
endings perfect                       7/135 ( 5.2%)  6.47
letters + directions                 21/135 (15.6%)  3.93
letters + dirs + endings             53/135 (39.3%)  3.23
all components (structure residual)  59/135 (43.7%)  2.93

No single fix rescues the metric — the mean point is wrong on several axes at once — but the compounding is steep and ordered, directions are the biggest single lever, and the tell is in the fine print: perfecting only the attempted directions is worth almost nothing. The sparsity and the accuracy are the same disease.

The mirrored compass. MCP's instructions define direction 1 as "a right-hander's forehand side (a lefty's backhand)" — the parenthetical is the tell. That sentence names a fixed side of the receiving half, mirrored by which end is receiving, not flipped by who's holding the racquet. zone() had been coding absolute court thirds: correct for a far right-handed receiver by coincidence, mirrored for every near receiver. Half of "barely above the random floor" was a sign error. Every plausible mapping went into a calibration table before any code, and the both-lefty t1 match is what adjudicated the handedness question: receiver-end mirror without a handedness flip scores 11/24 on the lefties; adding the flip drops it to 5/24 — the naive reading of the spec loses to its own parenthetical.

On top of the fixed compass, a measured signal ladder gives every shot a direction now — the workhorse is the receiver's next contact (where the ball was received is where this shot went; 77% right, 81% coverage on the dev match), backed by the near-half bounce (86%) and the v5 skeleton moonlighting as a crossing-plus-slope extrapolator. Even refusal policy was an experiment, not a principle: vetoing on disagreement kept precision flat while shedding net-right tokens, and since the metric prices a shrug the same as an error, commit-always won.

Direction attempts went 70% → 97%, accuracy when attempted 48% → 75% — held-out matches gaining right alongside the tuned one — and acceptance moved 3/135 → 7/135, with the ≤5-edit curve jumping 41.5% → 57.0%. Every other metric on all four scorecards is byte-identical, exactly the claim a direction-only change should be able to make. The meta-lesson joins the permanent file: when a component scores near the random floor, check the semantics before rebuilding the perception. A 27-point accuracy jump was sitting in a coordinate convention the whole time.

Full scorecard. Four matches, three surfaces, both tours, 168 points aligned to human charts as originally reported for this section (a different count from the n=135 used above for the token-edit decomposition — both numbers are carried over as originally stated):

metric              t1 night   t2 ctrl   t3 clay      t4 grass
server end          10/22      2/5       48/59 (81%)  37/49 (76%)
rally length ±1     13/22      5/5       36/59 (61%)  28/49 (57%)
serve zone          11/12      1/3        8/17        15/32
letters (aligned)    9/11      12/12     65/85 (76%)  15/28
ending type          9/16      2/3       19/42         9/33
acceptance ≤1 edit   3/22      1/5        3/59         0/49

acceptance at ≤ k token edits (n=135):
≤1: 2.2 → 5.2%    ≤2: 6.7 → 11.1%    ≤3: 14.8 → 23.7%    ≤5: 41.5 → 57.0%

The honest reads stay on the record: t4 (grass, Wimbledon) is 0/49 strict — its disease is structure (1.24 phantom insertions plus deletions per point), and no direction-digit fix repairs a token that shouldn't exist in the first place. t3's (clay, Roland Garros) missing shots are mostly the broadcast's fault, not the pipeline's — the RG editor cuts into rallies mid-point — and structure caps the all-components ceiling at 44%. The re-run decomposition already names the next levers in order: structure, letters, endings — one token per point, 30% right, as originally reported.

Two percent was the first honest look at how far this project has to climb. Five percent, after fixing the compass, is the second look. The real change isn't the number itself — it's that the climb now has arithmetic behind it instead of a guess.

Session cost: $0.00. Project total so far: about $4.15 of a $9 budget.