![\"Line](\"https://optic-yellow.com/assets/images/m3-camera-pan-6f1e81fe1d0e638f2621a185ad53538d.webp\")
That clip up top: the machine's new view of the players. Yellow stick\nfigures — wrists, elbows, hips, feet — drawn by a free model on real\nmatch frames, including the far player at ninety pixels tall. And one\nframe kept for honesty: the first version confidently drawing its\nskeleton on a ball kid.
\nToday the software stops seeing players as smudges and starts seeing\nswings. The question: does that finally move the forehand-or-backhand\ncall that's been stuck for two weeks?
\nPreviously. The machine finds players by comparing each frame\nagainst an empty-court photo (background\nsubtraction) — which yields a moving\nsmudge per player, no arms, no legs. The forehand/backhand letter has\nbeen read from which side of the smudge the ball arrives on, and that\nidea has formally hit its ceiling: careful\nexperiments showed no amount of\nre-anchoring the smudge changes the calls. Meanwhile this same week the\ncourt learned to fit itself and the\nmachine grew ears. The players are the\nlast part still made of fog.
\nThe upgrade is a technology every fitness app now uses: pose\nestimation — a model that marks seventeen body landmarks (wrists,\nelbows, shoulders, hips, knees, ankles) on any person in an image,\nturning a smudge into a stick figure. Instead of guessing a stroke from\nwhere a blob bulges, you can read the actual tennis: where the racquet\nwrist is, which side of the body it crossed to, how the shoulders line\nup at contact. A free, openly-licensed model does this at better than\nreal-time speed on this laptop — no cloud, no cost.
\nThe open question was never the near player, who fills a fifth of the\nscreen. It's the far player, who stands about ninety pixels tall —\nthumbnail-sized. Research on this exact problem predicted: shoulders\nand hips will hold up, wrists will get shaky. The audit stills in the\nclip say the prediction was right on both counts — the far player's\nskeleton is real and readable, and the model's own confidence in the\nfar wrists hovers around fifty-fifty, versus a solid two-thirds for\nshoulders.
\nTwo matches with human answer keys, the two where the letter call is\nbenchmarked: the Paris clay final and the Wimbledon final — grass being\nthe surface where the smudges smear worst. Every method graded on the\nsame shots against the same human charts:
\n| Surface | Smudge rule (shipped) | Skeleton rule |
|---|---|---|
| Clay | 36 of 44 — 82% | 42 of 50 — 84% |
| Grass | 16 of 30 — 53% | 23 of 35 — 66% |
On clay: a tie, two points better — exactly what the earlier\ncounterfactual experiments predicted, since on clean footage the smudge\nand the skeleton mostly agree about where a body is. On grass, where\nthe boxes fall apart: thirteen points better. The skeletons help\nmost precisely where the old method fails hardest — the same pattern as\nthe ears last post, and it's not a coincidence: every new witness earns\nits keep in the old witnesses' blind spots.
\nThe first version had a bug that one audit image exposed instantly. It\nmatched each skeleton to the player by picking the detected person\nnearest the smudge's center — and on one clay point, a rogue smudge box\n(an old known disease) sat so far from the actual player that the\nnearest human was a ball kid at the back fence. The machine\ncarefully drew a skeleton on the ball kid and graded his forehand.
\nThe fix is pure tennis sense: a candidate skeleton only counts if its\nfeet are standing in the striker's half of the court — the\nscreen-to-court conversion makes that a\none-line check. Ball kids kneel at the fences and net posts; umpires\nsit outside the lines; players play inside them. The fix costs\ncoverage — with strict standing-room rules and shaky far-player ankles,\nthe matcher now declines about half the shots rather than risk grading\nthe wrong human — and recovering that coverage by tracking skeletons\nacross frames, instead of judging each frame alone, is the obvious\nnext version. Fourth time this week that a rendered picture caught what\nthe accuracy numbers politely hid.
\nThe takeaway: given actual arms to read instead of a fog bank, the\nmachine's forehand-or-backhand call improved most exactly where it was\nworst — and the day's best evidence was one frame of a very\nwell-postured ball kid.
\nModule: experiments/pose_letters.py (report + audit stills in\noutputs/diag/; run with\nPYTHONPATH=.:experiments uv run --with rtmlib --with onnxruntime python experiments/pose_letters.py).
Stack: rtmlib (Apache-2.0) Body pipeline (person detector + RTMPose),\nONNX Runtime CPU, weights auto-downloaded; ~1 s/frame with detection on\nthis laptop. Posed only at charted contact frames (cheap gate test, not\na per-frame rollout).
Protocol: MCP truth letters on length-matched aligned clips (the eval\nconvention); all four players right-handed per match configs. Feature\nA \"wrist-cross\": sign of (right-wrist x − shoulder-midpoint x),\nmirrored for the far player (faces the camera). Feature B \"ball-side,\npose anchor\": ball x at contact vs shoulder midpoint — the re-anchoring\ncontrol. Results (same-shots blob baseline in parens): t3 — A 37/50\n(74%), B 42/50 (84%) vs blob 36/44 (82%); t4 — A 21/35 (60%), B 23/35\n(66%) vs blob 16/30 (53%). Wrist-cross underperforming ball-side is\nexpected at contact-frame-only sampling: two-handed backhands and\npreparation frames confuse a single-frame wrist test; windowed\nskeletons are roadmap #9's job (the keypoint-graph model, where\nfew-shot literature shows 2-3x gains).
Keypoint confidence (Lsho/Rsho/Lwri/Rwri means): near 0.70-0.74 /\n0.50-0.57 wrists; far 0.61-0.65 / 0.45-0.57 wrists — matching the\n\"shoulders hold, wrists shake\" prediction for ~90 px players at 720p.
v1→v1.1: nearest-blob matching graded a ball kid\n(outputs/diag/pose_blooper_ballkid.png, kept); v1.1 requires\ncandidate ankles to project (via the homography) into the striker's\ncourt half — far: court-y ∈ [−4, 11], near: [12.8, 28], x ∈ [−1.5,\n12.5] — then takes highest mean confidence, blob box demoted to\ntiebreak. Coverage cost: t3 graded shots 93 → 50 (no fallback when\nankles fail the test); temporal tracking recovers this in v2.
Hero renderer: experiments/render_pose_hero.py.
Session cost: $0.00 — openly licensed weights, laptop inference.\nProject total: ~$16.
\nNext question: in one day the machine learned to call lines by physics,\nhear impacts, fit its own court, and read swings — four new witnesses,\neach stronger where the others are weak. None of them are wired into\nthe shipped pipeline yet; they're experiments with receipts. The next\nbuild is the courtroom: the fusion decoder that hears every witness,\nweighs each one's track record, and writes the chart no single witness\ncould — with the grammar of tennis notation making illegal testimony\nimpossible to write down.
", "url": "https://optic-yellow.com/blog/stick-figures-beat-smudges", "title": "Stick Figures Beat Smudges", "summary": "The forehand-or-backhand call has been stuck for two weeks. The fix: draw a skeleton on every player and read the actual swing.", "date_modified": "2026-07-22T23:00:00.000Z", "author": { "name": "Trevor McCormick", "url": "https://github.com/trevormccormick" }, "tags": [ "Devlog", "Computer Vision" ] }, { "id": "https://optic-yellow.com/blog/the-court-that-fits-itself", "content_html": "That clip up top: eight tournament courts, two overlays each. Green is\nthe court fit a human built by clicking four corners. Orange is a free\npretrained network's fit — zero clicks. On seven courts they're a few\npixels apart. On the Roland Garros plate — the one that broke the old\nautomatic fitter — the orange lands first try. And on the night match,\nthe machine does something arguably better than fitting: it declines.
\nToday the software tries to take over the last job a human still does\nfor every single match: teaching it where the court is.
\nPreviously. Everything this machine calls — serves, bounces,\nwide-or-deep, who's standing where —\nruns through the homography, the screen-to-court-diagram conversion\n(primer) built from the court's painted lines.\nSince the very first fit, getting that\nconversion has ended one of two ways: an automatic fitter finds the\nlines, or — on stubborn footage — a human clicks the four corners by\nhand. The automatic fitter has a famous scar: at Roland Garros, faint\nbrushed-over baselines and reddish stands fooled it so badly its fit\nmissed the court by 74 pixels, and a human had to step in. Clicking\ncorners takes a minute. But a minute per match, times the thousands of\nmatches this project wants to chart, is the difference between a tool\nand a hobby.
\nThe fix wasn't invented here, and that's the point. Someone on the\ninternet already trained a small network to find tennis courts in\nbroadcast frames — not by hunting for four corners, but by recognizing\nfourteen landmarks at once: every corner, every T, every place two\npainted lines meet. It learned from 8,841 broadcast images across hard\ncourts, clay, and grass. It's free, it's small, and it runs on this\nlaptop in under a second per image.
\nFourteen landmarks instead of four is the difference between asking a\ntourist for the two street corners they remember and asking a letter\ncarrier who knows every house on the block. If a couple of landmarks\nare hidden or faint, twelve others still pin the court down — and the\nfitter can even check its own work, by fitting on some landmarks and\nmeasuring how well it predicted the ones it didn't use.
\nEach court's automatic fit, measured against the human's hand fit at\nsix landmarks (the four court corners and the two net posts):
\n| Court | Machine vs human, average gap |
|---|---|
| US Open (hard) | 2 pixels |
| Melbourne (hard, night session plate) | 3 pixels |
| Turin (indoor) | 5 pixels |
| Montreal (hard) | 7 pixels |
| Wimbledon (grass) | 8 pixels |
| Roland Garros — the 74-pixel disaster plate | 8 pixels, first try |
| Paris 2023 (clay) | 16 pixels — with a twist, below |
| Nadal–Shapovalov (night, dim) | no fit — the machine declined |
Seven of eight courts, fit to within a few pixels of a careful human,\nby a network that had never seen any of them. And the eighth result\nmight be the most important one on the board: on the dim night match,\nthe network found only 2 of its 14 landmarks — so instead of guessing\nfrom starvation-level evidence, the fitter abstained and left the\njob to a human. That's the behavior this whole project is organized\naround: an automatic call when the evidence is strong, a raised hand\nwhen it isn't — the same contract as the\nconfidence flag that grades every drafted point.
\nOne court disagreed with the human by 16 pixels, and I went in expecting\nto document the network's failure. The overlay says otherwise: on that\nclay plate, the orange lines sit squarely on the painted court, and\nthe green — the human hand fit that all the benchmarks treat as\ntruth — floats visibly inside the paint on the far side. The\n\"error\" may be the teacher's. It's not settled yet — the neutral\nreferee will be the pipeline's existing line-scorer, which measures any\nfit directly against the paint itself — but it's a healthy reminder\nthis series keeps getting taught: hand-made truth is still hand-made.
\nFirst run, every single fit came back wrong by the same enormous\namount — about 750 pixels, all eight courts. A consistent error is a\nconvention error, not a model error: the borrowed code, it turns out,\nalready doubles its coordinates internally (it assumes 720p footage),\nand my wrapper doubled them again. One rendered overlay exposed it\ninstantly; the numbers alone never would have named the cause. Third\ntime this week: eyes beat derivatives.
\nThe takeaway: the last per-match chore a human always did — teaching\nthe machine where the court is — just went automatic for seven courts\nout of eight, and on the eighth the machine did the next best thing to\nbeing right: it knew it didn't know.
\nModule: experiments/court_autofit.py (report + per-match overlays in\noutputs/diag/; run with\nPYTHONPATH=.:experiments uv run --with scipy python experiments/court_autofit.py).
Detector: yastrebksv/TennisCourtDetector — TrackNet-style heatmap net,\n640×360 input, 15 output channels (14 keypoints + center), pretrained\non 8,841 YouTube broadcast frames (hard/clay/grass), median error\n1.83 px refined / 2.83 px raw per its README. No license file — fine\nfor this research phase; the blueprint's risk register already flags\nretraining a self-labeled replacement before any commercial use (every\nhand-fit match auto-generates 14-keypoint labels through its known\nhomography). Weights: 42 MB, CPU inference ~1 s/plate.
Method: run on each match's plate_fit.png (the stabilized median\nplate the hand homography was fit on); solve via the author's\n12-configuration search (reimplemented numpy-only): each candidate\n4-point configuration fits a homography, is scored by mean prediction\nerror on the unused detected keypoints (cross-validation, not\nself-consistency), best wins. Convert reference-canvas px → meters\n(doubles corners (286,561)–(1379,2935) ↔ 10.97 × 23.77 m) and compose\ninto the repo's H_img_to_court convention.
Landmark deltas vs hand fits (mean/max px): t5 2.7/3.1, t6 2.4/4.0,\nt7 4.7/8.6, t2 7.3/13.9, t4 8.0/15.2, g1 8.0/12.2, t3 15.7/29.0;\nt1 abstained (2/14 kps; config search requires a complete 4-point\nconfiguration). Config cross-validation errors 2.5–7.0 px on accepted\nfits. Classical auto-fit's g1 failure: rms 74 px (LOG, cv-18 era).
The 2× bug: postprocess(heatmap, scale=2) upscales heatmap coords\ninternally; wrapper scaled again → uniform doubling, self-consistent\nconfig-xval, ~750 px landmark deltas. Caught by the first overlay\nrender.
Next wiring (blueprint roadmap #2's remaining half): use this as the\nproposer inside fitcourt, judged by the existing step-5 line-mask\nscorer with a calibrated accept threshold (the TVCalib abstention\npattern), plus the 1 Hz drift watchdog. Night feeds: try per-clip\nplates, gamma/CLAHE boost, or fine-tuning with the hand-fit frames the\nproject already owns.
Hero renderer: experiments/render_court_hero.py (montage of the\nper-match overlays).
Session cost: $0.00 — pretrained weights, laptop inference. Project\ntotal: ~$16.
\nNext question: the court now fits itself and the machine hears every\nimpact, but the players are still smudges — blobs from background\nsubtraction, which is why forehand-versus-backhand has been stuck at a\nceiling for two weeks. The research says the upgrade is stick figures:\nwrists, elbows, hips, feet, drawn on every frame by a free model that\nruns at 87 frames a second on this laptop. Time to give the smudges\nskeletons.
", "url": "https://optic-yellow.com/blog/the-court-that-fits-itself", "title": "The Court That Fits Itself", "summary": "A human has clicked four corners for every match this project ever charted. A free pretrained network just took the job — and knew when to refuse it.", "date_modified": "2026-07-22T21:00:00.000Z", "author": { "name": "Trevor McCormick", "url": "https://github.com/trevormccormick" }, "tags": [ "Devlog", "Computer Vision" ] }, { "id": "https://optic-yellow.com/blog/the-machine-grows-ears", "content_html": "That clip up top: a 15-shot rally from the Roland Garros final, with\nthe soundtrack drawn as a strip along the bottom. Green ticks mark the\nmoments the video pipeline detected a hit. Yellow spikes mark the\nmoments the microphone heard an impact. The red zones are the seconds\nwhere the ball tracker recorded nothing at all — and the yellow spikes\nkeep firing right through them.
\nToday the software gets a second sense: when its eyes blink, can its\nears carry the point?
\nPreviously. The ball tracker — the tiny specialist network that\nfinds the ball in every frame (the zero-dollar\nspecialist) — has one chronic\nweakness: it blinks. On clay especially, its track is full of holes,\nand every downstream call (hits, bounces, rally length,\nwide-or-deep) inherits the\nblindness. Last post, physics recovered the landing spots the eyes\nnever saw. This post goes after the holes themselves, with a witness\nthat doesn't have eyes at all.
\nThink about how you experience tennis from the cheap seats, or the next\nroom. You can follow the rhythm of a rally without seeing it: pock —\npock — pock. The strike of strings on a ball is one of the most\ndistinctive sounds in sport — sharp, short, and unlike anything else on\na tennis court. Every broadcast microphone picks it up, no matter where\nthe ball is, how small it looks on screen, or whether it just left the\nframe. Crowd roar rises and falls slowly; commentary murmurs along; the\npock is a spike.
\nThat's the whole idea: onset detection — scan the soundtrack for\nsudden bursts of energy in the frequencies where racquet impacts live,\nand mark each one's exact moment. No AI, no training: the software\nslices the sound into 11-millisecond snippets, watches how much new\nenergy each snippet brings, and flags the spikes that tower over their\nneighborhood. Notice the unit: 11 milliseconds. A video frame is 33.\nThe ears, when they're right, are three times more precise than the\neyes can ever be.
\nOne discovery before any listening could happen: every video this\nproject has ever downloaded is silent. The staging pipeline grabbed\nvideo-only streams from day one — it never needed sound until today. So\nthe soundtracks had to be fetched fresh, audio-only, from the same\nuploads. For the Roland Garros match the source was known. For the US\nOpen match it had to be re-found — and verified the way this project\nverifies everything: by receipt. The candidate video's length matched\nthe match reel to a tenth of a second, and then the real proof: after\nlining up the two recordings, a single fixed time-shift (95\nmilliseconds on one match, 310 on the other) snapped hundreds of\naudio spikes onto video-detected hits at once. A wrong recording can't\nfake that — there is no single shift that aligns someone else's match.
\nTwo matches, both already graded against human charts: the US Open\nfinal (hard court, where the eyes work well) and the Roland Garros\nfinal (clay, where they don't).
\n| Question | US Open (hard) | Roland Garros (clay) |
|---|---|---|
| Video hits confirmed by an audio spike within 2 frames | 282 of 645 — about 4 in 9 | 37 of 76 — about half |
| ...within 3 frames | 57% | 66% |
| Impacts heard during the eyes' blind time | 156 events in 67 seconds | 196 events in 78 seconds |
Read that last row again, because it's the headline. Across the clay\nmatch, the ball tracker spent 78 seconds of rally time seeing nothing —\nand the microphone logged 196 impacts inside exactly those seconds.\nThe witness that never blinks works best precisely where the eyes\nfail worst. That's what the yellow-spikes-in-red-zones moments in the\nclip up top are showing.
\nAnd a small piece of physics showed up in the data, unasked. When the\ndetector is set strict — only the loudest spikes — the number of\nimpacts per shot settles at almost exactly two. Two impacts per\nshot is precisely what a court hears: the racquet's pock, then the\nbounce's thud. The microphone is telling us it hears both, cleanly\nenough to count.
\nSet looser, the ears hear too much. A tennis broadcast is full of\nthings that go pock: line calls, grunts, the umpire's microphone\nthump, a ball kid's bounce, applause spikes. At the settings that catch\nthe most real hits, the detector logs about three impacts per shot —\nthe extra one is chatter. Asked to count rally lengths by ear alone\n(spikes divided by two), it lands within one shot of the human chart\nonly 3 times in 10 on hard court, worse on clay.
\nSo the verdict, stated precisely: audio v1 is a witness, not a\ncounter. It confirms moments and fills blind time with candidate\nevents; it can't yet tell a forehand from a line call. Teaching it the\ndifference — pock versus thud versus everything else — is a small\nclassification job, and the training data is free: thousands of\naudio moments already labeled by the video hits they align with. That's\nthe same flywheel this project keeps returning to — one sense\nteaching another, no human labeling anything.
\nThe takeaway: the machine's eyes blink, but the court always makes\nthe same two sounds per shot — and now the machine can hear them,\ntimed finer than its own eyes can see.
\nThe module is experiments/audio_hits.py (report:\noutputs/diag/audio_hits_report.txt; run with\nPYTHONPATH=.:experiments uv run python experiments/audio_hits.py).\nPure numpy — no librosa/scipy dependency.
Method: mono 22.05 kHz wav (yt-dlp bestaudio of the same uploads;\nidentity by duration fingerprint — g1 5039.40 s vs reel 5039.45 s\n[known ID]; t6 2412.22 s vs 2412.13 s [condensed-match upload found by\nsearch]). STFT 1024/256 hann; log-magnitude spectral flux over\n1–8 kHz; normalized by sliding 2 s median/MAD; peaks above a z-score\nTHRESH with 0.10 s minimum separation. A/V offset per match by grid\nsearch (±0.5 s, 5 ms steps) maximizing video hits matched within\n40 ms: +95 ms (t6, 187/645 matched in search), +310 ms (g1, 34/76).
THRESH protocol: swept on t6 only (the calibration match), frozen\nbefore grading g1. Sweep: z=5 → 44% hits matched @2fr, 3.27\nonsets/true-shot; z=8 → 38%, 2.53; z=12 → 32%, 2.00 (the\nracquet+bounce ratio); z=16 → 26%, 1.50; z=20 → 25%, 1.25. Frozen at\n5.0 (highest within 5% of peak match rate). Naive rally length\n(onsets/2, serve-to-last-hit window) within ±1 of MCP truth: 39/128\n(30%) t6, 2/21 (10%) g1.
Hole metric: ball-track gaps >3 frames between serve and last hit + 1 s;\nonsets falling inside them after offset correction: 156 in 67 s (t6),\n196 in 78 s (g1).
Staging finds, on the record: (1) extract.py cuts clips with -an\nand ingest always fetched video-only — the whole corpus is silent by\nconstruction; audio staging is now a first-class ingest step. (2) g1's\nclip cutter (the experiments/-grade score-bug pipeline) never\npersisted its reel windows; recovered because every clip measures\nexactly 1.6 s longer than its alignment window (verified on four\nclips), and the unknown pre/post pad split folds into the calibrated\nA/V offset. Cutters must persist their windows — noted for the package\npromotion.
Precedent: audio hit detection on broadcast tennis goes back to\nAPSIPA 2012 (Australian Open footage, F-score 68–77% audio-only);\nSony AI's table-tennis impact work (arXiv 2409.11760) reports\nmillisecond precision in controlled capture. Audio-visual fusion gains\non broadcast sports: arXiv 2011.04258. The v2 plan (racquet/bounce/\nother mel-CNN on video-hit-aligned pseudo-labels) is blueprint roadmap\nitem 8's transcript-alignment flywheel applied to sound. Hero renderer:\nexperiments/render_ears_hero.py (g1_point_05, 15 shots, 4.0 s of\nholes, 12 onsets inside them).
Session cost: $0.00 — the audio tracks were free, and everything runs\non the laptop. Project total: ~$16.
\nNext question: the ears confirm the eyes and cover their blinks, but\nboth witnesses still hand their evidence to a court fit that a human\nclicked into place. Four corners, every match, by hand — the last big\nmanual knob on the machine. The soccer world already builds courts\nthat fit themselves and, more importantly, know when they've failed.\nTime to steal that too.
", "url": "https://optic-yellow.com/blog/the-machine-grows-ears", "title": "The Machine Grows Ears", "summary": "The ball tracker blinks at the worst moments. Broadcast microphones don't. The software learns to listen for the pock of strings on ball.", "date_modified": "2026-07-22T18:00:00.000Z", "author": { "name": "Trevor McCormick", "url": "https://github.com/trevormccormick" }, "tags": [ "Devlog", "Computer Vision" ] }, { "id": "https://optic-yellow.com/blog/nobody-sees-the-ball-land", "content_html": "That clip up top: two real points from the US Open final. In each one,\nthe yellow dots are the machine watching the ball fly — until the dots\nstop, mid-air, because the tracker loses it. Then the freeze-frame: a\nwhite line extends the flight the machine did see, racing toward two\nfinish lines — the sideline and the baseline — and whichever it crosses\nfirst names the error. Both calls match what the human charter wrote.
\nToday the software learns to answer a question it has been failing,\ncompletely, since grading began: when a ball goes out, did it go out\nwide or deep?
\nPreviously. This machine watches broadcast tennis and drafts a\nshot-by-shot chart of every point, which then gets graded, correction\nby correction, against charts written by human volunteers (how the\ngrading works). Most parts of the\ndraft have been inching upward for weeks. One part has not moved at\nall: when a rally ends with a ball landing out, the machine scored zero\npercent at saying which way it went out. Not \"poor.\" Zero — across 159\ntries, on eight matches, twice confirmed by different methods. The\nreason is simple and physical: the ball tracker loses sight of the ball\na split second before an out-ball lands. Fuzzy yellow blur, busy\nbackground, and the one moment that decides the call is exactly the\nmoment the machine goes blind.
\nThe obvious fix — make the tracker hold on longer — is the one I'd been\nplanning. Then a research deep-dive turned up something better, hiding\nin plain sight.
\nThink about what Hawk-Eye actually shows you. That smooth animated arc,\nthe ball flattening onto the paint, the \"OUT\" verdict by two\nmillimeters — none of that is a photograph. No camera on Centre Court\ncaptures the ball touching the ground; at that instant the ball is a\nsmear of pixels, half-hidden, moving at highway speed. What the system\nphotographs is the flight. The landing is calculated — physics,\nfitted to the part of the flight the cameras saw clearly, finishes the\nsentence.
\nA veteran line judge does the same thing. She doesn't wait to study the\nskid mark. She reads the arc coming off the racket and knows the ball\nis sailing long before it lands. The information was in the flight\nthe whole time.
\nThat's the whole idea I stole this week: the boundary race — stop\ntrying to see the landing, and instead extend the flight the machine\ndid see until it crosses a line. If the extended path crosses the\nsideline first, the ball was going wide. If it crosses the\nbaseline first, it was going deep. Whichever finish line the flight\nreaches first names the error. (One more entry in the plain-English\nprimer, where the recurring ideas of this\nseries live.)
\nNo new AI. No new camera. Straight-line physics, fitted to the last\nstretch of dots the tracker recorded before it blinked — and two papers\nat this year's top computer-vision conference validate the family of\nidea, one of them placing landings within about 0.6 meters from a\nsingle broadcast camera.
\nI ran the race on every point in the benchmark — the eight matches\nwhere human charters already wrote down the truth — and looked only at\nthe points that genuinely ended with a ball landing out: 169 of them,\nof which 159 were plainly wide or deep (the rest were charted as both\nat once).
\n| Question | Before | Now |
|---|---|---|
| Wide and deep errors named exactly right | 0 of 159 | 75 of 159 — about half |
| Deep errors caught | 0 of 90 | 55 of 90 — about 6 in 10 |
| Wide errors caught | 0 of 69 | 20 of 69 — about 3 in 10 |
| When the race commits to wide-or-deep, how often is it right? | — | 75 of 90 — about 5 in 6 |
From nothing to half, with arithmetic. And when the race actually picks\na side — wide or deep, rather than hedging \"both\" or declining — it\nseparates the two at about five in six, on flights the tracker gave up\non.
\nThe misses are honest and mostly one story: the TV camera looks down\nthe court, so a flying ball always appears deeper than it really is —\nthe same distortion that makes every approach shot look like it's\nsailing long from your couch. That bias makes the baseline finish line\nfire early, so some wide balls get called deep or \"both.\" Fixing that\nlean is a real project for later, not a knob to fudge today.
\nRendering that clip up top wasn't decoration — it caught two real bugs\nthe scoreboard had quietly absorbed.
\nFirst: in one point, the machine's \"last sighting\" marker drew itself\nat the top of the screen while the ball's trail exited bottom-right.\nImpossible. The track data showed why: the ball supposedly moved 37\nmeters in three frames. Balls don't teleport — the tracker had lost the\nreal ball and latched onto something else, and those junk points were\npoisoning the physics fit. The fix is a rule of nature, not a tuned\nsetting: end the flight at the first physically impossible jump.
\nSecond: the race originally asked an earlier stage of the pipeline\nwhich end of the court the final shot was heading toward — and that\nstage was sometimes wrong. One point claimed the far player hit the\nlast shot, while the ball demonstrably died two meters beyond the far\nbaseline. Now the race reads the direction from the flight itself and\ntrusts nobody.
\nBoth bugs were invisible in the accuracy tables and obvious in one\nrendered frame. This series keeps re-learning the same lesson, so it's\nearned a bolding: eyes beat derivatives — when a number and a\npicture disagree, believe the picture.
\nOne more result, because this devlog keeps its failures on the record:\nI also tried the lazy version of \"integration\" — feeding the race's\noutputs as extra hints into the existing ending classifier, the one\nthat learned from 11,646 human-charted matches. It did nothing. Zero\npercent stayed zero.
\nThe reason is instructive. Winners — balls that land in and end the\npoint — are nearly half of all endings, and thanks to that same\nlooks-deeper-than-it-is distortion, the race happily extends many\nin-balls out too. A single flat classifier drowns the new signal in the\nold base rates. The race's real gift is conditional: given that a\nball went out, it names the flavor. Deciding whether the ball went out\nat all needs different witnesses — a net-death, an observed landing,\nand soon (next on the roadmap) the sound of the ball itself. The\narchitecture lesson: don't ask one judge to answer two different\nquestions.
\nThe takeaway: the machine stopped trying to see the ball land, read\nthe flight instead — the way Hawk-Eye and every good line judge always\nhave — and half of a stone-blind category came back at the cost of\nzero dollars and some algebra.
\nThe module is experiments/landing_spot.py in the court-vision repo\n(report: outputs/diag/landing_spot_report.txt; run with\nPYTHONPATH=.:experiments uv run python experiments/landing_spot.py).
Method (v1, deterministic, no constants fitted to truth): final flight\nsegment = contiguous ball-track run after the last charted contact,\ncapped at 2.0 s, cut at the first >6-frame hole and at the first\nteleport (>3.0 m/frame court displacement — ~90 m/s, physics bound; the\nsame no-teleport gate boxes.py applies to players). Straight-line fit\ncx(t), cy(t) over the last ≤8 samples in court coordinates; target half\ninferred from the flight (already-out position wins, else the y-slope\nsign, else which half the ball occupies). Race: time-to-sideline\n(singles line ± the 0.25 m OUT_MARGIN from endings.py) vs\ntime-to-baseline, extrapolation capped at 0.7 s past track death;\ncrossings within 2 frames of each other → x (wide-and-deep); neither\nwithin cap → abstain ?.
Population: evaluate.evaluate records, truth = mcp_ending_type\n(w/d/x), same denominators as experiments/learn_components.py plus\nmatch #8 (g1). Full confusion (rows = truth):
w d x ?
w 20 11 27 11 n=69
d 4 55 18 13 n=90
x 1 0 8 1 n=10
Loose (called any out flavor): wide 58/69 (84%), deep 77/90 (86%).\nCommitted w-vs-d discrimination: 75/90 (83%). Per-match strict correct:\nt7 27/58, t6 28/44, t5 10/29, t4 9/14, t3 4/10, g1 3/11, t1 1/2, t2\n1/1 — clay (t3, g1) collapses because thin tracks leave no segment to\nfit, which the planned clay fine-tune of the ball detector multiplies.
Version history within the session, all on the record: striker-derived\ntarget + no teleport gate → wide 30/69, deep 35/90 (overall 41%);\n+teleport gate → 31/69, 33/90 (41%); +flight-inferred target → 20/69,\n55/90, overall 83/169 (49%), with the wide→x leak (27) being the\ndepth-inflation bias concentrating inside the 2-frame \"both\" window.
The refuted integration: appending racer features (call one-hot,\nt_side, t_base, n_seg) to the 4-class leave-one-match-out random\nforest (learn_components.lomo, seed 0) moved wide/deep recall 0%→1%\nand cost overall accuracy (59.0%→57.4%). False out-calls on true\nwinners/net: 204/317 (64%) — the racer is a conditional classifier\nP(wide vs deep | out), not an out-detector; it will enter the pipeline\nas a per-slot likelihood in the planned grammar-constrained fusion\ndecoder, gated by independent out-ness evidence.
Literature anchors: TT3D (arXiv 2504.10035, CVSports @ CVPR 2025) and\n\"Where Is The Ball\" (arXiv 2506.05763, same workshop; 0.63 m mean\nlanding error on broadcast tennis, 81 clips/10 matches). The hero video\nrenderer is experiments/render_landing_hero.py (points t6_point_110,\ntruth wide; t6_point_117, truth deep).
Session cost: $0.00. Project total: ~$16.
\nNext question: the race can name an out-ball's flavor, but something\nstill has to say \"that ball went out\" in the first place — and the\ntracker's eyes have proven they blink at exactly the wrong moment. The\nnext witness doesn't have eyes at all. Every broadcast microphone\ncourtside hears the same thing you do: the pock of strings on ball.\nTime to give the machine ears.
", "url": "https://optic-yellow.com/blog/nobody-sees-the-ball-land", "title": "Nobody Sees the Ball Land", "summary": "The machine couldn't tell wide errors from deep ones. The fix: stop trying to watch the bounce and calculate it, like Hawk-Eye does.", "date_modified": "2026-07-22T10:00:00.000Z", "author": { "name": "Trevor McCormick", "url": "https://github.com/trevormccormick" }, "tags": [ "Devlog", "Computer Vision" ] }, { "id": "https://optic-yellow.com/blog/the-33rd-charter", "content_html": "That clip up top: the actual file, playing. All 134 points of the\nSabalenka–Pegula US Open final draft scroll past, and 39 light up\ngreen — \"trust this enough to correct instead of typing from\nscratch,\" a claim that held up 94% of the time on matches the model\nhad never seen. Two green rows then play against their video clips,\ndraft string and jump-to timestamp on screen. It ends with what the\nfile admits it can't do yet, and a question for anyone who charts\ntennis for real.
\nFor sixteen posts, this project has graded its own homework — checking\ndrafts against charts it already had the answers to. Today, for the\nfirst time, a draft goes to a real judge: someone who charts tennis\nfor a living, whose only job is to say whether it helped.
\nPreviously: cv-01 set the destination —\nthe Match Charting Project (charters call it the MCP), run by Jeff\nSackmann of Tennis Abstract, is volunteers watching tennis and writing\ndown every shot in a compact code; nearly 18,000 matches charted so\nfar. cv-14 gave the project a way\nto grade a machine-written line against a human one: how many\ncorrections an editor would need, the token edit distance\n(more here).\ncv-15 turned that grade into a\nstoplight — HIGH (\"start from this draft\") or LOW (\"re-chart it\") —\ntuned so the flag never grades its own homework\n(more on confidence).\ncv-16 grew the test to seven matches\nacross five broadcast feeds, 491 human-charted points, fixed a blind\nspot on the Wimbledon match, and landed the number this post leans on:\nHIGH is right about 94% of the time.
\nNone of it has been true for long. In 2025 the MCP ran on 32 active\ncharters,\ncovering about a quarter of tour matches — and Sackmann published a\nplea for more. That's the door this post walks through.
\nBeing graded correctly isn't the same as being useful. Groove a stroke\nagainst a ball machine for months and you prove your technique\nreproduces — never that it survives a live point, because a machine\ncan't return your shot and tell you how it felt. Every number this\nproject has published so far graded the pipeline's aim, not its use:\ndoes the drafted string match a human's, was a flag right when checked\nlater. Nobody who charts a match for a living has ever opened one of\nthese files and tried to work from it.
\nThat's the gap this post closes. The step where a tool's output leaves\nthe lab, lands on a real person's desk, and the only score that counts\nis what they report back, is the human loop: grading against a\nfixed answer stops, and getting judged by whether you saved someone\nreal time starts. Nothing in the pipeline changed this session — who's\non the other side of the net did.
\nIn tennis terms: the ask is what a doubles player makes of a\nprospective partner before ever stepping on court — play one set with\nme, then tell me if I made you faster or slower.
\nSeven files went public today, one per benchmark match, in the exact\ncolumn layout the MCP already uses for its own point-by-point files.\nThe familiar columns are all there — match, point number, set score,\ngame score, point score, server — a real charter could read those\nblind. Then come the columns Court Vision added:
\n| Column | What's in it | What it means |
|---|---|---|
1st | e.g. s5b2f1f2f2x@ | the machine's guess at the point, serve to last shot, in a charter's own shorthand |
2nd | always blank | can't yet tell a first serve from a second, so second-serve rallies are simply missing |
confidence | HIGH or LOW | HIGH: trust it enough to correct. LOW: rough hint, expect a rewrite |
conf_p | e.g. 0.902 | the raw score behind the label — closer to 1 is more confident |
clip | a filename | which extracted broadcast clip the row came from |
serve_s | seconds | where the serve happens in that clip — the jump-to |
n_shots | a small number | shots in the point, serve included |
A real row, from the US Open final, flagged HIGH at 0.902:\ns5b2f1f2f2x@. Read left to right: s5 — serve into the body. b2 —\nbackhand return, up the middle. f1, f2, f2 — three forehands,\none into an outer third, then twice down the middle. x@ — unforced\nerror, wide and deep. Six tokens, one point.
Every match's file, with how many points it drafted and how many got\nthe green flag:
\n| Match | Points drafted | Flagged trustworthy |
|---|---|---|
| Nadal vs. Shapovalov, Montreal 2017 | 24 | 10 |
| Federer vs. Haase, Montreal 2017 | 6 | 4 |
| Djokovic vs. Ruud, French Open final 2023 | 59 | 12 |
| Krejcikova vs. Paolini, Wimbledon final 2024 | 49 | 18 |
| Sinner vs. Zverev, Australian Open final 2025 | 71 | 3 |
| Sabalenka vs. Pegula, US Open final 2024 | 134 | 39 |
| Djokovic vs. Sinner, ATP Finals 2023 | 165 | 13 |
| Total | 508 | 99 |
99 of 508 — about one point in five — got the green flag; cv-16 found\nit right about 94% of the time on matches the grading model never\ntrained on. Start with the US Open final: best hit rate in the batch,\nand the model called who served right 95% of the time there too — a\ncamera that holds still pays off.
\nA drill doesn't prove anything until it's played against a real\nopponent. That's the whole reason this post exists.
\nA draft that hides its weak spots wastes an hour, so the weak spots\ncome first.
\nThe drafts are drafts. Across all 491 human-charted points behind\nthe whole benchmark, only 5.7% — roughly 1 in 18 — match the human's\nline closely enough to need just one fix. The real value sits in the\ngreen rows: HIGH means within five corrections, holding up 94% of the\ntime on matches the grading model never trained on (96% on matches it\ndid study — the friendlier, less trustworthy number, included\nanyway). LOW, 409 of 508 rows, means exactly what it sounds like:\nre-chart from the video, draft as hint, serve_s as the jump-to.
Every missed serve is invisible. The pipeline can't yet see a\nfault, so every point is written as if won on the first serve, and the\nsecond-serve column is empty on all 508 rows. On a real scoresheet,\nroughly a third of points start with a miss. Fixing that isn't a\nfootnote of the charter's pass — it's most of it.
\nThe shorthand isn't official yet. The s before a serve number is\nthe pipeline's own invention, not real MCP notation, and a ? shows\nup when the machine can't tell which way a shot went. The draft also\nskips most of what a full MCP chart captures — no depth, no court\nposition, no volley vs. lob vs. drop shot; endings and rough\ndirections only. Whether to write strict, official strings with\nblanks instead, or keep the uncertainty visible, is undecided —\nquestion three, below.
Seventeen rows have no point number. The machine caught the point\non video but couldn't match it to a spot on the scoreboard, so that\ncolumn is blank. The point happened; a person still has to say where\nit belongs.
\nOne answer key has problems of its own. cv-16's autopsy caught the\nWimbledon final's human chart running at half speed on some rallies —\nstrokes written down that the video contradicts. That file still\nships, 18 of its 49 rows green. Open that one first if you're hunting\nfor the model being too sure of itself.
\nIf you chart for the MCP — or ever have — three questions matter here,\nin the order they'd change what gets built next.
\nThe stopwatch. Chart ten points cold, straight off a clip.\nThen correct ten different green-flagged drafts against their\nclips instead. Which pass is faster, and by how much? The whole\nproject is a bet on that ratio, never yet measured by anyone who\nactually holds the pencil.
\nThe triage. Is a draft right 94% of the time, within five\ncorrections, the right bar for \"start from this\"? And with only\nabout one point in five earning that flag, does a pre-filled file\neven help mid-session — or does switching between \"check this\" and\n\"chart from nothing\" cost more than it saves?
\nThe schema. What would this file need before you'd reach for it\nmid-session — official MCP strings with blanks, a real fault\ncolumn, a charting-sheet layout instead of a spreadsheet, something\nelse entirely?
\nThe channel is an issue on\ncourt-vision\n— a one-line \"this wastes my time because X\" is signal, not rudeness.\nIt would be the single most valuable sentence anyone has contributed\nto this project.
\nWorth knowing before opening these files: the score and server columns\nexist because these drafts were lined up against the MCP's own\npublished charts — the same charts the 94% number is graded against.\nSo these files carry the MCP's license,\nCC BY-NC-SA 4.0:\ncredit the source, don't sell it, share what's built from it the same\nway. The draft strings and confidence columns are this project's own\naddition, released under that license to keep the files one whole\nthing.
\nExact numbers, for anyone doing their own math:
t1–t7, internal match IDs, same order as the table\nabove): t1 Nadal–Shapovalov, t2 Federer–Haase, t3 Djokovic–Ruud\n(clay), t4 Krejcikova–Paolini (grass), t5 Sinner–Zverev (night), t6\nSabalenka–Pegula, t7 Djokovic–Sinner (indoor). Points/HIGH per match:\ntable above.match_id, Pt, Set1/Set2, Gm1/Gm2, Pts, Svr, 1st/2nd — MCP's own\npoints-file columns, preserved as-is.Session cost: $0.00 — nothing ran but a video renderer. Project total:\n~$16. The next number that matters isn't the pipeline's to produce:\nit's one charter's minutes-per-point, with and without the draft.
", "url": "https://optic-yellow.com/blog/the-33rd-charter", "title": "The 33rd Charter", "summary": "Court Vision introduces itself to the real volunteers who chart tennis matches by hand, and asks whether its drafts actually save them time.", "date_modified": "2026-07-11T14:00:00.000Z", "author": { "name": "Trevor McCormick", "url": "https://github.com/trevormccormick" }, "tags": [ "Devlog", "Computer Vision" ] }, { "id": "https://optic-yellow.com/blog/the-benchmark-grows-up", "content_html": "That clip up top: watch the test set grow, tile by tile, from four\nmatches to seven — the point counter climbing from 135 to 491\nhuman-charted points. Then its self-reported trust score recalculates\nlive on screen — down, then back up — as the reason for the dip gets\nfound and fixed. It closes on a real point\nfrom the Sabalenka–Pegula U.S. Open final: a serve into the body, a\nbackhand up the middle, a forehand rally ending wide and deep —\nflagged correctly trustworthy at a 0.902 score by a version of the\ntool that had never watched that match before.
\nWould you trust a player's world ranking more if it came from one\ndazzling week at one tournament, or from a whole season played across\ndifferent surfaces and different opponents?
\nTwo posts back, cv-14 gave the\nproject its scorecard: count how many fixes — changes, insertions,\ndeletions — a human editor would need to turn the machine's chart\ninto the real one — an acceptance at\none fix or fewer. cv-15 then built a trust flag on top\nof that scorecard: every drafted point gets stamped HIGH or LOW, and\nHIGH means \"start correcting from this draft.\" But that flag had only\never been tested against 135 points from four matches — barely 1 in 9\nof which passed the one-fix bar to begin with — and cv-15 signed off\nwith HIGH at 93% reliable, plus a warning: 135 points wasn't enough\nto trust yet, and three more matches were already on the way.
\nHere's why that matters. A hot week can flatter a ranking — a small\nfield, a favorable draw, a home crowd — and it isn't wrong exactly,\nit's just thin. A full season is the harder test: more matches,\ntougher and more varied opponents, and the ranking is allowed to go\ndown afterward even though the player didn't get worse. It's finally\nbeing asked to hold up over more evidence, and a ranking that\nsurvives a full season is worth more than a flashier one built on a\nsingle great week, precisely because it was harder to earn.
\nThat's recalibration: redoing the trust-score math against a\nbigger, tougher set of examples, and trusting the number wherever it\nlands — even down. (More on how these trust scores get\nbuilt.) The confidence flag lives or dies on two\nnumbers: how often it's even willing to say HIGH at all — called\ncoverage — and how often HIGH is actually right when it does —\ncalled precision. It's the same trade a line judge faces: call\neverything close and you're helpful but wrong a lot; call only the\nobvious ones and you're always right but useless. The cv-15 flag had\nonly faced one hot tournament's worth of evidence. This session gave\nit a full season.
\nThe pool of candidates was every match the Match Charting Project's\nvolunteers have hand-charted — this project's answer key, explained\nhere — cross-checked against what's findable\non YouTube. The winning format was the \"condensed match\" upload: a\n20-to-40-minute re-cut that keeps nearly every point and skips only\nthe between-point filler, which lines up with the human chart far\nbetter than a highlights reel. Three finals made the cut: the\nAustralian Open final at night, the U.S. Open final, and the ATP\nFinals indoors in Turin — 575 more charted points to draw from.
\nAll three ran through the pipeline's existing machinery untouched —\nno new code, just settings. Each broadcast had one quirk: a court's\nblue paint too bright for the usual color threshold, a green sideline\napron too close to the court's own color, a hazy light-blue apron\nneeding a brighter ceiling — all one-line settings fixes. One old\nlesson got relearned, too: the recipe that converts a screen position\ninto a court position — the homography — has to\nbe built from footage where the camera is actually holding still. The\nfirst attempt smeared because the camera was still panning; a static\nfive-second window during the serve toss fixed it.
\nNew broadcasts also surfaced a new headache and caught an old bug.\nThe headache: all three hide the scoreboard\ngraphic at 0-0, briefly confusing the tool that\nuses it to tell points apart — solved by cross-checking the human\ncharter's own serve notes. The bug: the method for tracking which end\nof the court a player serves from had been quietly miscounting\ntiebreak games as a full 13th game instead of the shorter,\ndifferently-scored game it actually is — a one-line fix that flipped\na whole match's worth of serve-end guesses from mostly wrong to\nmostly right. And on the record: an overnight background job died\nsilently, and two matches' worth of finished work sat untouched until\nsomeone checked by hand.
\nNone of that dented the results. Scored against the pipeline's\nunchanged rules, the three new matches set new highs: the U.S. Open\nmatch got the serving end right 121 times out of 128 (95%), and\nTurin got rally length right, give or take one shot, 131 times out of\n157 (83%). On the strict pass-fail test, the bigger set held\nessentially flat: 28 of\n491 points needed no correction at all (5.7%), close to the earlier 7\nof 135 (5.2%). The pipeline didn't get worse by testing it harder.
\nOn the small set — 135 points, four matches — the confidence flag's\nHIGH rating was right 93% of the time, willing to call HIGH on 32.6%\nof points. Recalculated on the full season — 491 points, seven\nmatches, five broadcasts, same math, no changes — the honest numbers\ncame in lower: 88% right, called on 21.2% of points. That's not the\ntool getting worse. That's small-sample flattery washing out.
\nOne match makes the point almost too neatly. In the small test, the\nWimbledon match looked flawless: 11 for 11, 100% right, the best\nresult in the table. In the big test, with more than double the\nWimbledon points to check, that same match fell to 69% right while\nclaiming to flag more than half the match. The small test wasn't\nrigged — it just didn't have enough Wimbledon points yet to catch\nwhat was wrong. More data didn't break Wimbledon's score. It let the\nproject finally see it, and go looking for why.
\nThe why turned out to be a single broken check.
\ncv-15's best idea was a serve check: a real toss lands the ball's\nlaunch point more than 20 meters beyond the baseline, so a \"serve\"\nthat starts from inside the court is really a broadcast cutting into\nthe middle of an already-running point — and the draft gets marked\ndown for it. On its own, it had been the confidence project's single\nbiggest improvement.
\nBut it only works when the system identified the serve by watching\nthe ball's own flight. When the ball-tracking data is too spotty to\ntrust, it falls back to guessing the serve from the players' stance\nand position instead — and on the Wimbledon broadcast, every single\nserve was identified that second way. So the serve check had zero\nreal serves of its kind to examine, for an entire tournament's worth\nof points. It wasn't passing every point because it checked them and\nthey held up; it had nothing to check, and defaulted to \"trust it\"\nsilently, every time.
\nUnder that free pass, three failures went unnoticed. One point's\ndrafted chart recorded 8 shots, evenly spaced about a second and a\nhalf apart — but the real footage runs twice as fast, a shot every\n0.9 seconds, meaning the shot-detector had missed every other stroke.\nIt still looked trustworthy because the who's-hitting-next check\nhappened to add up anyway — the same ball-tracking blind spots hit\nboth checks the same way, so the errors canceled out instead of\nstanding out. A second clip opens\non a slow crossfade into a rally already underway — five real seconds\nof tennis before the stroke the stance method called \"the serve.\"\ncv-15 had already caught this trick\nonce, on the clay-court match, where the broadcast cut hard into the\nmiddle of a point; Wimbledon's editor does the same thing with a fade\ninstead of a cut — exactly what the (silently broken) serve check\nexists to catch. And a third case: a four-shot rally in the human's\nchart where the ball-tracking data never shows a single net crossing\n— the ball's path across the net is a\nrally's skeleton, and this rally had none.
\nThe fix wasn't to patch the broken check. It was to add two checks\nthat look at the whole point instead of just the serve. First: does\nthe ball cross the net, at rally speed, before the moment the chart\nclaims the serve happened? If so, the clip started mid-rally already\n— catching both the hard-cut and the fade. Second: does the charted\nshot sequence show any net crossings in the tracking data at all? No\ncrossings, no trust, however many shots the chart claims. Swapping\nthe one broken check for these two took Wimbledon's held-out score\nfrom 69% to 85%, and the project's overall number from 88% to 94%,\nwhile cutting the worst-case disaster flags roughly in half.
\nThe project already learned this lesson once, in a different\ncostume. Back in cv-14, the\nmachine's shot-direction calls came out backwards for a large share\nof points, because the code assumed a fixed side of the court instead\nof checking which end the receiver was standing at — nobody noticed\nuntil the scorecard itself gave the bug away. The Wimbledon check is\nthe same mistake in a different jersey: a trust score is only as\nhonest as its least-examined assumption. The project's best idea\nhad quietly been a rubber stamp on one entire broadcast. The audit\nquestion was never \"is this check smart\" — it's \"can this check even\nfire, here, on this footage.\"
\n| Test round | Willing to say HIGH (coverage) | Right when it did (precision) |
|---|---|---|
| Small test — 135 points, 4 matches (cv-15) | 32.6% | 93% |
| Big test — 491 points, 7 matches, before the Wimbledon fix | 21.2% | 88% |
| Big test — 491 points, 7 matches, after the Wimbledon fix | 19.6% | 94% |
The trade for that last jump is real: 1.6 fewer points of coverage,\nfor a genuinely more honest number — the right trade every time. See\nCourt Vision in Plain English for why coverage\nand precision have to be read together.
\nA let-cord sensor that never buzzes isn't proof of a clean serve. It\nmight just be unplugged — and the only way anyone finds out is by\nchecking whether it could have buzzed at all.
\nThe three matches. t5 = Sinner–Zverev, Australian Open final 2025\n(hard, night, AO feed). t6 = Sabalenka–Pegula, US Open final 2024\n(hard, WTA, USO feed). t7 = Djokovic–Sinner, ATP Finals 2023 (indoor\nhard, Tennis TV feed). 575 MCP points on the table, 372 clips\nextracted, 356 scored.
Config-only integration. Every match ran through the existing\nfitcourt / probe / extract / boundaries / align stages plus\none YAML per match — no script forks. Fixes landed in court_detect\nconfig: t5's bright coating starved the tophat threshold default\n(raised); t6's green apron shares an HSV band with its blue court (a\nsecond hull band, OR'd in); t7's light-blue apron rides V≈252 (value\nceiling opened). Homography fit window: t5's first window carried\n26.5 px of camera pan smear, fixed via a static 5-second serve-setup\nwindow (≤0.2 px residuals).
Zero-valued columns. All three broadcasters hide the score-bug at\n0-0, which splits game-start points into duplicate-score fragments.\nAdjudication: MCP's own serve columns arbitrate most groups (a human\nrow reading 4w plus a second-serve rally means the fragment pair is\none point, serve in the later piece); the rest fell to changeover\nparity and eyeballed frame strips. Losers get blanked so the eval\nskips rather than double-charges.
Tiebreak parity bug. The staged prior counted Turin's set 2 as 12\ngames + a tiebreak = 13 swap-units (odd). Wrong: a 7-4 breaker is 11\npoints with one internal end-change at 6, and the set-end change\ncancels it — even parity. The set-3 server-end vote had come back 17\nagree / 37 disagree, systematically flipped; correcting the prior\nflipped it to 37/17, verified on video.
Process failure. The run stalled overnight: the background-job\nwatcher died silently between t6 ball tracking and t7 players jobs —\nboth finished fine, nothing was listening. The rest of the run polled\njobs directly instead of trusting completion callbacks.
Per-match scorecard, same eval, same constants:
metric t5 AO night t6 USO WTA t7 Turin indoor
server end 53/71 (75%) 121/128 (95%) 133/157 (85%)
rally length ±1 47/71 (66%) 99/128 (77%) 131/157 (83%)
letters (aligned) 93/117 (79%) 120/148 (81%) 112/157 (71%)
acceptance ≤1 edit 2/71 10/128 9/157
Pooled acceptance: 28/491 (5.7%), up from 7/135 (5.2%).
Full leave-one-match-out table, before the Wimbledon fix:
LOMO (held out) high prec (≤5 edits) coverage
t1 night 10/10 (100%) 45.5%
t2 ctrl 3/3 (100%) 60.0%
t3 clay 11/12 (92%) 20.3%
t4 grass 18/26 (69%) 53.1%
t5 AO 3/4 (75%) 5.6%
t6 USO 34/35 (97%) 27.3%
t7 Turin 13/14 (93%) 8.9%
pooled 92/104 (88%) 21.2%
The strict ≤2-edit tier was re-attempted at n=491 and still dies in\nleave-one-match-out (0% coverage at a 19% base rate) — not shipped.
The t4 (Wimbledon) autopsy. Eight false-highs, each pulled apart\nagainst its chart CSV, the human's string, and frame strips. The\nlaunch-plausibility gate (cv-15's best find) only inspects ball-called\nserves; every t4 serve is stance-called; the gate was vacuous on the\nentire feed, contributing +0.59 logit of unearned trust per point.
Three named mechanisms: (1) t4_point_02, the half-cadence chart — 8\nshots at 1.5s spacing against a real ~0.9s cadence, striker chain\nalternating cleanly because the ball track is blind in the same\nplaces the hit detector is; (2) the dissolve-cut mid-rally join — a\ncrossfade into a live rally, 5 seconds of tennis before the stroke\nthe stance detector blessed as \"the serve\"; (3) the spineless rally —\na 4-shot chart with zero net crossings even at the weak gates.
Two dead ends, kept: weak-gated crossings don't recover the missing\nstrokes (point_02's 17-stroke rally yields 7 weak crossings); charted\ncadence is real but soft (AUC 0.62, marginal flags 50/50) — a sidecar\ncolumn, not a model feature.
What shipped: xr_pre_serve (rally-speed net crossings ending\nbefore the charted serve) and rally_spineless (a 3+-shot chart with\nno spine at all).
LOMO (held out) before after
t4 grass 18/26 (69%) @ 53.1% 17/20 (85%) @ 40.8%
pooled 92/104 (88%) @ 21.2% 90/96 (94%) @ 19.6%
No other fold fell below 92%. High-tier disasters halved, 12 → 6. The\ntrade was 1.6 points of coverage. Residual: t4_point_11 stays\nflagged at 10 edits — its dissolve-cut leaves no pre-serve crossings\nbecause the track is blind there too.
Final refit. The shipped scorer refit on all 491 flags 97 points\nhigh at 96% in-sample; exports regenerated for all seven matches —\n508 draft points, 99 flagged high.
Where things stand now: seven matches, five broadcasts, 491\nhuman-charted points on the books, and a HIGH flag that means \"start\nfrom this draft\" and is right about 94% of the time on the honest,\nheld-out test — for roughly 1 in 5 points confident enough to earn\nthe flag at all. The benchmark finally has the size its own warnings\nhad been asking for.
\nSession cost: $0.00 — three tournament finals' worth of brand-new\nanswer-key data, added for zero marginal dollars. Project total so\nfar: about $16.
", "url": "https://optic-yellow.com/blog/the-benchmark-grows-up", "title": "The Benchmark Grows Up", "summary": "A bigger test made the trust score honestly drop — then a check that could never fail got caught, and the score climbed back up.", "date_modified": "2026-07-11T02:00:00.000Z", "author": { "name": "Trevor McCormick", "url": "https://github.com/trevormccormick" }, "tags": [ "Devlog", "Computer Vision" ] }, { "id": "https://optic-yellow.com/blog/the-draft-grades-itself", "content_html": "That clip up top: the machine's actual draft of the Roland-Garros\nfinal, point by point — each gets a chart string, a confidence score,\nand a flag, HIGH rows glowing green. Three flagged points then play\nback with their draft on screen, from a clean ace (s4*: serve out\nwide, unreturned, 0.969 confidence) to a messy point full of \"not\nsure\" marks (0.933 confidence). The running tally: 59 points drafted,\n16 flagged \"start from this draft.\"
Say a machine hands a volunteer chart-corrector a stack of drafted\npoints — how's she supposed to know, before she's watched a single\none, which drafts are worth trusting and which she should chart from\nscratch?
\nPreviously: cv-14 gave the\nproject its report card — the token edit\ndistance, how many fixes an editor needs\nto turn the machine's chart into a human's, a point \"accepted\" at one\nfix or fewer. cv-10 set the rule for\ngrading any of it: score only against matches a human already\ncharted, never a match the pipeline saw while being tuned. And since\ncv-05, players get found for free by\nbackground subtraction — photograph\nan empty court, call anything different in a later frame a person.\nToday tests whether that free method is good enough, then teaches the\ndraft to flag its own reliability.
\nPicture two line judges. The first only calls a ball \"out\" when it\nlands a clear foot past the line. She's almost never wrong, but\nrarely speaks, leaving a player who trusts only her guessing on\neverything else. The second judge calls anything that looks close —\nshe catches nearly everything worth catching, but plenty of her calls\nare wrong on replay.
\nA confidence flag sits on that same spectrum, and two numbers say\nwhere. Coverage is how often it's willing to call HIGH at all —\nhow often the strict judge speaks. Precision is how often a HIGH\ncall is actually right. A flag that's always right but rarely fires\nisn't useful, and neither is one that fires constantly and is often\nwrong — you need both numbers together. (More in the\nprimer.)
\nMaking those numbers honest takes a third idea: calibration. Take\npoints a human volunteer has already charted — the answer\nkey — and check signals the pipeline keeps\nabout its own effort: did the serve call commit cleanly? does the\nball's path have gaps? did two vote-counting methods for \"who hit\nthat\" agree? Fit those signals to the real outcomes and the fit\nbecomes a small scoring formula.
\nThe trap: a formula that peeks at a match's answers while being built\nwill always look brilliant tested on that same match — an open-book\nquiz, not a test. So fit it on three matches, then grade it, cold, on\nthe fourth it's never seen, rotating until every match gets a turn\nheld back. Tennis translation: the system never grades its own\nhomework — the same rule cv-10 set for the pipeline, now applied to\nthe flag that grades its work.
\nToday's first question: how much of the draft's remaining mistakes\ntrace back to the free player-finder misjudging where people stand?
\nEvery shot call — forehand or backhand, roughly which third of the\ncourt — comes from one rule: whichever side of the player's outline\nthe ball arrives at, that's the shot. A bad outline poisons the call,\nalready the biggest single source of remaining edits, about 1.7\ncorrections per point. Sorting every scored shot by outline quality at\ncontact:
\n| player outline at contact | shot called right | shot called wrong | system gave up |
|---|---|---|---|
| clearly fine | 70 | 17 | 11 |
| shaky | 31 | 18 | 23 |
| missing entirely | 0 | 0 | 8 |
Nearly half of every shot call — 45% — was riding on a shaky or\nmissing outline; accuracy was 71% on clean outlines, 39% on shaky\nones. Ceiling: about 26 more correct calls.
\nFree cleanup — rejecting teleporting outlines, checking court-half,\nfilling short gaps, widening for legs-only silhouettes — bought back 3\nof those 26: about an eighth of the ceiling. The rest was missing\ndata, not noisy data: the far player usually isn't in the free\ntracker's output near contact. No cleanup invents a player the tracker\nnever saw.
\nThat earned the right to spend money: SAM 3 — the promptable AI that\ntracked this project's first ball, back in\ncv-02 — re-found both players in the\nRoland-Garros match only, prompted from the free tracker's own boxes,\nfor about $12. Shot-call accuracy went from 59% to 67%, mostly\noutlines that had been missing — 21 calls flipped right, one point\ngaining seven alone.
\nIt still didn't move the number that matters — full points a\nvolunteer would sign off with one fix or fewer, stuck at 3 of 59. A\nbetter shot call wasn't holding those points back, and the paid\ntracker inherited the blind spot it was hired to fix: where the free\ntracker drew no usable box, an automatic prompt had nothing to point\nat either. Verdict: the $0 tracker stays default — what every\nsingle-piece upgrade has said since cv-14: improving one ingredient\nnudges the score without changing which points a human would accept.
\nBefore any of that shipped, this project's first fourteen posts'\nworth of scattered experiment scripts — twenty-odd files, four\nnear-identical chart-builders kept in sync by hand — got folded into\none package: one chart-builder, one grading tool, one command-line\ntool, broadcaster differences moved into a settings file per match.\nCondition for merging: it had to reproduce every number this series\nhas published, byte for byte, before anything new shipped on top. It\ndid, down to 138 chart files and four scorecards.
\nWith the free player-finder settled and the tool consolidated, the\nactual flag got built — the scoring formula above, trained on the\npipeline's self-reported effort and graded the\nnever-grades-its-own-homework way.
\nTwo surprises turned up. Aces — serves the returner never touches —\nwere scored as if the machine had refused every shot in the point,\nsince there are no rally shots to refuse. Backwards: an ace is the\ncleanest point in the match, punished for being short. Fixed by\nteaching the formula that no shots after the serve means the point\nended cleanly.
\nStranger: on the Roland-Garros match, the broadcast editor sometimes\ncuts into a rally already in progress. The serve-detector, seeing the\nball cross the net for the first time in the clip, calls that crossing\n\"the serve\" — even though a real serve toss lands 20+ meters past the\nbaseline, and this one launched from inside the court. That's a clip\nthat joined mid-rally, not a serve. Teaching the formula that one tell\ntook held-out reliability from 65% to 84%.
\nOne more result belongs here because it failed: a stricter flag —\n\"sign off without watching the point at all,\" 85% accurate or better.\nTested the same honest way, it landed at 50% precision, a coin flip,\nwilling to call only 1.5% of points that confident. There aren't\nenough graded points to support it: of 135 hand-charted points, only\nabout 1 in 9 already lands within two corrections. That tier is dead,\nreported dead rather than dropped quietly — a confidence system that\nhides its own failures isn't one anyone should trust.
\nWhat did the shipped flag earn, tested the honest way? Read the pooled\nnumber the way you'd read a strong first-serve percentage — not a\nguarantee, but a number worth building on:
\n| match | HIGH flag right, held out | how often HIGH gets called |
|---|---|---|
| the night match | 11 of 11 (100%) | about half the points |
| the control match | 3 of 3 (100%) | about 3 in 5 points |
| the Roland-Garros match | 16 of 19 (84%) | about 1 in 3 points |
| the Wimbledon match | 11 of 11 (100%) | about 1 in 5 points |
| all four matches together | 41 of 44 (93%) | about 1 in 3 points |
HIGH means \"start from this draft\" — 93% of the time, held out, it\nneeds five corrections or fewer. It doesn't mean the draft is already\nright: only about a quarter of HIGH points need two corrections or\nfewer. Missing HIGH means \"expect heavy correction\" — more than half\nof those need six or more, so treat it as a hint, not a finished\nchart. The Roland-Garros match stays the shakiest, not because its\nmath is worse — its disease is missing footage, shots the editor cut\nbefore the camera ever saw them, which no signal can detect from film\nthat doesn't exist.
\nSaid the way a tennis fan would at dinner: a confidence flag only\nmeans something once it's been tested on points it never got to see\nthe answers to first.
\nShot-call sink. Ball-x vs. box-center-x at contact, gated on the\nball reaching the box; biggest post-direction edit sink, 1.71\nedits/point. Audit (178 aligned letters, 4 matches, pre-fix):
outcome sane implaus absent
right 70 31 0
wrong 17 18 0
refused 11 23 8
accuracy 71% 39% --
45% of letters read off a bad box; ceiling ~+26 strict letters of 209\nscored.
Free hygiene (tuned on t3, t1/t2/t4 held out): court-half\nplausibility, x-only teleport rejection, gap interpolation, letter\ngate widened to clip-typical body height. Net: 114/209 → 117/209.\nThree other fixes were built, measured, and removed (not detailed\nfurther here).
SAM-3 buy test. t3 = Roland Garros 2023 final (Djokovic–Ruud,\nclay), 59 clips; prompts auto-derived from the bgsub boxes, ~$12\nall-in. Same eval, same hygiene:
t3 letters bgsub+hygiene SAM-3
strict positional 67/114 (59%) 76/114 (67%)
21 gains, 15 of them refusals turned right (t3_point_25 alone +7).\nAcceptance stays 3/59. Default stays bgsub at $0.
Consolidation. courtvision/ package: one chart assembler, one\neval, one CLI; per-broadcaster differences moved into\ndata/matches/<id>.yaml. Gate: 138 chart CSVs byte-for-byte, 4\nscorecards byte-identical, acceptance 7/135 untouched. experiments/\nfrozen as history. One command: courtvision draft t3 → chart CSVs →\nconfidence per point → outputs/t3/export/t3_mcp_draft.csv.
Confidence layer. Per-point signals — serve commit and stance\nmargin, striker-chain conflicts, ball coverage and holes, refusal\nfractions, the direction signal-tier ladder, crossings-vs-shots\nconsistency — feed a small logistic regression (numpy), calibrated\nleave-one-match-out (LOMO) against token edit distance: fit on 3\nmatches, score the 4th, rotate.
Aces bug: a 100%-refusal charge for zero rally letters to refuse.\nMid-rally-cut signature named t3/clay in the codebase.
Failed tier target: ≤2 token edits at ≥85% precision; exact base rate\n11.1% (≤2-edit) of the 135 calibration points.
LOMO (held out) high prec (≤5 edits) coverage low ≤5 rate
t1 night 11/11 (100%) 50.0% 27%
t2 ctrl 3/3 (100%) 60.0% 50%
t3 clay 16/19 (84%) 32.2% 43%
t4 grass 11/11 (100%) 22.4% 50%
pooled 41/44 (93%) 32.6% 44%
Export. One CSV, Match Charting Project points schema (match,\npoint number, score state, server), machine string in the 1st-serve\ncolumn, plus confidence, conf_p, clip name, jump-to timestamp\n(serve_s). Across 4 matches: 138 draft points, 46 flagged high.
Precisely stated promise: given a match with a fitted homography\nand transcribed score-bug alignment, a draft for every tracked point\nat $0 marginal cost, HIGH on ~1/3 at 93% held-out reliability. Not\npromised: a HIGH point is right, a LOW point is chartable without\nvideo, or anything about rallies cut before broadcast. North star:\nacceptance ≤1 token edit = 5.2% (7/135), ≤5-edit curve = 57.0%.
None of this moves the project's real ceiling, unchanged since cv-14:\nonly 5.2% of drafted points — about 1 in 20 — are an exact-or-one-fix\nmatch to a human chart, and even counting anything within five\ncorrections as close enough, that's still just 57%. The ceiling isn't\nstuck on any piece of the pipeline anymore — it's stuck on the answer\nkey, only 135 hand-charted points, barely 1 in 9 of them already\nwithin two corrections. More matches are already being lined up, and\nthe moment they land, every number here gets recalculated against a\nbigger, harder answer key.
\nSession cost: about $12 — the SAM test, under a one-time $30 limit —\nthen $0.00 for everything built after it. Project total: about $16.
", "url": "https://optic-yellow.com/blog/the-draft-grades-itself", "title": "The Draft Grades Itself", "summary": "A confidence flag tells a volunteer chart-corrector which draft points to trust — and this post checks whether the flag itself can be trusted.", "date_modified": "2026-07-10T02:00:00.000Z", "author": { "name": "Trevor McCormick", "url": "https://github.com/trevormccormick" }, "tags": [ "Devlog", "Computer Vision" ] }, { "id": "https://optic-yellow.com/blog/within-one-token-of-a-human", "content_html": "That clip up top: an ad-point rally from the Djokovic–Ruud Roland Garros\nfinal, just under six seconds of video, with the machine's chart building\nitself token by token above the line a human volunteer wrote for the same\npoint. Four pieces light up green, one after another, because the machine\nand the human agree on every one of them: a second serve landing in the\nbody of the box, then three backhands in a row — two to the same spot,\nthen one that breaks the other way. Then the last ball dies in the net.\nThe human's shorthand for that is the letter n; the machine's chart\nshows a different mark instead, *, so that last piece glows amber — one\ndisagreement. Under the rule this post is about, one disagreement is\nstill close enough to count: the point is accepted. The video's\nclosing screen turns the whole story into four bars.
Here's the question this post is finally built to answer: if you handed\nthe machine's chart of a point to the volunteer who charted that same\npoint by hand, how many things would she have to fix before she'd sign\nher name to it?
\nPreviously: last time (cv-13), the\nsoftware learned to read the small scoreboard graphic in the corner of\nthe screen. That graphic stays pixel-for-pixel identical for the whole\npoint and changes only the instant a point ends, so reading it let the\npipeline do two things: glue video clips back together into whole points,\nand look up the exact matching point in a human volunteer's hand-charted\nrecord of the same match — the ground truth\nthis whole project has been grading itself against since\ncv-10. That match-up is what makes this\npost possible: for the first time, every point the machine drafts can be\nchecked, one for one, against what a real person wrote down for that\nexact point. Before today, the software also had a specific way of\nfinding hits and bounces — watching for the moment the ball's path bends\nbackward on itself, the court doesn't return\nserve, from\ncv-04 — a rule that mostly worked but\nquietly missed shots hit far from the camera.
\nThink about how a coach would grade a rookie scout's notes on a match.\nThe scout writes down what happened, point by point, in shorthand. The\ncoach already has her own notes on the same match. She doesn't grade\nshot by shot — she reads the scout's line for a point next to her own\nline for that same point and asks one blunt question: how many words\nwould I have to cross out, change, or add to turn the scout's version\ninto mine?
\nThat's exactly what this post's headline number measures, except the\n\"shorthand\" is real tennis charting notation and the \"coach\" is a\nvolunteer from the Match Charting Project — the same ground\ntruth this project has been grading itself\nagainst. Write a chart of one point as a list of pieces, or tokens: the\nserve and where it landed, then each shot's stroke and direction, then\nhow the point ended. Do that for the machine's version and the human's\nversion of the same point, then count how many of those pieces would\nneed to be changed, added, or removed to turn one list into the other.\nThat count is called token edit distance,\nand it's now the north-star number for the whole project — the honest\nanswer to \"would a human sign off on this.\"
\nA point counts as accepted when that count is one or less — the kind\nof chart a charter would wave through after fixing at most one thing.\nAnd there's a rule that keeps the machine honest: if it isn't sure and\nwrites a \"no idea\" placeholder instead of guessing, that placeholder\nstill has to be fixed later, so it costs exactly as much as getting the\nanswer wrong. Shrugging is not a way to dodge a mistake.
\nBefore any of that grading could happen, the event detector needed work.\nThe old approach hunted for a kink in the ball's flight path — the\nmoment its direction snaps backward, which is how a hit differs from a\nbounce — but a kink from a shot on the far\nside of the court, seen from a raised broadcast camera, can shrink to a\nsingle wobbling pixel that's easy to miss entirely. So the detector got\nrebuilt around a blunter, more reliable piece of tennis sense: a rally in\nprogress sends the ball across the net every single shot. Instead of\nsquinting at every wobble, the software now counts how many times the\nball crosses the net and adds one for the serve — and that crude count\nalone, before any fine-tuning at all, already beat the entire old\nwobble-hunting detector at guessing how long a rally ran. The rebuilt\ndetector went on to sharply improve how often it read stroke letters\ncorrectly, though one older test clip actually got worse in the process —\na small, honestly reported step backward, not swept under the rug\n(exact numbers in the appendix).
\nWith the rebuilt detector in place and every point matched to its human\noriginal, the project could finally run the grading question for real —\nacross four matches, three court surfaces, both the men's and women's\ntours: how many of 135 points would a volunteer charter sign off on with\nat most one fix?
\nThe answer: 3 points out of 135 — 2.2%. Before the detector rebuild,\nusing the old event logic, that number was zero. Two percent isn't a\ngood score. But it's the first honest score — a real distance to the\ntop of the mountain, measured in a unit a volunteer charter actually\ncares about, instead of a pile of separate stats that could all be\nquietly improving while the finished chart is still wrong.
\nA 2.2% pass rate means the typical point needed about seven fixes, on\naverage, to become the human's version — so the project broke that\naverage down by fix type to see where the effort was going. Missing\nshots — a shot the human charted that the machine's chart skipped\nentirely — were the single biggest source. But close behind, and the\nmost fixable-looking, was shot direction: which way a shot traveled,\nnot just what stroke it was. Getting the direction digit wrong, whether\nalone or combined with the wrong stroke letter, accounted for 39% of\nevery fix an editor would have to make — the largest chunk of the whole\nedit budget. And the direction guesses weren't just wrong often: the\nmachine only ventured a direction guess on 70% of shots, and when it\ndid, it was right just 48% of the time — barely above the 33% you'd\nexpect from picking one of three directions at random.
\nThat's a big enough number to be worth chasing, and the cause turned out\nto have nothing to do with cameras or vision at all — it was a\ndefinition problem. The Match Charting Project's own instructions define\none shot direction as \"a right-hander's forehand side\" and add, in\nparentheses, \"a lefty's backhand.\" Read carelessly, that sounds like\ndirection depends on which hand hit the ball. It doesn't: the\nparenthetical is the tell. Direction is defined relative to a fixed side\nof the court — specifically, the half that's receiving the shot — and\nit only happens to line up with \"forehand side\" for one particular kind\nof player. The pipeline's code, though, had been computing direction\nusing fixed court thirds, as if the same physical side of the court\nalways meant the same direction: correct by pure coincidence for a shot\ngoing to the far end, backwards for a shot going to the near end. It was\na compass mirrored for half the court the whole time. Once the\ndefinition got fixed — direction read relative to which half was\nreceiving, instead of a fixed physical side — direction accuracy jumped\nfrom 48% to 75%, the machine started venturing a guess on 97% of shots\ninstead of 70%, and overall acceptance nearly doubled: from 3 points out\nof 135 to 7, from 2.2% to 5.2%.
\nFour matches, three surfaces, both tours — here's how many of each\nmatch's points a human charter would sign off on with at most one fix,\nafter the compass fix went in:
\n| Match | Points compared | Accepted (≤1 fix) |
|---|---|---|
| Night match | 22 | 3 |
| Control match | 5 | 1 |
| Clay — Roland Garros final | 59 | 3 |
| Grass — Wimbledon final | 49 | 0 |
| All four, combined | 135 | 7 (5.2%) |
That's up from 3 of 135 (2.2%) before the direction fix. The honest low\npoint on the board is Wimbledon: zero points accepted, because its\nproblem isn't a wrong direction digit — it's shots that shouldn't be in\nthe chart at all, or shots missing from it (more in the appendix).
\nThe takeaway a tennis fan could repeat at dinner: what matters isn't how\nmany little stats are climbing, it's how many marks a red pen still\nneeds — and fixing one wrong assumption about which way \"direction\"\npoints nearly doubled how many points needed no more than one.
\nNet-crossing detector rebuild. Two passes. The serve detector went\nball-first — the serve is the track's first sustained net crossing, its\ndirection names the server's end — and clay server-end accuracy went\n7/58 to 48/59 (12% → 81%), grass 17/49 to 38/49, with the honest asterisk\nthat grass refuted the ball signal outright (white ball, white lines:\n26/49, a coin flip), so the grass detector kept the player-based signal\nand only the search window learned to slide. Then the coda pass went\nhunting for a per-event tell that separates a live rally from\nWimbledon's dead-ball aftermath, and refuted every candidate — landing\npositions, walking speed, coverage, all byte-similar between the two —\nuntil the rule that held was sequence-level tennis: a live rally sends\nthe ball across the net every shot.
That sentence turned out not to be a truncation rule; it was the\ndetector, inverted. The old design — M2-era cusp classification (the\nwobble-by-wobble hit/bounce guesser from cv-04), tuned on sparse SAM\ntracks (the ball-tracking AI's own thin, gap-ridden paths) — inspected\nevery wobble in the ball track and asked \"hit or bounce?\" Diagnosing all\n57 rally-count misses (not a sample) found the far-end miss systematic:\na far hit at 250+ px is a ~1 px/frame wiggle no near-tuned gate can feel,\nbut its flight crosses the net at full projected speed. The keystone\nmeasurement, taken before any design was written: raw net-crossing count
So v5 stops classifying wobbles. Net crossings are the rally skeleton:\nthey partition the point, each partition contains exactly one hit, and\nthe cusp machinery is demoted from counting shots to locating the hit\ninside its slot. Constants tuned on t3 (the clay match) only, everything\nelse scored untouched — the tuned tree went 42% → 61% on rally length\nwhile letters (aligned) tripled its base, 19/24 → 65/85. The named\nregression: t1 (the night match) fell 16/22 → 13/22. The skeleton needs\ncrossings, and the night reel's thin low-recall tracks amputate rallies\nthe old cusp counter used to pad out. A design that leans on the net\npays the price wherever the track can't see the ball cross it.
Edit-distance mechanics. Tokenize both strings as\n[serve+zone][letter+direction]*[ending] and accept a point iff the\nmachine's list is within one token edit of the human's — strict\nequality, a ? matches nothing, so refusing to commit costs exactly\nwhat being wrong costs. First reading, all four matches: 3/135, 2.2%\n(before v5 it was 0/135). Mean token edits per point: 7.18 across n=135,\nbacktraced to an alignment and binned:
edit category (mean/point, n=135) overall
deletion (shot never charted) 1.95
sub: letter AND direction wrong 1.61
sub: direction digit only 1.20
sub: ending token 0.70
insertion (phantom shot) 0.54
Direction digits sit in two of those bins and add up to 2.81 edits per\npoint — 39% of the entire edit budget — attempted on only 70% of shots\nand 48% right when attempted, barely above the 33% floor. Meanwhile the\neffort curve says the skeleton v5 just rebuilt is no longer the wall:\nstructure alone (shot count + letters) would already accept 23.7% of\npoints at the same ≤1 bar, per the original decomposition note (this\nfigure is carried over as originally reported alongside the ≤3-edit\ncurve below; the two are not obviously the same slice of the data, and\nthat ambiguity is preserved rather than resolved here). The headroom\ntable, fixing one component to ground truth at a time:
counterfactual accept ≤1 mean dist
baseline (v5 as charted) 3/135 ( 2.2%) 7.18
directions perfect (all shots) 9/135 ( 6.7%) 5.88
endings perfect 7/135 ( 5.2%) 6.47
letters + directions 21/135 (15.6%) 3.93
letters + dirs + endings 53/135 (39.3%) 3.23
all components (structure residual) 59/135 (43.7%) 2.93
No single fix rescues the metric — the mean point is wrong on several\naxes at once — but the compounding is steep and ordered, directions are\nthe biggest single lever, and the tell is in the fine print: perfecting\nonly the attempted directions is worth almost nothing. The sparsity\nand the accuracy are the same disease.
The mirrored compass. MCP's instructions define direction 1 as \"a\nright-hander's forehand side (a lefty's backhand)\" — the parenthetical\nis the tell. That sentence names a fixed side of the receiving half,\nmirrored by which end is receiving, not flipped by who's holding the\nracquet. zone() had been coding absolute court thirds: correct for a\nfar right-handed receiver by coincidence, mirrored for every near\nreceiver. Half of \"barely above the random floor\" was a sign error.\nEvery plausible mapping went into a calibration table before any code,\nand the both-lefty t1 match is what adjudicated the handedness question:\nreceiver-end mirror without a handedness flip scores 11/24 on the\nlefties; adding the flip drops it to 5/24 — the naive reading of the\nspec loses to its own parenthetical.
On top of the fixed compass, a measured signal ladder gives every shot a\ndirection now — the workhorse is the receiver's next contact (where the\nball was received is where this shot went; 77% right, 81% coverage on\nthe dev match), backed by the near-half bounce (86%) and the v5 skeleton\nmoonlighting as a crossing-plus-slope extrapolator. Even refusal policy\nwas an experiment, not a principle: vetoing on disagreement kept\nprecision flat while shedding net-right tokens, and since the metric\nprices a shrug the same as an error, commit-always won.
Direction attempts went 70% → 97%, accuracy when attempted 48% → 75% —\nheld-out matches gaining right alongside the tuned one — and acceptance\nmoved 3/135 → 7/135, with the ≤5-edit curve jumping 41.5% → 57.0%. Every\nother metric on all four scorecards is byte-identical, exactly the claim\na direction-only change should be able to make. The meta-lesson joins\nthe permanent file: when a component scores near the random floor, check\nthe semantics before rebuilding the perception. A 27-point accuracy jump\nwas sitting in a coordinate convention the whole time.
Full scorecard. Four matches, three surfaces, both tours, 168 points\naligned to human charts as originally reported for this section (a\ndifferent count from the n=135 used above for the token-edit\ndecomposition — both numbers are carried over as originally stated):
metric t1 night t2 ctrl t3 clay t4 grass
server end 10/22 2/5 48/59 (81%) 37/49 (76%)
rally length ±1 13/22 5/5 36/59 (61%) 28/49 (57%)
serve zone 11/12 1/3 8/17 15/32
letters (aligned) 9/11 12/12 65/85 (76%) 15/28
ending type 9/16 2/3 19/42 9/33
acceptance ≤1 edit 3/22 1/5 3/59 0/49
acceptance at ≤ k token edits (n=135):
≤1: 2.2 → 5.2% ≤2: 6.7 → 11.1% ≤3: 14.8 → 23.7% ≤5: 41.5 → 57.0%
The honest reads stay on the record: t4 (grass, Wimbledon) is 0/49\nstrict — its disease is structure (1.24 phantom insertions plus\ndeletions per point), and no direction-digit fix repairs a token that\nshouldn't exist in the first place. t3's (clay, Roland Garros) missing\nshots are mostly the broadcast's fault, not the pipeline's — the RG\neditor cuts into rallies mid-point — and structure caps the\nall-components ceiling at 44%. The re-run decomposition already names\nthe next levers in order: structure, letters, endings — one token per\npoint, 30% right, as originally reported.
Two percent was the first honest look at how far this project has to\nclimb. Five percent, after fixing the compass, is the second look. The\nreal change isn't the number itself — it's that the climb now has\narithmetic behind it instead of a guess.
\nSession cost: $0.00. Project total so far: about $4.15 of a $9 budget.
", "url": "https://optic-yellow.com/blog/within-one-token-of-a-human", "title": "Within One Token of a Human", "summary": "The project finally gets an honest report card: how many fixes would turn the machine's chart into a human's? Today's first answer: 2.2%.", "date_modified": "2026-07-10T01:45:00.000Z", "author": { "name": "Trevor McCormick", "url": "https://github.com/trevormccormick" }, "tags": [ "Devlog", "Computer Vision" ] }, { "id": "https://optic-yellow.com/blog/the-score-bug-is-the-point-id", "content_html": "That clip up top: a 21-second, sixteen-shot Djokovic–Ruud rally from\nthe Roland Garros final. The old segment-finder had chopped it into\nfour broken pieces — the tick marks on the timeline are where those\nold cuts fell, and each piece charted only 1 to 3 shots alone. This\npost's method proved all four belong to one point, glued them back\ntogether, and the same shot-detector read the reunited rally as 12 of\n16 shots correctly.
\nEvery TV director cuts away sometimes — a reaction shot, a replay, a\ndifferent angle. So how do you prove, from footage alone, that two\nclips are really the same point? Two test matches weren't enough, so\nthis session added harder ones: the 2023 Roland Garros final on clay,\nand the 2024 Wimbledon final on grass. Four matches now, three\nsurfaces, both tours, about 108 new points checked against a human's\nchart, for zero dollars. What broke wasn't one narrow piece — it broke\nthe assumption under every number on the board, and the fix had been\non screen all along.
\nPreviously: cv-10 gave the project its\nfirst honest report card, grading its draft charts against a match a\nhuman had already charted by hand. cv-12\nthen swapped in cheaper, more reliable ball-tracking software — and\nuncovered an accident: the old tracker's habit of losing the ball\nwhen a rally ended had secretly been flagging \"this point is over.\"\nThe new tracker never loses the ball, so that free signal vanished,\nleaving the project needing a real point-boundary detector.\ncv-06 had already learned a broadcast is\nmostly not rally footage — finding the stretches that even show the\ncourt took its own detective work.
\nPicture a broadcast with the sound off. Almost everything keeps\nchanging — players move, the camera pans, the crowd shifts — except\nthe scoreboard graphic in the corner. It isn't part of the court; it's\nstamped on top of the picture like a watermark, so a pan can't nudge\nit. And it obeys a rule nothing else on screen does: frozen, pixel for\npixel, for the whole length of a point, changing at exactly one\nmoment — when the point ends.
\nThat discipline is what this series calls the score bug as\nmetadata (more in the primer): the graphic,\njust by sitting there frozen, quietly carries the match's structure.\nTwo things fall out of it for free. It's a point ID — two chunks of\nvideo showing the identical frozen scoreboard are fragments of the\nsame point, however the editor cut between them, and can be safely\nstitched together. And it's a join key — a shared label matching this\nclip to its point in the human charter's own record.
\nThe clay final (Djokovic vs. Ruud) broke the court-mapping recipe that\nworked fine on hard courts. Turning the TV picture into a court\ndiagram needs the homography — the\nscreen-to-court conversion recipe — and its usual tricks assume a blue\nhard court with crisp white lines. Clay has neither: the \"find the\nblue court\" trick found nothing, and the trick that erases players by\nblending frames together smeared the faint lines into the dirt\ninstead. A clay-specific rebuild (appendix has the method) got it\nworking again, accurate to a fraction of a pixel.
\nThe grass final (Krejcikova vs. Paolini) broke the check on the\nrecipe instead: the software found a court mapping whose lines lined\nup neatly with each other — and it was wrong, like an umpire\nconfidently announcing a plausible score that isn't the real one.\nThat's an impostor fit. The fix: instead of \"do the few lines this\nmapping used line up?\", ask \"do all nine of the court's real lines\nland where this mapping says they should?\" A wrong mapping explains a\nfew lines and leaves the rest floating; the real one explains all\nnine. (One more check earned its keep on this match too, following\ncv-11 — the appendix has it.)
\nBoth matches mapped, the draft charts still came back mediocre — for a\nreason underneath every number, not inside any one of them. The\npipeline assumed one unbroken stretch of court-view video (a\nsegment) equals one point. That breaks in opposite directions,\ndepending on who's holding the editing scissors.
\nRoland Garros cuts away mid-rally, chopping one point across several\nsegments: the draft charted just 3.9 shots per point here against the\nhuman charter's real 8.9. The rally was there; the video just told it\nin pieces. Wimbledon does the opposite, almost never cutting between\npoints, so one fifteen-second stretch can hold two or three back to\nback — the draft overcounted there, 8.1 shots per point against the\nreal 7.2, stringing separate points into one fake rally.
\nThe two earlier test matches had simply gotten lucky: their\nbroadcaster cut almost exactly once per point, splitting the match for\nfree — the same kind of accident as the old tracker's habit of losing\nthe ball, doing a job nobody built it to do.
\nThis is the gap the score bug closes. Two chunks of video with the\nidentical frozen scoreboard, however the editor cut between them, are\nthe same point — merge them. One chunk where it changes partway\nthrough is secretly two points — split it there. Read the sequence of\nchanges and you get the real point-by-point structure, regardless of\nwhere the editor cut. It held with a wide, safe margin between \"same\npoint\" and \"different point\" readings, and a handful of broadcaster\nquirks that tried to break it — a scoreboard that grows after every\nset, a tiebreak banner that shoves it sideways, a fade-in that could\nfake a change that never happened — all got caught by looking closely\nat the pixels (appendix has the numbers).
\nOne theory got tested and killed outright, worth saying since it\ncontradicts what I'd guessed going in: I suspected slow-motion replays\nwere sneaking duplicate points into the count. They aren't. Every\nsuspicious duplicate score turned out to be fragments of one rally, or\na genuinely different point that passed through the same score — deuce\nrecurs, and the scoreboard alone can't tell those visits apart, only\ntheir order can. The last suspect was cleared by physics: the ball's\nown falling motion matched real-time gravity, and a slow-motion replay\nwould fall roughly four times slower.
\nWith segments stitched into points correctly, the clips lined up\nagainst the human record almost perfectly.
\nMatched row for row against the human-charted\nrecord — the independent answer key this project grades itself\nagainst:
\n| What was measured | Before | After |
|---|---|---|
| Clips matched to their correct point, clay | 59 of 68 | 59 of 59 — every one |
| Clips matched to their correct point, grass | 36 of 48 | 49 of 49 — every one |
| Rally length guessed within one shot, clay | 34% | 41% |
| Rally length guessed within one shot, grass | 15 of 36 (about 42%) | 21 of 49 (about 43%) |
| Serve zone guessed correctly, grass | 1 of 4 | 8 of 10 |
| Which player served, clay (honest step back) | 10 of 59 | 7 of 58 |
| How the point ended, clay (honest step back) | 11 of 34 | 9 of 30 |
The two rows that went down are real, worth naming rather than hiding:\nmerged clips can start mid-rally, so the software falls back to a\nweaker clue (simple alternation) for who served and how it ended. A\nfair trade for getting the boundaries right, but a real cost. Where\nthe merges had footage to work with, the payoff showed: the 16-shot\nrally up top went from four broken pieces to one point charted 12 of\n16 correctly, and another clay point came back a perfect 12 of 12.
\nTwo problems stay open, neither a scoreboard problem anymore. On clay,\nthe editor sometimes cuts into a rally already in progress, so some\nearly shots were never filmed — the scoreboard proves the clip is one\npoint, but can't invent missing footage. On grass, 19 clips still\ncount as two points when they're really one, because Wimbledon's dead\ntime sits inside that same frozen reading (mechanism in the\nappendix). Both are chart-reading problems now, not video-editing\nones — and they're next.
\nThe least glamorous graphic on the screen — a tiny scoreboard nobody\nwatches on purpose — is the one witness every broadcaster in the world\nagrees to keep rolling at all times, which makes it the most\ntrustworthy thing in the picture.
\nt3, clay (Djokovic–Ruud, RG world feed, 34 min highlights): M1\nhomography died three ways — \"blue court hull\" has no blue, and the\nclay-hue mask swallows the frame (crowd skin ≈ clay hue); median plate\nsmears lines into dirt (16 px pan, white-on-orange loses to\nwhite-on-blue); far baseline buried (V200 on V~195 clay). Rebuild:\nECC-stabilized plates, thin-bright tophat filter, verticals labeled by\nsymmetry (courtside Perrier crates give 20 Hough segments inside the\nhull; junk is asymmetric, lines aren't), fit basis swapping the buried\nbaseline for the far service line. Result: sidelines 0.4/0.5 px,\ncenter line 0.2 px.
t4, grass/WTA (Krejcikova–Paolini, Wimbledon feed): a\nwrong-but-self-consistent line assignment fit its own intersections to\nrms 1.7 px. Fix: score candidates by how well all nine model lines\nland on the observed mask — impostors strand the lines they didn't\nuse.
Handedness: MCP's metadata listed Vondrousova, a well-known\nleft-hander, as right-handed. Post-cv-11 policy: never trust the\nchart's \"hand\" column, look it up independently.
Segment note: some RG fragments in the 3.9-vs-8.9 split were\nwritten off by the staging pass as replays.
Score bug validation: same-point crops differ by ≤0.005\nchanged-pixel fraction, different-score crops by ≥0.033 — a 6x margin.\nTraps: the RG bug grows a column per set (digits slide ~27 px right);\nthe set-1 tiebreak banner drops it 14 px and drifts; a reference frozen\non a fade-in frame lets compression noise fake a change 100 frames\nlater. Fix: frozen reference per plateau, anchored on the median of a\nsettling window.
Replay hypothesis, refuted: every duplicate-score group was\nfragments of one rally or distinct deuce points (a 40-40 plateau sits\nbetween \"duplicate\" 40-AD clips: three deuce cycles; order carries\nwhat value alone can't). Cadence suspect acquitted: WASB-track ball\ngravity is unimodal across all 116 clips; 2x slow-mo would sit ~4x\nlow.
Remaining gaps, mechanism: t4's 19 over-counted clips fail because\nWimbledon's dead time lives inside the point's own plateau, the dead\nball stays inside the court-y envelope, and the freeze-#3 gate can't\ntouch it — the t1 dead-ball coda in a new costume.
Session cost: $0.00. Project total: about $4.15 of a $9 budget. The\nbenchmark is now four matches, three surfaces, both tours — and every\npoint ID in it comes from the one witness every broadcaster agrees to\nkeep on screen.
", "url": "https://optic-yellow.com/blog/the-score-bug-is-the-point-id", "title": "The Score Bug Is the Point ID", "summary": "A TV scoreboard graphic nobody notices turns out to be the only reliable way to tell where one point ends and the next begins.", "date_modified": "2026-07-10T01:30:00.000Z", "author": { "name": "Trevor McCormick", "url": "https://github.com/trevormccormick" }, "tags": [ "Devlog", "Computer Vision" ] }, { "id": "https://optic-yellow.com/blog/the-zero-dollar-specialist", "content_html": "That clip up top: the same Federer rally, tracked two different ways.\nLeft, the old general-purpose tracker's trail of yellow dots — a\ncomet's tail — thins out and dies partway through; it wrote down only\nsix of twelve shots. Right, a tiny tennis-only specialist's trail stays\nsolid to the last shot. Same clip, same notation software underneath\nboth — the only thing that changed is who's watching the ball. The\nspecialist charted all twelve shots, and got every one of the seven\nshot symbols it committed to right.
\nBefore spending another dollar chasing a tennis ball across the screen,\nI stopped and asked a simpler question: has somebody already solved\nthis?
\nHere's where things stood. The pipeline had just been graded for the\nfirst time against a real match a human had already hand-charted — an\nout-of-sample test, rules frozen before\nanyone saw the answers, on a night match between Nadal and Shapovalov.\nThat test caught a mirror-image\nbug — forehand and backhand swapped, because the software assumed a\nright-handed player against two lefties — fixed with a tennis fact, not\nnew vision code. A second test was already promised: a sunlit day\nmatch, Federer against Haase. Every bit of ball-tracking so far had\nbeen done by SAM 3, a general-purpose \"find that object\" AI that needed\na box drawn around the ball by hand, and that charged a small fee per\nclip.
\nPicture two people you could hire to watch the ball for you. One is a\ntouring pro: capable of almost anything, but before every point you\nhave to walk over and tell them exactly what to watch. The other has\nhad one job their whole career — they've stood on the same line,\nwatching where the ball lands relative to it, thousands of times a day\n— and never once needed a pep talk first.
\nSAM 3 is the touring pro. WASB, the program introduced in this post, is\nthe line judge: about the size of a phone photo, and it has only ever\nlooked at tennis balls. Show it a photograph and it lights up how\nlikely the ball is to be at each spot on screen, the way a heat map\ndoes, and picks the brightest spot, fresh, thirty times a second, no\nmemory of the last one. That's a tiny specialist versus a huge,\npromptable generalist — more at Court Vision in Plain\nEnglish — and it's what this post turns on. One\nside effect matters later: with no memory, WASB can't lose the ball\nthe way SAM can. SAM carries the ball forward frame to frame —\ntracking, in the technical sense — and\na tracker that loses its grip stays lost. WASB never had a grip to\nlose.
\nAn evening of careful research, checking every claim against more than\none source, settled two things. First: finding the ball in a tennis\nbroadcast is already a solved, free, open problem — a specialist almost\nexactly like WASB is out there, pretrained and downloadable today. Our\nown SAM-based tracking had been reinventing that wheel, worse, for a\nsmall fee on every clip.
\nSecond, and this matters more: every system like that stops before\nthe notation. None of them read a broadcast and write out who hit what,\nwhere, in the shorthand a human charter would recognize. That gap — not\nthe pixels, the grammar — is the real target here. Somebody wants it\nfilled, too: the volunteer project supplying our human answer keys runs\ntoday on just 32 unpaid, part-time people, and its founder has publicly\nasked for more help.
\nOne more gap we could close ourselves: nobody had published a\nhead-to-head test of a generalist against a specialist on a ball this\nsmall and fast. So we ran one.
\nWe wired the specialist's ready-made weights into the pipeline and left\nthe notation-writing software untouched, so any difference in the\noutput could only be the tracker's doing. It won almost everywhere it\nmattered — numbers below. It also did something SAM structurally\ncouldn't: thirteen night-match clips never got a usable starting box for\nSAM, so SAM tracked zero frames of them. WASB needs no starting box at\nall — it tracked all thirteen anyway, 73–97% of frames each, no\nclicking, no cost.
\nThen something broke.
\nOn the night match, the number this project cares about most — shots\ncounted in a point, checked against the human's count — got worse\nwith the better tracker. Before the swap it had seven of eleven points\nright; after, only five. Every miss counted too many shots, never too\nfew.
\nI went back to the photographs. Match point: the ball resting at the\nframe's bottom edge, the crowd already on its feet,\nmid-standing-ovation. A dead ball bouncing quietly in the corner while\nboth players stood around between points. A ball drifting through the\ncrowd above the back fence, long after the point ended. In every case,\nthe point was over. The ball-finder was not.
\nHere's the mechanism. SAM tracks — it locks onto one ball and remembers\nit. When a rally ended and the ball sailed out of frame or vanished\ninto the crowd, SAM's memory broke: its trail of dots died, right\nthere, at the end of the point. Nobody designed that as a signal, but\nthe software downstream — the part that turns ball positions into a\nchart — had quietly been leaning on it anyway, never told in so many\nwords when a point was over; it had just been reading SAM's dying trail\nas the whistle. WASB never gets lost, because it never had hold of the\nball to begin with — it just keeps finding it, dead ball and standing\ncrowd and all, and the chart-writer kept faithfully logging \"shots\" out\nof a point already finished. A bug had secretly been doing somebody\nelse's job.
\nThe first fix I tried didn't work: a ball still in play after a real\nhit moves fast, a dead ball bouncing after a point should be slow, and\nthat gap looked clean — until the day match's real serves turned out\njust as fast as the fake bounces, and the line I'd drawn cut through\nreal shots too.
\nWhat worked was tennis, not more tuning: a ball still part of the point\nnever leaves the court's playable area — the same screen-to-court\nconversion, the homography, used elsewhere in\nthis project. Every phantom shot sat outside that area; every real\nshot, across both matches, stayed inside it, serves excepted (a legal\ntoss goes up and out on purpose). One rule, one exception, and the\nphantom shots disappeared. Checked again against the human chart, the\nnight match's shot-count came back to ten of eleven — the best this\nproject has scored — while the day match's numbers didn't move by a\nsingle shot: proof the fix cleaned up the one problem it targeted and\ntouched nothing else.
\n| What we measured | SAM, the generalist | WASB, the specialist |
|---|---|---|
| Ball found on screen — night match | found it 82% of the time | found it 87% of the time |
| Ball found on screen — day match, bright glare | found it 74% of the time | found it 88% of the time |
| Points counted correctly — day match, 5 points checked | 1 of 5 | 4 of 5 |
| Cost to track a match's worth of clips | a small fee per clip | $0 |
Good, here, is the specialist matching or beating the generalist\neverywhere that counts, at zero cost — which is what happened, except\non a handful of easy, brightly lit clips where the generalist still\nedged ahead by a few points.
\nSometimes the scariest bugs in a pipeline aren't the ones that break\nsomething — they're the ones that had quietly been holding something\nelse together.
\nLandscape review: 23 sources fetched, top 25 claims adversarially\nverified, 25 survived. Specialist = WASB-SBDT (BMVC 2023, NTT, MIT\nlicense), pretrained tennis weights, 6 MB, F1 94–96 on the\nbroadcast-tennis benchmark. TennisProject replicates our classical\nstack (ball, court keypoints, homography, players, bounces). No\nproduct/paper/repo found ingests broadcast video and emits a symbolic\nshot-by-shot chart: SwingVision wants your own phone footage, TennisViz\nconsumes Hawk-Eye's tracking feed, Hawk-Eye runs ~EUR 60–70k/court/week.\nMatch Charting Project: ~17,800 matches, ~a quarter of tour matches,\nSackmann's January post pleads for more charters.
A/B, project shorthand: night match = t1, day match = t2. ~200-line\ndriver (hard-asserts CUDA; ours runs on MPS, 15–20 fps), frozen chart\nloop untouched. SAM's disaster clips evaporate under the swap\n(55%→93%, 55%→87%); its only wins are 3–7 points on easy floodlit\nclips.
Regression: t1 rally length 7/11 → 5/11, all over-counts. Failed fix:\nvelocity magnitude — coda bounces swing 109–210 px/frame vs real hits'\n8–33, a beautiful gap until t2's serves landed inside it. Fix that\nheld: one positional gate, serve-window exemption, three new\nconstants, frozen-era constants untouched — t1 lands at 10/11, t2\nbyte-identical before/after. All 60 dev-reel clips retracked at $0,\nincluding 37 SAM never touched.
Session cost: $0.00. Project total: ~$4.15 of $9 — and nothing ahead of\nus charges by the frame anymore.
", "url": "https://optic-yellow.com/blog/the-zero-dollar-specialist", "title": "The $0 Specialist and the Load-Bearing Bug", "summary": "A free specialist replaces an expensive all-purpose tracker — and exposes that its rival's biggest flaw was secretly marking the end of every point.", "date_modified": "2026-07-10T01:15:00.000Z", "author": { "name": "Trevor McCormick", "url": "https://github.com/trevormccormick" }, "tags": [ "Devlog", "Computer Vision" ] }, { "id": "https://optic-yellow.com/blog/t1-right-hardware-wrong-sign", "content_html": "That clip up top is match point of Shapovalov's win over Nadal,\nMontréal 2017 — the point that made his career. Our software's chart\nassembles on the top line; a human volunteer's runs below it. The\nhuman's line ends f+1* — forehand (f), closing in on the net (+), to\na corner (1), a clean winner (). The software's ends with Shapovalov\nflat on his back, most labels still ?, its symbol for \"couldn't\ntell.\" That gap is what this post closes.*
Could software, rules locked exactly as last tuned, watch a match it\nhad never seen and write a chart a tennis fan would call the same\nmatch?
\nTwo posts back, the software learned to guess who hit each\nshot by weighing three signals — did\nthe ball reach that side, the serve call, where the shot landed. Last\npost, a human volunteer's chart of a real\nmatch — Nadal against Shapovalov, at\nnight, in Montréal — became the project's first answer key, ending a\nstretch of calling its own lines: fine for practice, worthless as\nproof. (Why that matters: Court Vision in Plain\nEnglish.) Today's rule, set before a single\nclip played: nothing changes before the scorecard.
\nEvery tennis fan carries this rule without thinking: players swap ends\nafter the first game, then after every odd-numbered game total, and\nevery six points in a tiebreak. Tell a fan the score is four games to\nthree and they'll say, without looking, which end each player stands\nat — arithmetic, not a guess. The software had to learn that same\narithmetic: no coin toss to watch, just a ball arriving from the near\nor far side of the screen, so it computes which end a player stands at\nfrom the score alone.
\nThat arithmetic matters beyond serve calls. When software is wrong in\na way that flips on one fact it never checked, the fault usually isn't\nwhat it's measuring — it's a label it assumed instead of computed.\nCall this the mirrored error: every miss points the same\ndirection, so the tool's eyes work fine and only a naming convention\nflipped — a line judge with perfect eyesight calling \"out\" every time\nshe means \"in.\" You wouldn't trust her calls, but you wouldn't send\nher for glasses.
\nStabilizing the footage, building the empty-court reference photo (the\nmedian-frame trick that erases\nplayers and ball so anything left over stands out), and tracking\nplayers ran on the Montréal footage unchanged. Far-side coverage\nimproved: 100% here versus 96% on the practice reel, because\nfloodlights kill the shadows that had tricked the daytime footage.
\nA new problem showed up — the fourth time something that isn't a\nplayer has fooled the player-finder. This court's ball kids crouch at\nthe net posts, and the code assuming \"the biggest moving blob is the\nplayer\" grabbed them, opening serve-toss windows above a ball kid's\nhead, once above a chair umpire's. The fix is geometry: a server stands at\nthe baseline, so the bootstrap checks each candidate's foot position\nin real court coordinates — the screen-to-court\nconversion the project runs on — and rejects\nanyone standing where a server never would.
\nThe serve detector called a direction confidently on 11 of 25 clips\nhere, down from 40 of 60 on the practice match — a harder test: a\ndifferent match, at night.
\nThe human chart already knows who served every point; changeover\nparity turns that into \"which end that server stood at,\" once you\nknow where the match started. Run it both ways: \"Nadal started far\"\nexplains 11 of 14 confident calls, \"near\" explains only 3. Nadal\nstarted far — a real accuracy figure of 11 of 14, about 79%, up from 6\nof 8 on the practice match, checked by eye instead of a chart.
\nBall tracking still runs behind a spending gate: a human eyeballs a\nproposed box before the project pays to send it to the paid,\nprompt-it-yourself tracker. Of 25 proposed\nboxes, 13 were approved — nine on the first look, three more after the\nball-kid fix, one via a backup rule that just follows whatever's\nmoving fastest. Twelve were sent, $0.85 total, eleven usable; the\ntwelfth, a shaky approval, died twelve frames in. That check is 6 for\n6 across both matches at picking winners. It stays.
\nEleven points, chart locked, graded against the human's:
\n| What was checked | Result | What it means |
|---|---|---|
| Got the server right | 7 of 11 | Whose racket started the point |
| Rally length, within one shot | 7 of 11 | How many shots the point took |
| Serve landing spot in the box | 1 of 2 | Only two points had enough signal to grade |
| Forehand/backhand, exact match | 2 of 11 — about 1 in 5 | The weakest reading on the sheet |
| Forehand/backhand, allowing a mirror | 9 of 11 — better than 4 in 5 | Flip the label and it's the strongest reading |
That last pair is the headline — the best bad result this project has\nproduced. Nadal and Shapovalov are both left-handed; the software\nstill assumes every player is right-handed, a risk flagged and frozen\non purpose so a lefty-heavy match would expose it. A lefty's forehand\nlands where a righty's backhand does, and vice versa — assume the\nwrong hand and the label flips, though the geometry stays right. Nine\nmirrored calls out of eleven means the software found the right side\nof the body at contact almost every time — just the wrong name for it.\nOne line of match configuration — telling it who's left-handed — turns\ntoday's 2 of 11 into roughly 9 of 11 at the next freeze: the cheapest\nseven-point improvement this project will ever buy.
\nThe rest of the misses hold up too. A tie-break rule letting a strong\nserve call override a shaky forehand/backhand vote had rescued one\nconfusing point on the practice match; here it pushed two calls wrong\ninstead. And rally length runs long because some points include a\nfault — a missed first serve — before the point that counts; the human\nchart logs both separately, but the software counted the fault as part\nof the rally. Starting the count from the first confident serve fixes\nboth.
\nWhen every miss points the same direction, you haven't found a\nbroken tool — you've found a fact about the match it never bothered to\ncheck.
\nAn automated system charted eleven points of a tennis match from\nbroadcast video and got graded against a human's chart of those\npoints — the actual thing this project set out to test, now with a\nnumber instead of an impression. The number is bad in the most useful\nway a number can be: the biggest error has a direction, a known cause,\nand a one-line fix.
\nTest-set discipline held — nothing tuned mid-grade. Session cost\n$0.85; project total about $3.65 of the $9 budget. Next: handedness\ngoes into the match configuration, then a control match — Federer\nagainst Haase, two right-handers — answers whether this was ever\nreally about left-handedness at all.
", "url": "https://optic-yellow.com/blog/t1-right-hardware-wrong-sign", "title": "T1: Right Hardware, Wrong Sign", "summary": "Graded against a real human chart for the first time, the system's biggest miss was a lefty problem, not a broken tool.", "date_modified": "2026-07-10T01:00:00.000Z", "author": { "name": "Trevor McCormick", "url": "https://github.com/trevormccormick" }, "tags": [ "Devlog", "Computer Vision" ] }, { "id": "https://optic-yellow.com/blog/t1-answers-in-the-back", "content_html": "That clip up top: two court outlines on the same night-match frame. Red is\nthe camera math borrowed as-is from the practice match; green is a fresh\nfit built from this match's own lines. Red drifts visibly off the real\ncourt — and its own accuracy numbers still said it looked fine.
\nEvery setting this project has tuned so far was checked by the one person\nwho tuned it. That's a chair umpire overruling his own call and then\ncongratulating himself for getting it right — the call might be correct,\nbut nobody should trust it just because he says so.
\nPreviously: the plan is to feed broadcast video in and get the same\nshot-by-shot notation out that Match Charting Project volunteers\nhand-code by watching real matches (cv-01).\nEvery dial since then — how confidently motion counts as a swing, how the\nserve toss gets spotted, whose shot each ball is credited to — was tuned\non one practice match, Zverev–Gasquet, and checked with my own eyes\n(cv-09). So was the\nhomography — the rule, built from the court's\npainted lines, that converts a screen position into a court position\n(cv-03).
\nA line judge who grades his own overrules isn't cheating, exactly — he's\njust human, and human eyes trust their own first read. Tennis's fix is a\nsecond set of eyes that never made the call. This project needs the same\nfix: an answer key built by someone who never saw its output. This\ndevlog calls that ground truth (more: Court Vision in Plain\nEnglish) — and Jeff Sackmann's Match Charting\nProject supplies it: volunteers who hand-code real matches, shot by\nshot, in the exact notation this project is trying to produce.
\nThere's a second half to the rule, the half that makes a test count\ninstead of a rehearsal: the moment a match gets used to fix a bug, it\nbecomes practice, not proof. So every setting gets locked first —\nwritten down, untouched — before the software meets a match it never\ninfluenced. Freeze first, grade second: that's a benchmark.
\nThe Match Charting Project has hand-charted more than 7,500 men's\nmatches; the practice match isn't one of them. A match from the same\ntournament, court, and broadcast package is: Canada Masters 2017, Nadal\nvs. Shapovalov, round of 16 — 225 points, charted point by point, at\nnight under floodlights, both players left-handed. Neither fact was\ntested on the practice match: its shot calls assume a right-hander, its\nmotion settings were tuned in daylight. Same court and graphics package,\nso any failure here traces to what actually changed.
\nHeld in reserve: Federer vs. Haase, the semifinal — day, both\nright-handed, the closest match here to practice conditions. If the\nsoftware struggles there too, lefties and floodlights aren't why.
\nThe test's rules were locked before any of it ran: no setting changes\nonce grading starts (one part stayed adjustable — appendix). For every\nclip that lines up with a human record, the check compares who served,\nwhere the serve landed, how long the rally ran, and how often the\nshot-by-shot calls match — counting only what the software committed to,\nnot its shrugs. That 225-point answer key is public, free with\nattribution, and cost nothing to get.
\nEach clip first has to be found, then matched to a point. Finding them\ngave the practice match the most trouble; here it barely put up a\nfight — point the software at the file paths and the new camera math,\nand 25 clips of real play fell out, no setting touched. Matching them\nmeans reading a score bug — the little scoreboard graphic in the corner,\nfrozen pixel-for-pixel during a point, changing only when the point ends\n(more: Court Vision in Plain English) — and finding\nthe line in the human's chart with that same score.
\nOne convention nearly wrecked it: the human record lists the server's\nscore first; this match's bug always lists Nadal's first, serving or\nnot. Every Shapovalov service game disagreed on which number came\nfirst, costing six wrongly-matched points before I caught the pattern.\nBefore the fix, 17 of 25 clips lined up; after it, 23 of 25 did.
\nThe remaining two aren't errors, they're genuine ties: a 40–Advantage\nscore that happened twice in the same game, and a deuce from a game that\nreached deuce ten times — indistinguishable by score alone. The process\neven caught a mistake of mine: I'd misread one clip's serve indicator by\neye, and the human chart showed Shapovalov had actually served.
\nThe camera math built on the practice match missed by about 20 pixels on\nthis new footage — different filming day, camera remounted. Refitting it\ntook three failed attempts before a fourth worked.
\nAttempt one reused the first homography experiment's approach,\nunchanged. Under floodlights, the paint boundary where the court's blue\nsurface meets its green surround reflects light almost exactly like a\nreal white line. The fit grabbed that paint edge and drew a court 15 to\n20 pixels too big.
\nAttempt two made the software pickier about what counts as a white\nline — which killed the paint problem, and the far baseline and both\nsidelines along with it. Not enough real lines left to build a court\nfrom.
\nAttempt three searched narrow bands near where the practice match's\nlines had been. It failed more quietly: one band locked onto ground\npaint 70 pixels off; another mixed in doubles pixels while hunting for a\nsingles line. The fit then reported a great accuracy number for\nitself — but that number only measures agreement with the pixels the fit\nchose to trust, and some of those were the wrong paint. Drawn onto the\nactual frame and eyeballed, the center line was obviously off, by 32\npixels: ten times what the fit had reported.
\nWhat worked: a human — me — zoomed the frame in six times and picked the\ncourt's four corners by eye, once. The same kind of one-time nudge the\nvery first tracking experiment needed, when someone had to draw a box\naround the ball before it would track\n(cv-02). Even that failed twice at\nfirst: the corners read near the baseline were the singles corners,\nnot the doubles corners the court is actually measured to, further out\nand blurred into paint at the frame's edge. Two pieces of hand-worked\ngeometry confirmed which was which. The final fit mixed far doubles\ncorners with near singles corners, holding attempt two's lines back as\nan independent check.
\n| What got checked | Result | What \"good\" looks like |
|---|---|---|
| Clips found in the highlights reel | 25 of 25 | Every real point located, none missed |
| Clips matched to a specific human-charted point | 23 of 25 (2 are genuine ties, not errors) | As close to all of them as honest ambiguity allows |
| Final camera-math accuracy | Service line 0.8 px · Baseline 1.4 px · Center line 6.1 px | Single-digit pixels, not tens |
| Cost of both answer keys | $0.00 | Free — public data, no paid tools |
This is the third time this project has hit the same wall, so it's a\nstanding rule now: a number can vouch for a wrong answer, but a drawn\nline on the actual picture can't lie the same way. Every camera-math fit\ngets rendered onto the video and eyeballed before it's trusted — which is\nexactly what the two colored outlines in the clip up top are for.
\nTwenty-five clips found, twenty-three tied to a line from a human's\nchart, a court that measures accurately, every setting still frozen.\nCost for both sessions: $0.00 — public data and an afternoon staring at\npixels. Next: turn this into an actual scorecard, machine against human,\npoint by point.
\nFootage: the 8:21 TennisTV highlights reel. Ground truth lives in the\nrepo under data/mcp/ (CC BY-NC-SA, attributed). Eval protocol, exact\nterms: constants frozen; segmentation could still be re-tuned for the new\nreel's cuts (finding points isn't the hypothesis — charting them is);\nmetrics per aligned point are server end, serve zone, rally length, and\nletter accuracy over committed letters only; coverage reported beside\naccuracy; no silent cherry-picking.
Score-bug join: MCP's Pts column is server-first, the on-screen bug is\nNadal-first. point_06's serve marker was misread by eye; the join\ncorrected it.
Homography attempt 1 was M1's recipe, verbatim; under floodlights the\nblue-to-green paint boundary reads V≈155, S≈55, the same signature as a\nreal line at grazing angle. Attempt 3's guided bands were seeded from\nthe dev prior; its self-reported residual was a beautiful 2.9 px,\nagainst the 32 px actually on the ground — plausible-but-wrong, caught\nonly by rendering the reprojection and looking at it.
The fix echoes the M0 ball-click precedent: one human input per match.\nThe two cross-checks that settled which corners were real — far-corner\nmidpoint lands at x=603 against a measured center line of 606, near-corner\nreads implied 643; pixels-per-meter is linear in image y (48 at the far\nbaseline through 80 at the near service line), extrapolating to ~100 at\nthe near baseline and predicting a doubles span of ~1095 pixels, exactly\nthe white band's raw extent. The final fit held attempt 2's tight-mask\nlines back as untouched validators rather than fitting to them.
That clip up top: a single point charting itself live. A box tracks the\nball, a second box flashes onto whichever player the software thinks\njust struck it, and a line of notation builds along the bottom, shot by\nshot. Late in the clip, watch that second box drift off the court and\nland on a spectator — hang onto that, it matters later.
\nFreeze a rally at the instant of contact, and sometimes you genuinely\ncan't tell who just hit the ball — it's caught in the air, equally far\nfrom both players.
\nThis project is teaching software to watch broadcast tennis and write\nout the same shot-by-shot notation a human volunteer hand-writes for the\nMatch Charting Project (cv-01). The last\nentry solved the pipeline's final\nmanual step — the serve toss rises alone above the server's head, so the\nsoftware now finds and boxes the game ball itself — and produced the\nfirst complete points, start to finish. But the column recording who\nstruck each shot broke tennis's own rule that hitters alternate,\nwhenever the ball was caught mid-air at contact, equally far from both\nplayers.
\nTennis already has a model for resolving exactly this kind of\ndisagreement: a chair umpire weighing a disputed call. No single witness\ngets trusted blindly — an electronic call, a line judge's shout, the\nmark on the court can each say something different — and the umpire\ndecides which to believe, not just which spoke first.
\nThat's the idea that replaced \"whoever's closest.\" It leans on the\nsport's strongest fact: hitters alternate. A rally has exactly two\npossible stories — near player struck first, or far player did — so\ninstead of deciding shot by shot, the software decides once per rally,\nscoring both stories against independent evidence and keeping whichever\none wins. That's a weighted vote: several independent signals, each\ntrusted by a different amount, added together into one decision.
\nThe alternating pattern itself is allowed to bend in exactly one place:\nif the ball vanishes from the tracker for a stretch of frames, a hit\ncould be hiding inside that gap, undetected — and only there can the\nnear-far-near-far pattern skip a beat.
\nThe first fix tried was the obvious one: whoever's closest to the ball\nat contact hit it. It failed immediately — right at the instant a hit\nis detected, the ball is often out in open court, closer to neither\nplayer than the other.
\nThe next idea reached for physics instead of position: after a real\nhit, the ball should move away from whoever struck it. Converted into\ncourt coordinates, that direction came back backwards after hits from\nboth ends. The culprit is the homography — the recipe that turns\na spot on the TV screen into a spot on the court diagram, more on it in\nCourt Vision in Plain English — which only tells\nthe truth for things touching the ground. It's been lying about anything\nairborne since the very first post that built\nit; bounces happen on the ground, so they\nwere always honest, and nobody had noticed the lie until this rule\nleaned on it.
\nThree witnesses get called per rally, in order of trust. Touch\ncounts most: did the ball reach exactly one player's own patch of court\nduring the contact window? The serve call counts less — a starting\nhunch, not gospel. Which half the ball landed in counts least: a\nbounce detector — the \"speed suddenly collapses\" signature from Court\nVision in Plain English — only watches the\nfar half of the court right now, so it can vote \"the near player hit\nit\" but not the other way round yet. Half a witness, but an honest one.\nEvery vote that loses stays on the record instead of getting erased — a\nbuilt-in honesty check, so a witness wrong more often than it should be\ncan't quietly hide its bad calls.
\nMaking the serve call outvotable paid off the first time it mattered.\nIn one point, the serve detector was confident the point started at the\nnear end; touch and landing evidence together pointed far; the vote\nwon. A frame-by-frame check of the footage sided with the vote.
\nOnce a rally's pattern is settled, proximity becomes trustworthy again\nfor a narrower job: pinning down the exact contact frame, now that it's\nknown which player to look near. But the stroke-type letter — forehand\nor backhand — only gets written down if the ball demonstrably reached\nthat player's own patch of court, not just \"somewhere nearby.\" That\npatch scales with how big the player looks on screen: the far player is\nonly about 55 pixels tall, so one fixed-size box for both would reject\nnearly every far-end letter before it had a chance.
\nTracking boxes also go rogue at the worst moments — one wandered onto a\nspectator standing by a Perrier cooler, another landed on a court shadow\ninstead of a person. That's the box that drifted off-court in the clip\nup top. Whenever that happens, the letter stays a question mark — the\nsame \"declined to guess\" mark explained in the notation\nlegend — rather than a wrong one.
\nRead with that legend, the clip's string is: a serve down the T (s6);\na forehand toward one side (f1); a forehand where the stroke is\ncertain but the side isn't (f?); a backhand toward the other side\n(b3); a backhand with the side unresolved (b?); a shot nobody's sure\nwho hit, though the ball's own path still says the other side (?3); a\nforehand back up the middle (f2); and an ending not written yet, so\nit's ? too. Spot-checked against video, the committed letters held up.
Before any paid tracking ran, the plan got a free review: every\ncandidate starting box went into one grid of still images, eyeballed by\nhand. That alone killed most of the backup ball-finder's picks — the\nmethod that kicks in with no confident serve toss, chasing \"the fastest\nsmall thing moving,\" fooled by lens glare, shadow edges, court-line\njunctions, the net band, a FedEx cooler, even a ballkid. Of 29 backup\ncandidates reviewed, only 3 survived. Three points moving in a straight\nline isn't proof of a flying ball, especially mid-camera-pan — a\nbackground object can fake that chain too.
\nThe same pass fixed the main method for free. The search box for the\ntoss above a server's head used to be a fixed size, so on far-end serves\nit was searching well above a player barely 55 pixels tall — effectively\nsearching the crowd. Scaling that box to the player's own size, plus a\nfilter dropping racket-shaped blobs, recovered seven far-end tosses the\nold search had thrown into the stands.
\n23 clips went to the paid tracker; 18 came back usable. Five were lost\noutright — SAM, the promptable tracker from Court Vision in Plain\nEnglish, locking onto a stuck box instead of the\nball, including every borderline case that had looked \"good enough\" on\nthe free review. The eyeball review, not any automatic score, was the\nreal ground truth.
\n| What got measured | This session | What \"good\" looks like |
|---|---|---|
| Full points charted | 21, up from 5 before this session | Every point in the reel, eventually |
| Serve-toss ball-finder | 18 of 25 tries, 72% | This is the primary method — most tries should land |
| Backup ball-finder, after free review | 3 of 29 candidates, about 1 in 10 | A rare fallback, not a real plan |
| Clips sent to the paid tracker | 23 sent, 18 came back usable | Every clip returns a clean track |
Both serve codes finally show up correctly on far-end serves — six of\nthem — s5 (to the body) and s6 (down the T), the zone\ncodes that had existed in the code for a while but\nwere never switched on; serves had been getting a cruder guess instead,\nthe same rough \"which third of the court\" scale every other shot uses.\nOne string came back fully committed: s5f3b3? — a body serve, a\nforehand toward one side, a backhand toward the same side, and the same\nunwritten ending every string gets right now.
Put it the way a tennis fan would: don't trust whoever's standing\nclosest, trust the best evidence — and let it overrule your own serve\ncall if it has to.
\n? cases: a \"far\" box on a spectator by the\nPerrier cooler, a \"near\" box on a court shadow. Frame-verified\ncommits: 3/3 on point_16, plus a point_24 spot-check.serve_zone() is actually called now — v1 had defined the function\nand then coded serves by full-court thirds anyway. Conflicts column\n(votes that lost) kept on record as the honesty metric.point_36: s6f1f?b3b??3f2? (loop v2 demo clip — far box drifts onto a spectator)
point_24: s5f3b3? (complete string, spot-checked)
Session cost: about $1.50. Project total: about $2.80 of the $9 budget —\nthis low because the free eyeball review caught problems before they\nbecame paid API calls. The toss bootstrap carried this session; the\nbackup ball-finder still needs a rethink, not a tune-up.
", "url": "https://optic-yellow.com/blog/m3-the-ball-gets-a-vote", "title": "M3: The Ball Gets a Vote", "summary": "When the software can't tell who hit the ball, it holds a vote instead of guessing — and the ball outvotes the serve call, on camera.", "date_modified": "2026-07-10T00:30:00.000Z", "author": { "name": "Trevor McCormick", "url": "https://github.com/trevormccormick" }, "tags": [ "Devlog", "Computer Vision" ] }, { "id": "https://optic-yellow.com/blog/m3-toss-is-the-bootstrap", "content_html": "That clip up top: a whole point, serve included, charting itself — the\ntracking box locks onto the ball at the toss on its own, no clicking,\nand the notation string builds along the bottom. First time this\npipeline has gone start to finish, and it's still a little rough.
\nEvery rally tracked so far needed a person to draw a box around the ball\nby hand first. Fine once. Absurd sixty times. The fix: a fact about\ntennis itself. Every point begins the same way, and the software can now\nspot that beginning on its own.
\nTwo posts back, the first experiment taught a promptable tracking AI\ncalled SAM 3 to hold a tennis ball through a\nrally — but only after a human circled that exact ball with a box, by\nhand: it could follow once pointed at,\nnever find one alone. Last time out ended on a\ndebt: its notation string was a rally's\nmiddle, no beginning, because nothing yet knew where a point starts.
\nNo rally begins in the middle. Every point opens with one unmistakable\naction: a player stands still, tosses one ball straight up, and swings.\nNothing else moves like that — the receiver stands still, every spare\nball sits still in a ball kid's hands, and the one thing rising alone is\nthe game ball.
\nThe bootstrap: instead of a person pointing at the ball every clip,\nthe pipeline uses something tennis always does anyway — the toss — to\npoint at the ball for itself. It already compares each frame to a photo\nof the empty court and flags what's different, the same trick that\nfinds the players for free. The only\nnew blob near a still server, right before contact, is the rising toss —\nthe software boxes it itself (finding, not yet\nfollowing), then hands the box to the\ntracking software. Detect once, then track. The click is gone.
\nThe pipeline has been fooled before, by ball kids holding spare balls\nand a line judge who erased himself standing still. This time it was\nbroadcast graphics. Without a confident serve, the software falls back\nto chasing the fastest small mover on court — and its first two guesses\nwere the scoreboard graphic (more on that box\nlater) flipping on every score change, and a small\nnet sign vibrating in the breeze.
\nThe fix: a ball in flight keeps one direction and one rough speed, frame\nafter frame — flying ballistically. A flickering graphic doesn't fly\nanywhere, so requiring three straight consistent frames knocked both\nout. Two clips still beat it, fooled by lens glare off the court's own\nlines during a camera pan — unsolved, noted rather than hidden.
\nSix clips ran through the paid tracking service for 55 cents total,\nless than a candy bar. Those tracks met the hit-and-bounce\nreader on a complete point for the first\ntime, and found something no mid-rally clip could show: servers\ndribble before the toss. Bouncing to yourself is a real hit and a real\nbounce, over and over, so the detector — correctly, by its own rules —\ncharted the warm-up as rally. Fix: the software already knows which\nframe the point starts on. Discard anything before it.
\nTwo clips produced a full serve-to-final-shot line with nobody typing\nmid-point. One reads, shot by shot: serve toward the middle (s2), a\nbackhand the software couldn't place (b?), backhand wide (b1),\nforehand same side (f1), forehand back to the middle (f2), backhand\nwide the other way (b3), two more backhands, and an ending it couldn't\ncall — eight shots, serve confirmed right. The other was two shots and\nout: an unclear serve, one unplaced forehand return, an ending it\ncouldn't call.
Checking both by eye: serve call right, shot count rally-shaped — but\nthe column saying who hit each shot broke the serve-return pattern\nwhenever the guess for contact caught the ball mid-air, equally far from\nboth players. Nearest-player-to-the-ball means nothing once the ball is\nclosest to nobody. The letters aren't trustworthy yet; the shape under\nthem is.
\n| What got checked | What came back | Good enough? |
|---|---|---|
| Serve call | Right, both points | Yes |
| Shot count | Rally-shaped | Yes |
| Who hit each shot | Breaks on mid-air contact | Not yet |
| Broadcast impostors | Score bug + net sign caught; 2 clips still fooled | Mostly |
| Cost | 55 cents, six clips | Well inside budget |
Say it the way a tennis fan would: you don't need to point at the ball\nif you already know every point starts with a toss.
\npoint_16: s2b?b1f1f2b3b?b3? (serve ✓ — 8 shots, rally-shaped)
point_31: s?f?? (short point: serve, return, over)
This session's six clips cost 55 cents; project total is about $1.30 of\na $9 budget. 48 clips stay untouched on purpose — every fix still ahead\nis free to try on the six already paid for.
\nThe point string that opened this stretch read ??2?1?1?2?2?2???,\nalmost all question marks, no s anywhere: a rally with a middle and no\nbeginning. It now starts with an s.
That clip up top: every serve the pipeline felt sure about, clip\nafter clip — which player served, and which side of the court, deuce\nor ad, flashed the instant the toss goes up.
\nSixty broadcast clips are sitting there ready to be charted, and every\none starts with the same problem: telling an actual tennis player\napart from everyone else standing near a court who isn't one.
\nPreviously: the last post sliced a\nhighlights reel into sixty complete-point clips using screen-to-court\nmath as a camera fingerprint. That math, the homography — matching\nthe court's painted lines to its real dimensions to turn screen\nposition into court position, from Pixels to\nMeters — carries a free trick: median-blend\nmany frames of the same court and the players, who never stand still\nfor long, vanish into a photo of an empty court. Two posts\nlater, that trick already found both players\nin one rally well enough to call every forehand and backhand. Doing it\nacross sixty new clips would otherwise mean paying SAM 3 — the\npay-per-prompt \"find that object\" AI that tracked a ball for about\nsixty cents in the second\npost — roughly $0.15 a clip, $9 for the\nbatch, every rerun.
\nThis session tried to cash in that free trick everywhere: build the\nempty-court photo per clip, then flag whatever pixel changed as a\nplayer. One catch — both photos need the same camera position. A\nbroadcast camera looks fixed, but any camera operator worth their job\nnudges the shot to keep the rally centered, the way your own eyes\nwould tracking the ball from courtside. A pixel-by-pixel comparison\nnotices every nudge: the baseline in one photo no longer sits on the\nbaseline in the other. It's like tracing a picture on an Etch A\nSketch, then shaking the toy before checking your work — nothing\nlines up.
\nThe fix: stabilization, nudging every frame back into line with\nthe others before comparing anything, so the same pixel always means\nthe same spot on the court. More on why the empty-court trick needs a\nstill camera: Court Vision in Plain\nEnglish.
\nThat math, first fit two posts back on a sixteen-second rally, found\nalmost no camera movement — so the code kept one screen-to-court\nconversion for the whole clip. A full point can run a minute or more,\nand these sixty clips run that long. Aligning every frame of a clip\nto its own first frame and measuring the gap told a different story:
\n
The broadcast camera, measured, over one point: a sharp jump — as\nmuch as 32 pixels — mid-point, on top of slow drift throughout.
\nThe camera operator was following the play — invisible to a viewer,\nfatal to the comparison. It blurs the empty-court photo, and near the\nfar baseline, thirty-some pixels of drift is worth meters of\ncourt-position error. Every frame now gets stabilized first.
\nWith frames stabilized, the near player — big, always moving — showed\nup in essentially 100% of frames; the far player, barely 35%. The\nempty-court photo had swallowed him.
\nHe's small, and barely moves between points, so a photo built from\none clip bakes in a faint standing copy of him — the same trick that\nerased a stationary line judge two posts\nago.
\nThe fix: build the photo from five clips instead of one — this clip\nplus its four nearest neighbors, aligned and blended by taking the\nmiddle value at every pixel again. Players stand somewhere different\nevery point, so five ghostly copies never land on the same pixel\ntwice, and the blend votes them all out. Far-player coverage: 35% to\n96%. Near player: still ~100%. Cost: nothing.
\nThe first serve-detector run flagged several clips with the far\nplayer standing 13.5 meters from the court's center, on a court only\n10.97 meters wide. Nobody serves from outside their own court.
\nThose weren't players. They were ballkids, crouched past the\ndoubles sidelines inside the patch of court scanned for the far\nplayer, shifting weight just enough to out-move a motionless server.\nThird case of mistaken identity: the very first\nsession swept spare-ball-holding\nballkids into \"the tennis ball,\" and two posts\nago a stationary line judge got swept into\n\"the empty court.\" To a pixel-by-pixel comparison, anyone standing\nnear a court looks like they belong on it. Fix: the far-side scan\nregion is now cut tight to the court; the near side stays generous.
\nWith players tracked, the serve should be readable from posture: the\nserver hugs the center mark, the returner waits wide — which side of\nthe mark gives away deuce or ad, for free — and the toss stretches\nthe server's outline to full reach, so blob height should spike right\nat the toss.
\nVersion one checked only distance to the center mark, and failed\nfast: some clips open mid-rally with no toss to find, some end on a\nleftover zoom fragment, and — the costly lesson — returners often\nstand nearly as far from the mark as a server does, so a return swing\ncan read as a toss.
\n![\"Five-frame](\"https://optic-yellow.com/assets/images/m3-serve-textbook-678808c076f31ebd4f2378a1296dd9af.webp\")
Right: a clean toss, full stretch, on the flagged frame.
\n![\"Five-frame](\"https://optic-yellow.com/assets/images/m3-serve-fooled-5c5f9f783c250135bf0317d826245ab7.webp\")
Wrong: the flagged \"server\" is the returner, off the mark,\nmid-swing — every check in version one said yes.
\nVersion two clears several checks before it commits — tracked early\nand reliably, right distance from center mark and baseline, a genuine\nheight spike — or it refuses, with reasons attached, instead of\nguessing.
\nResult: 40 of 60 clips get a confident serve call; the other 20 are\nrefused, on the record — clips that open mid-rally or rejoin\nmid-point, plus two long clips too thin on far-player tracking to\ntrust, including the marathon 49-shot rally from the last\npost. Spot-checking a sample of the\nconfident calls by eye: 6 of 8 named the right server, 5 of 8 found a\nusable toss moment — honest, not perfect. Geometry alone tops out\nhere; a serve is, definitionally, the ball leaving the hand and\ncrossing the net, and once the ball itself is tracked over these\nclips, it settles what posture can only guess at.
\n| What got checked | Result |
|---|---|
| Far player, before the fix | seen in 35% of frames — mostly a ghost |
| Far player, after the fix | seen in 96% of frames — essentially solved |
| Near player, throughout | seen in ~100% of frames — never a problem |
| Clips confident enough for a serve call | 40 of 60 |
| Clips that honestly refused to guess | 20 of 60 |
| Spot-check: correct server named | 6 of 8 |
| Spot-check: usable serve moment found | 5 of 8 |
A camera that looks perfectly still to a viewer at home can still\nwreck a pixel-by-pixel comparison — so before comparing two photos of\nthe court, make sure they were taken from exactly the same seat.
\nAlignment: each frame is registered to its clip's first frame using\nECC (Enhanced Correlation Coefficient) — an alignment method —\nrestricted to pure translation (x/y shift, no rotation/scale).\npoint_29 is the 32px-spike example, now the mandatory first step\nbefore plate-building, background subtraction, or homography\nprojection. The free trick cashed in this session: frame-minus-median\nalready IS a player detector, left over from M1's clean-plate trick.\nGhost-plate fix: a 5-clip window (target + 4 nearest temporal\nneighbors), each ECC-aligned before median-merging; zero added API\ncost, pure numpy.
Serve detector v1: single heuristic, distance-to-center-mark only.\npoint_33: clean, correctly-flagged case. point_40: failure case,\nflagged \"server\" is the returner, mid-swing, off-hash, passing every\nv1 check anyway. v2: four required gates — (1) early-clip track\ncoverage, (2) distance to center mark, (3) distance to own baseline,\n(4) a genuine local peak in blob height. Any failed gate produces an\nexplicit no_confident_serve result with reasons attached, not a\nforced guess. Refusal log flags cold-opens, mid-point rejoins, and two\nlong clips with sparse early far-tracks — including the 49-shot rally\nfrom the cutting-room post (cv-06).
Also queued: fps as a real parameter — this reel runs 25fps against\nthe original clip's 30fps, and velocity thresholds are secretly\nper-frame numbers.
Session tally: player positions across all sixty clips, serve details\non two of three, three wrong assumptions caught — for $0.00. Project\ntotal: still about seventy-five cents, all spent in the very first\ntracked-rally session. SAM 3, the paid\ntracker, has exactly one job left: finding the ball.
\nNext: a ball detector on these same sixty clips. Then the serve call,\nhits, bounces, and shot letters should click into a complete point,\nwritten in the Match Charting Project's own notation — the human-made\nanswer key this project is chasing. The one\nworking chart string so far, from two posts\nago, was just the middle of a rally. The next\none should have a beginning, a middle, and an end.
", "url": "https://optic-yellow.com/blog/m3-static-camera-isnt", "title": "M3: The Static Camera Isn't", "summary": "The 'still' broadcast camera actually drifts mid-point — and that breaks a free trick for spotting players in video.", "date_modified": "2026-07-09T23:59:00.000Z", "author": { "name": "Trevor McCormick", "url": "https://github.com/trevormccormick" }, "tags": [ "Devlog", "Computer Vision" ] }, { "id": "https://optic-yellow.com/blog/m3-the-cutting-room", "content_html": "That clip up top: sixty point candidates, one after another, cut\nautomatically out of a 24-minute highlights reel — almost every one\nstarts on a serve.
\nWatch a tennis broadcast and count how much of it is actually a point\nbeing played. Not much. There's a replay after nearly every big rally, a\ncoach's-box shot, a close-up between points — actual tennis is a small\nslice of the screen. Today's question: can software find, on its own,\nonly the real points inside a whole reel of broadcast footage? It can —\nand it turned up the best plot twist of the project so far.
\nThe project's goal, set on day one, is\nsoftware that writes the same shot-by-shot notation Match Charting\nProject volunteers write\nby hand — that first post already flagged finding the right clips as its\nown job, then four posts put it off. An earlier\nsession built the conversion from screen\nposition to court position, matched to the court's own painted lines.\nThe most recent session called every forehand\nand backhand across a whole rally — but that rally was one 16-second\nclip that starts and ends mid-point, with no serve, ending, or score\nanywhere in it.
\nThat clip's match — Zverev versus Gasquet, Montreal 2017 — also has a\n24-minute extended highlights reel: same broadcast, same camera,\ndozens of complete points, cut together with close-ups, crowd shots, and\nslow-motion replays. So today's job isn't a tennis question. It's an\nediting-room one: out of the reel's 36,234 individual pictures, which\nshow the real camera, and which are decoration?
\nA tennis broadcast has one main camera — high, behind the baseline,\nshowing the whole court — the only camera a Match Charting Project\nvolunteer could chart from. Everything else the director cuts to is a\ndifferent camera, and a highlights reel is built almost entirely out of\nthose cutaways.
\nThe first idea, tried and broken below, was to spot the difference by\ncolor: the court is blue, the ground around it is green. What actually\nworked reused something already built: an earlier session's conversion\nfrom screen position to court position, the homography, fit once by\nmatching the court's painted lines to its real dimensions (more on it\nhere). That conversion doubles as a camera\nfingerprint — a test for whether a frame was shot from that one\ncalibrated camera position. Project the true shape of a tennis court\nthrough it onto a new frame: on the main camera, the lines land exactly\non the paint, because the conversion was built for that camera's\nposition. On any other camera they miss — the projected court floats\noff into empty space instead of matching anything real.
\nChecking a frame becomes checking a few dozen spots: does court-colored\npaint sit where the projection says it should, and does\nthe strip just outside the doubles lines correctly not have court\npaint in it? Pass both, it's the main camera. Fail either, it's a\ncutaway — including an actual replay, which would otherwise chart the\nsame point twice.
\nThe first classifier measured blue and green in each frame against a\nthreshold, tuned on frames from late in the match. It looked great —\nevery accepted frame really was the main camera.
\nThen came the check nobody skips: looking at what it rejected. Six\nframes were obviously real points. All from the first set. All in\nblazing daylight.
\n![\"A](\"https://optic-yellow.com/assets/images/m3-blue-merge-fail-73e381c604983e418289d5fe4ecbb357.webp\")
The blue region found in a daytime frame: the court and the shaded\nseating bowl behind it, fused into one shape covering half the picture.
\nTwo things were going wrong, both about light. In daylight, the\nshaded seats read as the same blue as the court and merge into one giant\nshape. At the same time, the sunlit ground around the court washes out\ntoo pale to register as green. The match starts at noon in full sun\nand ends at night under floodlights, and a color rule tuned for one end\nbreaks at the other.
\n![\"Two](\"https://optic-yellow.com/assets/images/m3-framing-check-f3949bba096c087dc25179978e6b5cab.webp\")
The projected court outline, laid onto two frames of the new reel —\ndaytime and evening. Pixel-perfect on both: same broadcast, same\ncamera.
\nFifty spots that had to be court-blue, seventeen just outside the\ndoubles lines that had to not be, checked frame by frame across the\nwhole reel. Smooth out single-frame flickers, keep only stretches of\nthree seconds or longer, and the reel breaks into 60 segments, 725\nseconds of chartable tennis — about twelve minutes out of\ntwenty-four.
\n![\"Twenty](\"https://optic-yellow.com/assets/images/m3-seg-starts-666b51dd8a416c4d4eff122cbe10c664.webp\")
The first twenty segment openings. The service toss shows up again and\nagain — highlight editors cut in just before the serve, so it arrives\nfree with every cut.
\nAll sixty openings got checked by eye: every one is a real broadcast\nview, every one runs to a dead ball. Broadcast editors, it turns out,\nalready hand this project exactly the unit it needs: complete points.
\nEvery long stretch between segments got checked too, turning up twenty\ngenuine cutaways — including sideline and net-camera replays, failing\nthe same way.
\nOne gap was a real loss.
\n![\"Side-by-side](\"https://optic-yellow.com/assets/images/m3-zoom-drift-fa51aad0cac24a3f602df4625f0168f0.webp\")
Early in set one the broadcast used a tighter zoom. The court still\nread correctly inside, but the bigger court filling the frame swallowed\nthe outside-the-lines check. 131 seconds of real tennis, rejected.
\nThe \"camera never moves\" assumption broke for those two minutes — a\nknown hole, on the record. The fix, if needed: re-fit the conversion per\nstretch instead of once for the whole reel, which the earlier\nconversion work can already do automatically.
\n![\"Timeline](\"https://optic-yellow.com/assets/images/m3-point-timeline-289e3c9eaa492b38898614cd2bf2b6d3.webp\")
Twenty-four minutes of reel: 60 green segments found, one orange\nconfession.
\nAll sixty candidate clips went to disk. Segment 51 stood out\nimmediately: 59.6 seconds of unbroken main-camera footage, by far the\nlongest of the sixty.
\nThat's the point this match is famous for — a 49-shot rally, the one\nthat shows up in YouTube video titles. The very first clip this project\nwas built on turns out to be a 16-second\nslice from the middle of that same rally. The segment\nfinder, which knows nothing about tennis and only checks colors and\nlines, found the entire point on its own — serve included.
\nEvery session so far has been charting an excerpt of a much longer\nstory. It just got handed the whole thing.
\n| What was checked | Result |
|---|---|
| Frames sorted, real point or not | 36,234 |
| Real point segments found | 60, about 12 minutes of footage |
| Segment openings checked by eye | 60 of 60 correct |
| Genuine cutaways caught, replays included | 20 |
| Footage lost to a camera-zoom gap | 131 seconds |
| Longest segment found | 59.6 seconds — the match's famous 49-shot rally |
A tennis broadcast spends most of its time showing you everything except\nthe tennis, so before software can chart a single point, it first has to\nfind the one camera actually pointed at it.
\nSource: same Zverev–Gasquet Montreal 2017 broadcast, 24-minute extended\nhighlights reel, 36,234 frames.
v1: blue-fraction / green-fraction thresholds on HSV, tuned on\nlate-match frames. High precision, low recall — 6 rejected frames from\nset one (daylight) were true positives. Two simultaneous causes: (1)\nshaded seating bowl reads as court-hue in daylight and merges into a\nsingle ~50%-of-frame contour; (2) sunlit apron washes out to saturation\n38 vs. threshold 40.
v2: reuse the homography fit from the original 16-second rally clip as a\npose check — project the court model's corners onto every reel frame.\nOn-camera frames land pixel-perfect, daytime and evening. 50 interior\nsample points must read court-blue, 17 points just outside the doubles\nalleys must not. Boolean per frame, median-smoothed, segments trimmed to\nruns ≥3s. Result: 60 segments, all starts/ends manually verified.
Audit of inter-segment gaps: 20 confirmed cutaways, including\nsideline/net-cam replays (fail cleanly — different camera pose). One\nfalse negative: early-set tighter zoom (131s), interior probes still hit\nbut the enlarged court swallowed the apron-margin probes. Not yet fixed:\nrefit homography per block rather than globally; the fitting step is\nalready automatic.
Plot twist: segment 51 = 59.6s, the match's 49-shot rally (its\nYouTube-title fame). The original tracked-rally clip is a 16-second\nexcerpt from its middle.
Session cost: $0.00 — HSV thresholding + 67 pixel probes/frame (50\ninterior + 17 exterior). Project total: ~$0.75.
Open issue: reel is 25fps; the hit/bounce detector was tuned at 30fps,\nand its velocity thresholds are secretly per-frame deltas, not\nper-second — fps needs to become an explicit parameter before that\ndetector runs on these clips.
Chart state: ?f2f1f1b2f2f2b?? — leading ? = unknown serve zone,\nseven letter+number pairs (f2 f1 f1 b2 f2 f2 b?) = shots called last\nsession (shot 7's direction still open), trailing ? = unknown ending\ncode.
This session cost nothing — no AI calls, just color math and pixel\nchecks. Project total: still about seventy-five cents. One thing to fix\nbefore next time: this reel plays at 25 pictures a second, while the\nhit-and-bounce detector was built on a clip that ran at 30, and its\nspeed cutoffs quietly assume that number.
\nEvery blank left in the chart — the serve, the last shot's direction,\nhow the point ends — now lives inside these sixty clips, with a 49-shot\nmonster in the middle waiting to stress-test everything downstream.\nNext: point anatomy — finding the serve inside a segment, and the ending\nafter the last bounce.
", "url": "https://optic-yellow.com/blog/m3-the-cutting-room", "title": "M3: The Cutting Room", "summary": "Most of a tennis broadcast isn't the match — the software has to spot the one camera angle that actually shows it.", "date_modified": "2026-07-09T23:45:00.000Z", "author": { "name": "Trevor McCormick", "url": "https://github.com/trevormccormick" }, "tags": [ "Devlog", "Computer Vision" ] }, { "id": "https://optic-yellow.com/blog/m3-who-hit-that", "content_html": "That clip up top: both players outlined and tracked, FOREHAND or BACKHAND\ncalled out at every swing in the hitter's color, feet moving live on the\ncourt diagram — and along the bottom, the shot-by-shot notation typing\nitself out.
\nCall a shot forehand or backhand and you're answering a geometry\nquestion: which side of the hitter's body did the ball come off? You\ncan't answer that without knowing where the player was standing — and\nuntil now, this project had tracked the ball but never found the\nplayers. Too much for one sitting: session one of several, starting\nwith finding the players.
\nHere's what came before. This project's whole bet: feed it broadcast\ntennis video, get back the exact notation Match Charting\nProject volunteers\nhand-write for real matches — the kickoff\npost has the pitch. Session\none tracked the game ball through a\nrally for about sixty cents, once shown a box around it. Next, the\nhomography — read off the court's painted\nlines — turns a TV-picture spot into a\ncourt-diagram spot, except anything airborne, which converts to its\nshadow's position instead. Then hit-and-bounce\ndetection read events out of kinks in the\nball's path — a hit reverses it, a bounce doesn't, the court doesn't\nreturn serve — producing the first rough\nchart: ??2?1?1?2?2?2???.
Here's the trick, older than computer vision — it's just averaging.\nStack dozens of photos of the same stretch of court, one per frame. Add\nup how often each spot is occupied: spots that barely change are the\ncourt itself, net and lines and grass; spots that keep changing are\nsomething moving through the scene.
\nTreat the barely-changing version as a stand-in photo of an empty court,\ncompare every real frame to it, and whatever's different is a player.\nThat's background subtraction. (More in the\nprimer.) This project already ran the\nsame trick once, to erase the players and get a clean court photo for\nmeasuring its lines — today, in reverse.
\nThe catch: the trick only knows \"moves\" or \"doesn't move.\" Stand still\nlong enough on camera and you get folded into \"the court\" — a line\njudge at parade rest for a whole rally erases himself as completely as\nthe baseline.
\nThe plan: draw a box around each player in one frame and let SAM 3 —\nthe promptable \"find that object\" AI from the\nprimer — track each box through the clip, same\ntrick that caught the game ball in session\none.
\n![\"Frame](\"https://optic-yellow.com/assets/images/m3-prompt-boxes-8c6e3ddca5533148fb90d30fb08b5b64.webp\")
The two boxes, drawn by eye and checked before spending real money. The\nline judge, in red, stands just outside the far box — deliberately left\nout.
\nTwo boxes went in. One came back — a single rectangle per frame, the\nunion of both players, stretched wide enough to swallow the line judge\nbehind Gasquet too. Same lesson as session one: \"tennis ball\" once\nmatched every ball; now two boxes merged into one.
\nUseless boxes, but the response also carries the raw shape data behind\nthem — an outline of what the AI thinks belongs to something. Split it\napart wherever it stops touching itself, and every frame resolves into\nthree shapes: Zverev, Gasquet, the line judge.
\nTwo of those are players. Finding the third is background subtraction\nagain: across the rally, add up how often each spot is covered by a\nshape, as a percentage. Anything covered more than 85% of the time\nbelongs to someone who never left — the judge, at parade rest\nthroughout. Subtract it, and two players are left.
\n![\"Broadcast](\"https://optic-yellow.com/assets/images/m3-static-judge-4a92e1a3770ad4a7cb4dbe02ad685ab3.webp\")
The one figure the AI outlined that never moved. Noticing that is the\nentire trick for telling a line judge from a player.
\nHe erased himself. Nobody taught the software what a line judge looks\nlike. The only rule: if it doesn't move, it isn't playing.
\nWhat's left splits into near player and far player, by which half of\nthe frame each shape's feet land in.
\n![\"Two](\"https://optic-yellow.com/assets/images/m3-players-traj-44a04497707705d3dd466f2d563db905.webp\")
Both players, every frame of the rally. The two short flat patches\naround frames 390 and 425 are the tracker briefly wandering onto\nanother far-side figure, both after the rally had ended, but on the\nrecord.
\nResult: both players tracked across all 480 frames, zero gaps. The ball\nitself, tracked in session one, only managed 470 of 480, with ten\ndropouts — easier for this AI than a small, fast ball.
\nSmall twist on an old caveat: the homography only tells the truth for\nthings touching the ground — an airborne ball converts to its shadow's\nposition, not its own. Feet are always on the ground, so a player's\nposition comes out exact, every frame. The most trustworthy dots in\nthis project belong to the players now, not the ball.
\nWith both players placed, deciding who struck a ball has three checks:\nwhich player was closest at the hit; which end of the court it was\nstruck from, the same read used last session for hits and bounces; and\nthe simple fact that a rally alternates — whoever didn't hit last, hits\nnow. All three agreed, on all seven hits — three signals nobody tuned\nfor this question, landing on the same answer anyway.
\nOnce a player's position is trustworthy, calling the shot needs no more\nAI — it's geometry. Ball on the right of a right-hander at contact is a\nforehand, on the left a backhand — the same call from the couch.
\nOne wrinkle: the camera-facing player has left and right flipped, so\nhis measurement gets mirrored. One liability: handedness is assumed,\nnot detected — a left-handed player would get every call flipped, a\ndebt the series collects later.
\nThe verdict: forehand, forehand, forehand, backhand, forehand,\nforehand, backhand — seven shots, seven calls, checked by pulling the\nvideo frame for every swing, the eyes-over-math discipline that saved\nlast session.
\n![\"Five-frame](\"https://optic-yellow.com/assets/images/m3-verify-backhand-553fc32eb53eb9b4783a017fed46eb66.webp\")
Shot seven, the rally's last swing: Gasquet's one-handed backhand,\nracket coiled high before he sweeps through it. If you're going to\neyeball-check a backhand, make it his.
\nSeven for seven — but one deserves an asterisk. On shot three, the ball\ncrossed so close to Gasquet's body that the measurement came out just\n1.5 pixels to the forehand side — a coin-flip margin. The video settles\nit: a low pickup off his right shoe, unmistakably a forehand. Right,\nbut only because the coin landed the correct way up.
\nThe two versions of this rally's chart, side by side:
\nbefore: ??2?1?1?2?2?2???
after: ?f2f1f1b2f2f2b??
Read the finished string left to right:
\n? is the serve — unknown, since this clip never shows\none.f2 f1 f1 b2 f2 f2 b are the seven shots, in order: forehand,\nforehand, forehand, backhand, forehand, forehand, backhand — this\nsession's addition. The digits (settled last session) mark which\nthird of the court each shot crossed into. Shot seven has no digit —\nit ends the point, coded differently.? marks are where the ball landed and how the point\nended — also missing from this clip.The chart also gained something the earlier version couldn't show:\nwhere each player was standing when they hit.
\n![\"Top-down](\"https://optic-yellow.com/assets/images/m3-shot-map-579a65d101101cd5079501dc29c3c79d.webp\")
Seven arrows — who hit it, which wing, from where, to where. Dashed\narrows land on the far side, still accurate only to within a meter or\nso.
\nZverev played right around his baseline; Gasquet, more than three\nmeters behind his — court positioning anyone who's watched him play\nwould recognize.
\n| What we checked | What happened | What \"good\" looks like |
|---|---|---|
| Both players tracked, every frame | 480 of 480 frames, zero gaps | The ball itself only managed 470 of 480, with ten short dropouts |
| Who hit each shot, three ways | All three signals agreed, on all seven hits | Independent checks landing on the same answer, unprompted |
| Forehand vs. backhand, checked by eye | 7 of 7 correct | Every one of the seven swings verified against actual video |
Stand still long enough on a tennis court, and even a computer starts\nto think you're part of the furniture.
\nThe box-merge failure. SAM 3's two-box prompt returned one\nunioned bounding box per frame, not two — same \"prompts fuse into one\nconcept\" behavior as the text prompts in session one; per-object\nstructure doesn't survive a single API call. The box's right edge\nlanded at roughly pixel 807, exactly far enough to include the line\njudge. The response does include the raw run-length-encoded (RLE)\nmasks — space-separated start/run pairs over the flattened frame,\nundocumented like the box format. Decoded and split into connected\ncomponents, each frame resolves into three spatially disjoint blobs\n(Zverev, Gasquet, line judge); nobody occludes anybody else in image\nspace.
\nThe static filter. For every pixel, the fraction of the 480\nframes it's covered by any mask (np.mean over the stacked binary\nmasks, time axis). Anything \"on\" over 85% of the time is treated as\nstatic and subtracted from every frame's mask before splitting the\nrest into players — the same averaging that built the player-free\ncourt photo for the homography, run in reverse.
The forehand/backhand formula. Shot type = sign of (ball\nx-position minus striker's blob-center x-position) at contact,\nmirrored for the far player. Handedness is assumed, not detected.\nShot 3's raw offset was −1.5 pixels — inside the noise floor of a\nsingle pixel of foot-tracking error, essentially arbitrary before\nvideo confirmed it; needs a confidence threshold and a bodyline\npolicy before an untuned rally. Position precision differs by side:\nnear-side (Zverev, whose three shots came from 0 to 1.3 meters behind\nhis baseline) converts with high confidence; far-side (Gasquet) is in\nmeters-per-pixel territory, so his \"more than three meters behind the\nbaseline\" is directionally solid, not precise.
\nSession accounting. One paid API call (the two-box SAM 3 prompt);\neverything after ran locally at no cost.
\nSession cost: $0.15 — one API call, everything after it just\narithmetic on numbers already in hand. Running total: about\nseventy-five cents.
\nThe remaining question marks didn't move — the serve, where the ball\nlanded, how the point ended, all still blank, since a mid-rally\nhighlight can't show a point starting or ending. Next session: better\nsource footage — full match video, complete points, camera cuts and\nall.
\nThe chart still reads ?f2f1f1b2f2f2b?? — seven question marks fewer\nthan it started with, three still to go. Time to earn the rest of the\npunctuation.
That clip up top: the pipeline calling out HIT and BOUNCE in real time,\npinning every one onto a court diagram the instant it happens.
\nThe last post ended with a warning that turned into today's homework: the\nsystem can only trust the ball's screen position at the exact moments it's\ntouching the court — everything else is guesswork. So today's question:\nwatching nothing but the ball's path across the screen, can software find\nevery hit, and every bounce?
\nThe detector I built today ended up calling all thirteen of those moments\ncorrectly. It got there by being wrong twice first, and both mistakes\ntaught me more about how a tennis ball moves than the successes did.
\nPreviously: last time, the software\nlearned to convert \"where the ball is on your TV screen\" into \"where the\nball is on the actual court\" — the same trick a fan does by eye, guessing\nwhere a shot landed. The catch, carried straight into today: that\nconversion only tells the truth for things touching the ground. A ball in\nmid-air converts to the position of its shadow, not the ball itself. So\nthe software needs to know exactly when the ball is on the ground before\nit can trust anything else it says.
\nA ball in a rally does exactly two things, over and over, and they look\nnothing alike in its path.
\nWhen a player hits the ball, it reverses direction — back the way it\ncame, sharp and immediate. When the ball bounces, nothing reverses. It\nkeeps traveling the same way, just lower and slower. A racket sends the\nball backward; the court just takes some steam off it and lets it keep\ngoing. That's the whole idea in one line: the court doesn't return\nserve. (More on reading events out of a flight path: Court Vision in\nPlain English.)
\nFirst move: turn the tracked dots into a graph of position and speed over\ntime, and look at it.
\n![\"Three](\"https://optic-yellow.com/assets/images/m2-velocity-structure-ba07189b654207d4df67731c483d9a8c.webp\")
The ball's up-down screen position, how fast that's changing, and how\nfast its shadow moves lengthwise down the court — across the rally's 290\nusable frames (each a single still picture — more in Pixels and\nframes), with 10 short gaps where the tracker\nbriefly lost the ball.
\nTwo traps, both from camera angle. First: a ball's up-down position on\nscreen isn't the same as its real height — a ball at the far baseline\nstill looks \"high\" simply because it's small and distant on screen — the\npretty arcs you'd expect a bounce to draw aren't real; they're height and\ndistance tangled together. Second, and stranger: the shadow's sideways\nspeed spikes to the equivalent of 200 meters per second when the ball is\nhigh on the far side. Nothing on court moves that fast — it's the camera\nexaggerating airborne motion.
\nMy first version found every cusp in the ball's up-down speed — where it\nstops falling — then called hit or bounce by whether direction flipped.\nIt found nine such moments and called eight of them hits.
\nThat's not how tennis works — almost every stroke is preceded by a\nbounce. Something was badly wrong, and the fix wasn't better math. It was\ndoing what a chair umpire does with a disputed call: stop arguing and\ngo look at the replay. I pulled the actual video frames around every\nflagged moment.
\n![\"Three](\"https://optic-yellow.com/assets/images/m2-verify-strip-2b4a8b4beebd4c7f9a20fc3e989fcdaf.webp\")
Frame 44, flagged as a hit. The ball is at Zverev's feet; his racket is\nnowhere near it. That's a bounce.
\nWatching those frame strips built a hand-verified answer key for the\nrally, and exposed two ways the detector was fooling itself.
\nNear-side bounces looked like hits. Right after a real bounce, the\nshadow speed collapses into a small, noisy number whose sign flips often\nenough to trick the direction test. Fix: stop asking whether direction\nflipped; ask how fast the ball leaves instead. A real return leaves fast;\na bounce's outgoing speed collapses toward zero.
\nFar-side bounces were invisible, not just hard. After a far bounce,\nthe ball rises and recedes at once — both push its screen position the\nsame way, so there's no cusp to find at all. What does exist: the same\nshadow-speed exaggeration from before. Airborne, the far shadow races at\nwildly inflated speed; the instant the ball lands, that exaggeration\nvanishes and the shadow slows near the truth. That collapse marks the\nbounce, even where nothing in the picture looks like one.
\nThe first version of that collapse-detector immediately flagged a moment\nmid-rally as a bounce. The video showed the ball sailing over the net, a\nmeter and a half up — nowhere near bouncing.
\nIt took a minute to understand: the collapse isn't instant at ground\ncontact — it unwinds gradually through the descent, fading as the ball\nloses height. An early \"slowing down\" is just the ball still falling.\nThe bounce is the last collapse of a descent, not the first.
\nRe-checking frames after that fix surfaced one more find, and it wasn't\nabout the ball: my own hand-built answer key had quietly stopped early,\nbefore the rally actually ended. The tracked data ran further — one more\nfar bounce, then Gasquet's final swing, both missed on my first pass. Even\na hand-checked answer key needs its own second check.
\n![\"The](\"https://optic-yellow.com/assets/images/m2-events-trajectory-a59188832f8c235af28c0e7812b1c038.webp\")
The finished detector: seven hits, six bounces, all thirteen matching the\nvideo.
\nThirteen events found, checked against video, correctly told apart as hit\nor bounce — zero wrong calls. Because the screen-to-court conversion\nalready exists, each one drops onto an actual scale drawing of the court:
\n![\"Top-down](\"https://optic-yellow.com/assets/images/m2-event-map-2535f60d580cb0aae9e571ff59b39db2.webp\")
Thirteen dots and X's on a court diagram. Solid yellow (near-side\nbounces) are trustworthy to within centimeters. Hollow dots (far-side)\nare only good to within a few meters — the same camera-angle problem as\nabove.
\nThe near-side bounces landed at 19.4, 19.6, and 20.4 meters down the court\n— about three and a half meters inside the baseline, exactly where pro\ngroundstrokes land. Nobody told the software that; it just fell out of the\nmath.
\nRead the sequence out loud: hit, bounce, hit, bounce, hit — a rally,\ndescribed shot by shot. No shot types or directions yet, but for the\nfirst time this reads like the start of a match chart, not a vision demo.
\n| What was checked | Result |
|---|---|
| Events found in the rally | 13 (7 hits, 6 bounces) |
| Correctly told apart, hit vs. bounce | 13 of 13 |
| Wrong calls | 0 |
| Near-side bounce landing spots | within centimeters of real groundstroke depth |
| Far-side bounce landing spots | good only to within a few meters |
A hit throws the ball back where it came from; a bounce just lets it keep\ngoing, a little slower. That one difference — reversed or not — is enough\nto read a whole rally's worth of events out of a wobbly, camera-distorted\nflight path. The court doesn't return serve.
\nDifferentiated the M0 track: 290 usable frames, 10 small gaps. Two traps:\nimage-y mixes height with court depth (M0's \"parabolas\" are projection\nmixtures); along-court shadow velocity spikes to ±200 m/s when the ball is\nhigh on the far side.
Detector v1: cusps in vertical velocity, classified by direction flip.\nFound 9 events, called 8 hits. Root-caused by pulling video frames at\nevery event rather than debugging further — frame 44 flagged as a hit\nshowed the ball at Zverev's feet, racket nowhere near it.
Failure 1, near-side bounces classified as hits: post-bounce shadow\nvelocity collapses into noise (-2 to -4 m/s) and its sign flips, fooling\nthe direction test. Fix: classify on outgoing speed, not sign — real\nreturns leave at 5+ m/s (often 40); bounces collapse toward zero.
Failure 2, far-side bounces structurally invisible: post-bounce the ball\nrises and recedes simultaneously, both pushing image position the same\nway — no cusp exists in that coordinate. Fix: shadow-speed collapse —\nairborne shadow runs -30 to -50 m/s of amplified nonsense, on-ground\nshadow runs true speed (~-5).
False positive at frame 71: collapse detector v1 flagged it as a bounce;\nvideo shows the ball crossing the net, 1.5m up. Root cause: collapse\nunwinds gradually through the descent — early collapse candidates are\ndescent-in-progress, not landing. Fix: bounce = last collapse of a\ndescent, not first.
While re-verifying, found my own ground truth had stopped at frame 243\nwhile the track runs to ~298 — one more far bounce, then Gasquet's final\nswing, found by the detector, missed by the manual labels.
Detector v4 (final): 7 hits, 6 bounces, 13/13 matched against\nframe-verified ground truth, 13/13 classified correctly, 0 false\npositives. Near-side bounce depths: 19.4, 19.6, 20.4m — ~3.5m inside the\nbaseline, consistent with pro groundstroke depth.
Caveats: thresholds tuned to one clip, generalization untested. Far-side\npositions good only to a few meters — at the far baseline, one frame of\ntiming error is meters of shadow, and the ball is already moving again by\nthe next frame. Near-side positions look genuinely good.
Running total across the first three builds — an object tracker, a\nscreen-to-court conversion, a frame-verified event detector — is still 60\ncents, all spent on the first one. The other two were just math run over\ndata already collected.
\nNext: turning this sequence of events into an actual chart — splitting\nfootage into rallies, figuring out shot direction, and writing it all out\nin the Match Charting Project's own notation. That's the point where this\nstops being a computer-vision project and starts being a tennis project.
", "url": "https://optic-yellow.com/blog/m2-hits-and-bounces", "title": "M2: The Court Doesn't Return Serve", "summary": "Teaching software to spot every hit and bounce in a rally, just from watching the ball's path bend.", "date_modified": "2026-07-09T21:00:00.000Z", "author": { "name": "Trevor McCormick", "url": "https://github.com/trevormccormick" }, "tags": [ "Devlog", "Computer Vision" ] }, { "id": "https://optic-yellow.com/blog/m1-pixels-to-meters", "content_html": "That clip up top: the yellow box tracking the ball on the broadcast\npicture, next to the same rally redrawn as dots on a plain, top-down\ncourt diagram — the ball's real position, in meters, drawn live as the\nrally plays out.
\nLast time, the software could hold onto a tennis ball for an entire\nrally on screen. Today's question: can it tell you whether that ball\nlanded in the court, or a foot wide of it?
\nPreviously: the kickoff post set the\ngoal — teach software to watch a broadcast match and produce the\npoint-by-point notes volunteers hand-type for the Match Charting\nProject, a slow,\nhuman-powered dataset the project wants to speed up. Last\ntime, an AI followed one specific ball\nthrough a whole rally for about sixty cents. What came back was a list\nof screen positions — pixel coordinates, the colored dots a video frame\nis built from (more in Court Vision in Plain\nEnglish). A screen position alone can't say\nwhether a shot landed in or well wide.
\nYou already do this without noticing. Watch a match on TV, and when a\nball lands, you can point to roughly the right spot on a paper court\ndiagram — even though the camera sits up high behind the baseline, so\nthe far side of the court looks small and squeezed while the near side\nlooms huge. Your brain quietly un-squeezes it. That un-squeezing rule —\nturning \"this spot on the TV screen\" into \"this spot on the court\" — is\nwhat computer vision calls a homography, the screen-to-court-diagram\nconversion (more on it here). Software needs the\nrecipe spelled out, and it gets one the way a chair umpire settles an\nargument: from the court's own painted lines. Match what the camera sees\nto the real dimensions — every court on earth is 10.97 by 23.77 meters —\nand the conversion falls out.
\nOne catch to tuck away first: this recipe only works for things sitting\non the ground. It converts the court's flat surface, not the air\nabove it. A player's feet are on the ground, so their position converts\nhonestly. A ball in flight isn't touching the ground, so the same recipe\nreports where the ball's shadow would be — not the ball. Hold onto\nthat; it's the twist waiting at the end of this post, and it echoes\nthrough the rest of the series.
\nFinding painted lines is easier once you clear the players off them. The\ntrick: the broadcast camera holds still for the whole rally (checked in\nthe appendix). Stack every frame of the clip, and for each dot on\nscreen, take the middle value it shows across every frame — its median.\nA player is only ever briefly at any one dot, so the middle value votes\nthem out of existence. The lines never move, so they survive razor\nsharp. It's the same free trick the Plain English\nprimer uses to explain how the\npipeline later finds players — here it just clears the stage.
\n![\"The](\"https://optic-yellow.com/assets/images/m1-clean-plate-5be255aa56d73b6e0377ac52bb4670af.webp\")
A tennis court with nobody playing tennis on it — built by stacking\nframes that all had somebody on it.
\nOne line of code, and both players vanish, the ball vanishes, the lines\nstay razor sharp. A couple of line judges, who barely move, leave faint\nghosts, and the on-screen graphics stick around too — neither matters\nfor what's next.
\nCourt lines are white paint on a blue court. So: isolate the big blue\nrectangle first, and only look for white dots inside it. That one rule\nkeeps the crowd's white shirts, courtside signage, and the scoreboard's\ntext out of the search entirely; the blue court fills about a quarter of\nthe frame.
\nRun a standard line-finding routine on what's left, and 48 short\nsegments come back, covering every painted line on the court.
\n![\"The](\"https://optic-yellow.com/assets/images/m1-lines-overlay-b9e58a320f07348b7934c42272475945.webp\")
Every line the court owns — plus one it doesn't.
\nEvery real line, except the search also grabs an impostor: the net\ncord reads as a line too. It's white, roughly horizontal, and\nstretches most of the way across the frame — but it isn't painted on\nthe ground, and it isn't even straight; a net sags under its own\nweight. Any scheme that just grabbed the horizontal lines would trip\nright over it.
\nThe fix is almost embarrassingly simple. Group the found lines into\nfamilies — side-to-side, toward-the-camera — and keep only the\noutermost member of each: the top and bottom side-to-side lines are the\nbaselines, the left and right toward-the-camera lines are the outer\nsidelines. The net sits in the middle by definition, so it can never be\nan outer line; it's excluded automatically. Cross those four outer\nlines and you get the court's four corners. Hand those four points, plus\nthe court's real dimensions, to a standard piece of geometry software,\nand out comes the homography.
\nFour corners is about as little as a homography can be built from. Is\nit any good? The honest test: three pairs of lines never went into\nbuilding the conversion at all — the singles sidelines, both service\nlines, and the center line. Run the full court diagram back through the\nconversion and see where those untested lines land.
\n![\"The](\"https://optic-yellow.com/assets/images/m1-reprojection-1ab943adf321df9ba71a432345c2732b.webp\")
Yellow: the lines the conversion was built from. Orange: lines it had\nnever seen before this moment — landing right on the paint anyway.
\nThey land almost exactly on the paint, off by less than one screen dot\nevery time. From four corners, on the first try. I had a refinement step\nplanned in case the numbers came back rough, and deleted it.
\nNow the payoff: take last post's ball positions — one screen dot per\nframe — and run every one through the new conversion, then plot them on\na true-to-scale court diagram.
\n![\"Top-down](\"https://optic-yellow.com/assets/images/m1-track-court-7cada45bda460aed1f3c2200989ac462.webp\")
The same rally, seen from directly above for the first time. Color\nmarks time.
\nMost of it reads like tennis: the ball crossing the net, dots clustering\nlow near the baselines where players make contact. Some of it doesn't —\na handful of points land more than 18 meters past the far baseline, well\noutside any real court. Not a fit error; the untested lines above just\nproved this conversion is accurate to a fraction of a screen dot. It's\nthe catch from earlier, on schedule: the conversion only knows the\nground, and a ball spends most of a rally in the air. What this chart\ndraws, for every airborne moment, is the ball's shadow, not the ball.\nThe shadow is truthful at exactly one instant — when the ball touches\ndown — and drifts further from reality the higher it flies.
\nWhich hands the next post its whole job: find the moments the ball\ntouches the ground — the bounces — because those are the only moments\nthis chart isn't lying.
\n| Check | What happened | What \"good\" looks like |
|---|---|---|
| Camera holding still all rally | Stacked \"empty court\" photo lined up with frame one almost perfectly | Should land right on top |
| Lines found automatically | 48 segments, every real court line — plus the net cord, caught and thrown out | — |
| Accuracy on lines never used to build the fit | Off by well under one screen dot | Under ~1 dot = dead-on |
| Human clicks / cost | Zero clicks, $0.00 | — |
The ball — tennis's famous hard problem — needed a paid AI and a human's\nmouse click to follow. The court — the part that sounds like real\ncomputer vision — needed four corners and free arithmetic. By the end\nof the day, the hard problem and the easy problem had traded chairs.
\nnp.median(frames, axis=0).This step cost nothing — no clicks, no paid API calls, just code on my\nown laptop. Total for the milestone: $0.00.
\nNext question: the ball's path on screen is smooth curves broken up by\nsudden kinks. Is every kink a hit or a bounce? Time to find out if that\nidea survives contact with real data.
", "url": "https://optic-yellow.com/blog/m1-pixels-to-meters", "title": "M1: From Pixels to Meters", "summary": "How four painted corners taught the software to turn a TV picture of the court into a real, to-scale court diagram — for free.", "date_modified": "2026-07-09T18:00:00.000Z", "author": { "name": "Trevor McCormick", "url": "https://github.com/trevormccormick" }, "tags": [ "Devlog", "Computer Vision" ] }, { "id": "https://optic-yellow.com/blog/m0-first-tracked-rally", "content_html": "That clip up top: an AI following the game ball through a full broadcast rally. The box is the model's answer, not something I added by hand.
\nBefore software can chart a single stroke of tennis, it has to do the\nsimplest thing a ball kid does without thinking: watch the right ball, the\nwhole rally through.
\nShort answer: yes, it can — with a catch, and getting there took four tries\nand two dead ends. Here's how.
\nThe kickoff post laid out the plan: teach\nsoftware to watch broadcast video and produce the same shot-by-shot chart\nthat Match Charting Project volunteers write by hand today — who served\nwhere, how the rally went, who won the point. This post is step one:\nbefore anything smarter can happen, the software has to prove it can keep\nits eyes on one ball.
\nPicture the court between points: the ball actually in play, plus every\nspare — tucked in ball kids' hands, a player's pocket, by the net post.\nShout \"watch the ball!\" and anyone would reasonably ask which one. That's\neffectively the instruction the AI got on the first attempt: the words\n\"tennis ball.\" It's what's called a promptable model — instead of\ntraining it on tennis ahead of time, you just tell it, in the moment,\nwhat to look for. Told to find \"tennis ball,\" it found all of them, game\nball included, and drew one box around the whole group. That's matching a\ndescription, fresh, everywhere it fits — not following one ball.
\nThe fix: point instead of describe. Draw one box, once, around the one\nball actually in play, in a single picture. That's a different kind of\ninstruction — not \"find anything that looks like this\" but \"follow this\nexact object\" — and the model carries that one ball forward through the\nvideo, remembering which ball is the right one from picture to picture.\nThat's called tracking: locking onto one specific object and\nfollowing it through the clip, rather than spotting fresh matches from\nscratch each frame. (More on the difference, with more tennis examples:\nCourt Vision in Plain English.) A text\nprompt matches every instance of a concept in the picture; a box drawn\naround one specific ball tracks that ball, and only that ball.
\nI grabbed a 16-second broadcast rally — Zverev vs. Gasquet, in Montreal —\nto test that idea on real footage. Finding a usable clip was its own\nsmall project: the first highlight compilation kept cutting away from\nthe rally to close-ups and crowd shots after a few seconds. It took two\ntries to find one continuous shot of a full rally from a single camera.
\nI asked the AI to find \"tennis ball, tennis player.\" The picture looked\nright for the first several seconds — clean outlines around both players.\nThen, partway through the clip, it abruptly flipped: instead of the\nplayers, it was now outlining the entire court. The model had lost its\ntrain of thought and locked onto something else entirely. (Why, and\nexactly when, is in the appendix.)
\nDropping the players and asking only for \"tennis ball\" did return one box\nfor every single picture in the clip — 480 of them, which sounded\npromising. Then I plotted where that box actually sat, frame by frame:
\n![\"The](\"https://optic-yellow.com/assets/images/m0-trajectory-textprompt-91a5d42727ce3f4300fb847dbee17400.webp\")
The box the AI called \"the ball,\" over time. Long flat stretches — a\nball in a live rally never sits still like that.
\nA ball in play is never still for long; this chart was, flat stretch\nafter flat stretch. Drawing the box back onto the video explained why: it\nwasn't tracking one ball, it was covering all of them. Broadcast courts\ncommonly have something like six balls in view at once — the game ball\nplus every spare a ball kid is holding — and the AI had drawn one\nenormous box around the entire group, roughly the size of half the court.
\nThe actual game ball — a smudge maybe eight pixels wide, moving at\n200 kilometers an hour, contributing almost nothing to a box that size.
\nNew plan: skip the words, point at the object. I found the game ball by\neye in one frame, drew a box around it, and let the model carry that ball\nforward and backward through the rest of the clip on its own. Most of the\nclip came back with sensible, ball-sized boxes — a real improvement. Then\nthe very first numbers I decoded came out negative, which a box's width\ncan never be. Turned out to be a units mix-up, not a tracking failure\n(full story in the appendix). Read correctly, the numbers lined up\nexactly on the ball I'd pointed at.
\nWith the numbers read correctly, the tracked path finally looked like a\nreal rally:
\n![\"The](\"https://optic-yellow.com/assets/images/m0-trajectory-boxprompt-5b6778ca716df4bfe651033fdbeff24f.webp\")
The game ball's tracked position, after pointing at it in one frame and\nletting the model follow it both ways through the clip.
\nThree clean trips of the ball from baseline to baseline, with a sharp\nreversal in direction at each hit — exactly the shape a ball traces when\nit's actually being struck back and forth. Those reversals are exactly\nthe signal a later step will use to work out precisely when each hit\nhappened.
\nTwo surprises. I'd only pointed at the ball in one middle frame, yet the\nmodel filled in its position both backward and forward on its own. And\ntrack death has a clean tell: once the ball leaves play, the tracked box\ndoesn't vanish — it freezes in place and balloons to several times its\nnormal size. A dead track looks like a frozen, oversized box, as readable\na signal as a late line call. (Exactly where, and the fix, is in the\nappendix.)
\n![\"A](\"https://optic-yellow.com/assets/images/m0-boxframe-8bd16a9b987939c01b599afedd957f40.webp\")
One frame from the rally, tracked path drawn in — the box sits on the\ngame ball, not a ball kid's spare.
\nFirst milestone: done, at least for the ball — which was the part I\ngenuinely doubted going in.
\n| What was tried | What happened |
|---|---|
| Describe the ball in words | Locked onto every ball on court as one giant, motionless box |
| Point at the one ball with a box | Tracked cleanly for 470 of 480 frames — about 98% of the clip, essentially the whole rally |
| Total cost, all four tries combined | About $0.60 — dead ends included |
Two things are still open, not quietly skipped. Players aren't tracked as\ncleanly as the ball yet — the same fix should work, but that's a guess,\nnot a result. And the box is still hand-drawn: a person clicks the ball\nonce per rally to start the pipeline. Workable, but not the automation\nthis project needs. Both, in depth, are in the appendix below.
\nTell an AI to watch \"the tennis ball\" on a broadcast court and it will\ndutifully watch every ball in the picture at once. Point at the one that's\nactually in play, and it will follow just that one, the whole rally\nthrough.
\nClip and tools. 480 frames, 720p, 30fps, via yt-dlp. Compute:\nfal.ai's hosted SAM 3 endpoints —\nsam-3/video (masked overlay\nvideo) and sam-3/video-rle\n(RLE — run-length-encoded — masks plus per-frame boxes). About half a\ncent per 16 frames, no GPUs to manage.
Attempt 1 detail. Prompt \"tennis ball, tennis player\", both\nendpoints. Player segmentation looked clean for ~8 seconds — correctly\ncatching the line judges and ball kids too, since they're dressed like\nplayers. Around frame 240 the mask flipped to segmenting the entire\ncourt. Logs: Completed chunk 2/2. fal processes video in chunks; the\nsecond chunk appears to re-run concept detection from scratch and land on\na different match — chunk boundaries are concept-drift boundaries. The\nRLE endpoint, same prompt: one box per frame, a single giant region\nmerging everything matched, every confidence score null.
Attempt 2 detail. Prompt \"tennis ball\" alone, detection threshold\n0.3. Returned 480 boxes (one per frame), each the union of every tennis\nball visible in frame — game ball contributing almost nothing to a box\nthat size.
Attempt 3 detail. Game ball located by eye in frame 240, pixel (622,\n462); box prompt anchored there. 470/480 frames returned boxes. The\nnegative widths on first parse were a units bug, not a tracking bug:\nbox-prompted responses come back as [cx, cy, w, h] — normalized\ncenter-plus-size — not the [x1, y1, x2, y2] corners the text-prompt\nresponses use. Documented nowhere, as far as I can tell; found by staring\nat impossible numbers until the shape of the mistake became obvious.
Attempt 4 detail. Frames 0–290 returned ball-sized boxes tracing\nclean flight arcs, propagated in both directions from the single\nframe-240 prompt. Track death: box collapses to a frozen, oversized shape\nonce the ball leaves play around frame 300. Dropping any box wider than\n0.05 (normalized) cleaned that up entirely.
Open items. Player tracking: original scope was \"ball and both\nplayers\"; attempt 1's player masks were promising until the chunk-boundary\nbug ate them, and it wasn't revisited once the ball was solved. Players\nare bigger, slower, and hundreds of pixels tall, so box prompts should\nhandle them too, but that's untested. The human-drawn box: still one\nmanual click per rally to seed the track. Candidate fix, for later:\ntext-prompt for every ball in frame, then keep whichever one is moving.
All in: about sixty cents, four tries, two dead ends. Next question: now\nthat the pipeline can find the ball in a picture, can it work out where\nthat ball sits on the court itself? A different kind of problem — less\nabout winning an AI's cooperation, more plain geometry homework.
", "url": "https://optic-yellow.com/blog/m0-first-tracked-rally", "title": "M0: Can SAM 3 Hold Onto a Tennis Ball?", "summary": "An AI that finds objects by description can't tell one tennis ball from six on court — until you point at the one in play.", "date_modified": "2026-07-09T12:00:00.000Z", "author": { "name": "Trevor McCormick", "url": "https://github.com/trevormccormick" }, "tags": [ "Devlog", "Computer Vision" ] } ] }