Stick Figures Beat Smudges

That clip up top: the machine's new view of the players. Yellow stick figures — wrists, elbows, hips, feet — drawn by a free model on real match frames, including the far player at ninety pixels tall. And one frame kept for honesty: the first version confidently drawing its skeleton on a ball kid.

Today the software stops seeing players as smudges and starts seeing swings. The question: does that finally move the forehand-or-backhand call that's been stuck for two weeks?

Previously. The machine finds players by comparing each frame against an empty-court photo (background subtraction) — which yields a moving smudge per player, no arms, no legs. The forehand/backhand letter has been read from which side of the smudge the ball arrives on, and that idea has formally hit its ceiling: careful experiments showed no amount of re-anchoring the smudge changes the calls. Meanwhile this same week the court learned to fit itself and the machine grew ears. The players are the last part still made of fog.

Pose estimation: drawing the swing

The upgrade is a technology every fitness app now uses: pose estimation — a model that marks seventeen body landmarks (wrists, elbows, shoulders, hips, knees, ankles) on any person in an image, turning a smudge into a stick figure. Instead of guessing a stroke from where a blob bulges, you can read the actual tennis: where the racquet wrist is, which side of the body it crossed to, how the shoulders line up at contact. A free, openly-licensed model does this at better than real-time speed on this laptop — no cloud, no cost.

The open question was never the near player, who fills a fifth of the screen. It's the far player, who stands about ninety pixels tall — thumbnail-sized. Research on this exact problem predicted: shoulders and hips will hold up, wrists will get shaky. The audit stills in the clip say the prediction was right on both counts — the far player's skeleton is real and readable, and the model's own confidence in the far wrists hovers around fifty-fifty, versus a solid two-thirds for shoulders.

The head-to-head

Two matches with human answer keys, the two where the letter call is benchmarked: the Paris clay final and the Wimbledon final — grass being the surface where the smudges smear worst. Every method graded on the same shots against the same human charts:

Surface	Smudge rule (shipped)	Skeleton rule
Clay	36 of 44 — 82%	42 of 50 — 84%
Grass	16 of 30 — 53%	23 of 35 — 66%

On clay: a tie, two points better — exactly what the earlier counterfactual experiments predicted, since on clean footage the smudge and the skeleton mostly agree about where a body is. On grass, where the boxes fall apart: thirteen points better. The skeletons help most precisely where the old method fails hardest — the same pattern as the ears last post, and it's not a coincidence: every new witness earns its keep in the old witnesses' blind spots.

The blooper, kept

The first version had a bug that one audit image exposed instantly. It matched each skeleton to the player by picking the detected person nearest the smudge's center — and on one clay point, a rogue smudge box (an old known disease) sat so far from the actual player that the nearest human was a ball kid at the back fence. The machine carefully drew a skeleton on the ball kid and graded his forehand.

The fix is pure tennis sense: a candidate skeleton only counts if its feet are standing in the striker's half of the court — the screen-to-court conversion makes that a one-line check. Ball kids kneel at the fences and net posts; umpires sit outside the lines; players play inside them. The fix costs coverage — with strict standing-room rules and shaky far-player ankles, the matcher now declines about half the shots rather than risk grading the wrong human — and recovering that coverage by tracking skeletons across frames, instead of judging each frame alone, is the obvious next version. Fourth time this week that a rendered picture caught what the accuracy numbers politely hid.

The takeaway: given actual arms to read instead of a fog bank, the machine's forehand-or-backhand call improved most exactly where it was worst — and the day's best evidence was one frame of a very well-postured ball kid.

For the technical reader

Module: experiments/pose_letters.py (report + audit stills in outputs/diag/; run with PYTHONPATH=.:experiments uv run --with rtmlib --with onnxruntime python experiments/pose_letters.py).

Stack: rtmlib (Apache-2.0) Body pipeline (person detector + RTMPose), ONNX Runtime CPU, weights auto-downloaded; ~1 s/frame with detection on this laptop. Posed only at charted contact frames (cheap gate test, not a per-frame rollout).

Protocol: MCP truth letters on length-matched aligned clips (the eval convention); all four players right-handed per match configs. Feature A "wrist-cross": sign of (right-wrist x − shoulder-midpoint x), mirrored for the far player (faces the camera). Feature B "ball-side, pose anchor": ball x at contact vs shoulder midpoint — the re-anchoring control. Results (same-shots blob baseline in parens): t3 — A 37/50 (74%), B 42/50 (84%) vs blob 36/44 (82%); t4 — A 21/35 (60%), B 23/35 (66%) vs blob 16/30 (53%). Wrist-cross underperforming ball-side is expected at contact-frame-only sampling: two-handed backhands and preparation frames confuse a single-frame wrist test; windowed skeletons are roadmap #9's job (the keypoint-graph model, where few-shot literature shows 2-3x gains).

Keypoint confidence (Lsho/Rsho/Lwri/Rwri means): near 0.70-0.74 / 0.50-0.57 wrists; far 0.61-0.65 / 0.45-0.57 wrists — matching the "shoulders hold, wrists shake" prediction for ~90 px players at 720p.

v1→v1.1: nearest-blob matching graded a ball kid (outputs/diag/pose_blooper_ballkid.png, kept); v1.1 requires candidate ankles to project (via the homography) into the striker's court half — far: court-y ∈ [−4, 11], near: [12.8, 28], x ∈ [−1.5, 12.5] — then takes highest mean confidence, blob box demoted to tiebreak. Coverage cost: t3 graded shots 93 → 50 (no fallback when ankles fail the test); temporal tracking recovers this in v2.

Hero renderer: experiments/render_pose_hero.py.

Session cost: $0.00 — openly licensed weights, laptop inference. Project total: ~$16.

Next question: in one day the machine learned to call lines by physics, hear impacts, fit its own court, and read swings — four new witnesses, each stronger where the others are weak. None of them are wired into the shipped pipeline yet; they're experiments with receipts. The next build is the courtroom: the fusion decoder that hears every witness, weighs each one's track record, and writes the chart no single witness could — with the grammar of tennis notation making illegal testimony impossible to write down.