M3: Who Hit That?

That clip up top: both players outlined and tracked, FOREHAND or BACKHAND called out at every swing in the hitter's color, feet moving live on the court diagram — and along the bottom, the shot-by-shot notation typing itself out.

Call a shot forehand or backhand and you're answering a geometry question: which side of the hitter's body did the ball come off? You can't answer that without knowing where the player was standing — and until now, this project had tracked the ball but never found the players. Too much for one sitting: session one of several, starting with finding the players.

Here's what came before. This project's whole bet: feed it broadcast tennis video, get back the exact notation Match Charting Project volunteers hand-write for real matches — the kickoff post has the pitch. Session one tracked the game ball through a rally for about sixty cents, once shown a box around it. Next, the homography — read off the court's painted lines — turns a TV-picture spot into a court-diagram spot, except anything airborne, which converts to its shadow's position instead. Then hit-and-bounce detection read events out of kinks in the ball's path — a hit reverses it, a bounce doesn't, the court doesn't return serve — producing the first rough chart: ??2?1?1?2?2?2???.

Spot the player by spotting what never changes

Here's the trick, older than computer vision — it's just averaging. Stack dozens of photos of the same stretch of court, one per frame. Add up how often each spot is occupied: spots that barely change are the court itself, net and lines and grass; spots that keep changing are something moving through the scene.

Treat the barely-changing version as a stand-in photo of an empty court, compare every real frame to it, and whatever's different is a player. That's background subtraction. (More in the primer.) This project already ran the same trick once, to erase the players and get a clean court photo for measuring its lines — today, in reverse.

The catch: the trick only knows "moves" or "doesn't move." Stand still long enough on camera and you get folded into "the court" — a line judge at parade rest for a whole rally erases himself as completely as the baseline.

Two boxes, one blob

The plan: draw a box around each player in one frame and let SAM 3 — the promptable "find that object" AI from the primer — track each box through the clip, same trick that caught the game ball in session one.

The two boxes, drawn by eye and checked before spending real money. The line judge, in red, stands just outside the far box — deliberately left out.

Two boxes went in. One came back — a single rectangle per frame, the union of both players, stretched wide enough to swallow the line judge behind Gasquet too. Same lesson as session one: "tennis ball" once matched every ball; now two boxes merged into one.

Useless boxes, but the response also carries the raw shape data behind them — an outline of what the AI thinks belongs to something. Split it apart wherever it stops touching itself, and every frame resolves into three shapes: Zverev, Gasquet, the line judge.

Two of those are players. Finding the third is background subtraction again: across the rally, add up how often each spot is covered by a shape, as a percentage. Anything covered more than 85% of the time belongs to someone who never left — the judge, at parade rest throughout. Subtract it, and two players are left.

The one figure the AI outlined that never moved. Noticing that is the entire trick for telling a line judge from a player.

He erased himself. Nobody taught the software what a line judge looks like. The only rule: if it doesn't move, it isn't playing.

Both players, the whole rally, no gaps

What's left splits into near player and far player, by which half of the frame each shape's feet land in.

Both players, every frame of the rally. The two short flat patches around frames 390 and 425 are the tracker briefly wandering onto another far-side figure, both after the rally had ended, but on the record.

Result: both players tracked across all 480 frames, zero gaps. The ball itself, tracked in session one, only managed 470 of 480, with ten dropouts — easier for this AI than a small, fast ball.

Small twist on an old caveat: the homography only tells the truth for things touching the ground — an airborne ball converts to its shadow's position, not its own. Feet are always on the ground, so a player's position comes out exact, every frame. The most trustworthy dots in this project belong to the players now, not the ball.

Who hit that?

With both players placed, deciding who struck a ball has three checks: which player was closest at the hit; which end of the court it was struck from, the same read used last session for hits and bounces; and the simple fact that a rally alternates — whoever didn't hit last, hits now. All three agreed, on all seven hits — three signals nobody tuned for this question, landing on the same answer anyway.

Forehand or backhand? It's just geometry

Once a player's position is trustworthy, calling the shot needs no more AI — it's geometry. Ball on the right of a right-hander at contact is a forehand, on the left a backhand — the same call from the couch.

One wrinkle: the camera-facing player has left and right flipped, so his measurement gets mirrored. One liability: handedness is assumed, not detected — a left-handed player would get every call flipped, a debt the series collects later.

The verdict: forehand, forehand, forehand, backhand, forehand, forehand, backhand — seven shots, seven calls, checked by pulling the video frame for every swing, the eyes-over-math discipline that saved last session.

Shot seven, the rally's last swing: Gasquet's one-handed backhand, racket coiled high before he sweeps through it. If you're going to eyeball-check a backhand, make it his.

Seven for seven — but one deserves an asterisk. On shot three, the ball crossed so close to Gasquet's body that the measurement came out just 1.5 pixels to the forehand side — a coin-flip margin. The video settles it: a low pickup off his right shoe, unmistakably a forehand. Right, but only because the coin landed the correct way up.

The chart gets its letters

The two versions of this rally's chart, side by side:

before:  ??2?1?1?2?2?2???
after:   ?f2f1f1b2f2f2b??

Read the finished string left to right:

• The first ? is the serve — unknown, since this clip never shows one.
• f2 f1 f1 b2 f2 f2 b are the seven shots, in order: forehand, forehand, forehand, backhand, forehand, forehand, backhand — this session's addition. The digits (settled last session) mark which third of the court each shot crossed into. Shot seven has no digit — it ends the point, coded differently.
• The last two ? marks are where the ball landed and how the point ended — also missing from this clip.

The chart also gained something the earlier version couldn't show: where each player was standing when they hit.

Seven arrows — who hit it, which wing, from where, to where. Dashed arrows land on the far side, still accurate only to within a meter or so.

Zverev played right around his baseline; Gasquet, more than three meters behind his — court positioning anyone who's watched him play would recognize.

The scoreboard

What we checked	What happened	What "good" looks like
Both players tracked, every frame	480 of 480 frames, zero gaps	The ball itself only managed 470 of 480, with ten short dropouts
Who hit each shot, three ways	All three signals agreed, on all seven hits	Independent checks landing on the same answer, unprompted
Forehand vs. backhand, checked by eye	7 of 7 correct	Every one of the seven swings verified against actual video

The takeaway

Stand still long enough on a tennis court, and even a computer starts to think you're part of the furniture.

For the technical reader

• The box-merge failure. SAM 3's two-box prompt returned one unioned bounding box per frame, not two — same "prompts fuse into one concept" behavior as the text prompts in session one; per-object structure doesn't survive a single API call. The box's right edge landed at roughly pixel 807, exactly far enough to include the line judge. The response does include the raw run-length-encoded (RLE) masks — space-separated start/run pairs over the flattened frame, undocumented like the box format. Decoded and split into connected components, each frame resolves into three spatially disjoint blobs (Zverev, Gasquet, line judge); nobody occludes anybody else in image space.

• The static filter. For every pixel, the fraction of the 480 frames it's covered by any mask (np.mean over the stacked binary masks, time axis). Anything "on" over 85% of the time is treated as static and subtracted from every frame's mask before splitting the rest into players — the same averaging that built the player-free court photo for the homography, run in reverse.

• The forehand/backhand formula. Shot type = sign of (ball x-position minus striker's blob-center x-position) at contact, mirrored for the far player. Handedness is assumed, not detected. Shot 3's raw offset was −1.5 pixels — inside the noise floor of a single pixel of foot-tracking error, essentially arbitrary before video confirmed it; needs a confidence threshold and a bodyline policy before an untuned rally. Position precision differs by side: near-side (Zverev, whose three shots came from 0 to 1.3 meters behind his baseline) converts with high confidence; far-side (Gasquet) is in meters-per-pixel territory, so his "more than three meters behind the baseline" is directionally solid, not precise.

• Session accounting. One paid API call (the two-box SAM 3 prompt); everything after ran locally at no cost.

Cost, and the next gap

Session cost: $0.15 — one API call, everything after it just arithmetic on numbers already in hand. Running total: about seventy-five cents.

The remaining question marks didn't move — the serve, where the ball landed, how the point ended, all still blank, since a mid-rally highlight can't show a point starting or ending. Next session: better source footage — full match video, complete points, camera cuts and all.

The chart still reads ?f2f1f1b2f2f2b?? — seven question marks fewer than it started with, three still to go. Time to earn the rest of the punctuation.