M0: Can SAM 3 Hold Onto a Tennis Ball?
That clip up top: an AI following the game ball through a full broadcast rally. The box is the model's answer, not something I added by hand.
Before software can chart a single stroke of tennis, it has to do the simplest thing a ball kid does without thinking: watch the right ball, the whole rally through.
Short answer: yes, it can — with a catch, and getting there took four tries and two dead ends. Here's how.
The kickoff post laid out the plan: teach software to watch broadcast video and produce the same shot-by-shot chart that Match Charting Project volunteers write by hand today — who served where, how the rally went, who won the point. This post is step one: before anything smarter can happen, the software has to prove it can keep its eyes on one ball.
Picture the court between points: the ball actually in play, plus every spare — tucked in ball kids' hands, a player's pocket, by the net post. Shout "watch the ball!" and anyone would reasonably ask which one. That's effectively the instruction the AI got on the first attempt: the words "tennis ball." It's what's called a promptable model — instead of training it on tennis ahead of time, you just tell it, in the moment, what to look for. Told to find "tennis ball," it found all of them, game ball included, and drew one box around the whole group. That's matching a description, fresh, everywhere it fits — not following one ball.
The fix: point instead of describe. Draw one box, once, around the one ball actually in play, in a single picture. That's a different kind of instruction — not "find anything that looks like this" but "follow this exact object" — and the model carries that one ball forward through the video, remembering which ball is the right one from picture to picture. That's called tracking: locking onto one specific object and following it through the clip, rather than spotting fresh matches from scratch each frame. (More on the difference, with more tennis examples: Court Vision in Plain English.) A text prompt matches every instance of a concept in the picture; a box drawn around one specific ball tracks that ball, and only that ball.
The setup
I grabbed a 16-second broadcast rally — Zverev vs. Gasquet, in Montreal — to test that idea on real footage. Finding a usable clip was its own small project: the first highlight compilation kept cutting away from the rally to close-ups and crowd shots after a few seconds. It took two tries to find one continuous shot of a full rally from a single camera.
Attempt 1: describe the ball and the players together
I asked the AI to find "tennis ball, tennis player." The picture looked right for the first several seconds — clean outlines around both players. Then, partway through the clip, it abruptly flipped: instead of the players, it was now outlining the entire court. The model had lost its train of thought and locked onto something else entirely. (Why, and exactly when, is in the appendix.)
Attempt 2: describe just the ball
Dropping the players and asking only for "tennis ball" did return one box for every single picture in the clip — 480 of them, which sounded promising. Then I plotted where that box actually sat, frame by frame:
The box the AI called "the ball," over time. Long flat stretches — a ball in a live rally never sits still like that.
A ball in play is never still for long; this chart was, flat stretch after flat stretch. Drawing the box back onto the video explained why: it wasn't tracking one ball, it was covering all of them. Broadcast courts commonly have something like six balls in view at once — the game ball plus every spare a ball kid is holding — and the AI had drawn one enormous box around the entire group, roughly the size of half the court.
The actual game ball — a smudge maybe eight pixels wide, moving at 200 kilometers an hour, contributing almost nothing to a box that size.
Attempt 3: point at it instead
New plan: skip the words, point at the object. I found the game ball by eye in one frame, drew a box around it, and let the model carry that ball forward and backward through the rest of the clip on its own. Most of the clip came back with sensible, ball-sized boxes — a real improvement. Then the very first numbers I decoded came out negative, which a box's width can never be. Turned out to be a units mix-up, not a tracking failure (full story in the appendix). Read correctly, the numbers lined up exactly on the ball I'd pointed at.
Attempt 4: the payoff
With the numbers read correctly, the tracked path finally looked like a real rally:
The game ball's tracked position, after pointing at it in one frame and letting the model follow it both ways through the clip.
Three clean trips of the ball from baseline to baseline, with a sharp reversal in direction at each hit — exactly the shape a ball traces when it's actually being struck back and forth. Those reversals are exactly the signal a later step will use to work out precisely when each hit happened.
Two surprises. I'd only pointed at the ball in one middle frame, yet the model filled in its position both backward and forward on its own. And track death has a clean tell: once the ball leaves play, the tracked box doesn't vanish — it freezes in place and balloons to several times its normal size. A dead track looks like a frozen, oversized box, as readable a signal as a late line call. (Exactly where, and the fix, is in the appendix.)
One frame from the rally, tracked path drawn in — the box sits on the game ball, not a ball kid's spare.
The verdict
First milestone: done, at least for the ball — which was the part I genuinely doubted going in.
| What was tried | What happened |
|---|---|
| Describe the ball in words | Locked onto every ball on court as one giant, motionless box |
| Point at the one ball with a box | Tracked cleanly for 470 of 480 frames — about 98% of the clip, essentially the whole rally |
| Total cost, all four tries combined | About $0.60 — dead ends included |
Two things are still open, not quietly skipped. Players aren't tracked as cleanly as the ball yet — the same fix should work, but that's a guess, not a result. And the box is still hand-drawn: a person clicks the ball once per rally to start the pipeline. Workable, but not the automation this project needs. Both, in depth, are in the appendix below.
Tell an AI to watch "the tennis ball" on a broadcast court and it will dutifully watch every ball in the picture at once. Point at the one that's actually in play, and it will follow just that one, the whole rally through.
For the technical reader
Clip and tools. 480 frames, 720p, 30fps, via yt-dlp. Compute:
fal.ai's hosted SAM 3 endpoints —
sam-3/video (masked overlay
video) and sam-3/video-rle
(RLE — run-length-encoded — masks plus per-frame boxes). About half a
cent per 16 frames, no GPUs to manage.
Attempt 1 detail. Prompt "tennis ball, tennis player", both
endpoints. Player segmentation looked clean for ~8 seconds — correctly
catching the line judges and ball kids too, since they're dressed like
players. Around frame 240 the mask flipped to segmenting the entire
court. Logs: Completed chunk 2/2. fal processes video in chunks; the
second chunk appears to re-run concept detection from scratch and land on
a different match — chunk boundaries are concept-drift boundaries. The
RLE endpoint, same prompt: one box per frame, a single giant region
merging everything matched, every confidence score null.
Attempt 2 detail. Prompt "tennis ball" alone, detection threshold
0.3. Returned 480 boxes (one per frame), each the union of every tennis
ball visible in frame — game ball contributing almost nothing to a box
that size.
Attempt 3 detail. Game ball located by eye in frame 240, pixel (622,
462); box prompt anchored there. 470/480 frames returned boxes. The
negative widths on first parse were a units bug, not a tracking bug:
box-prompted responses come back as [cx, cy, w, h] — normalized
center-plus-size — not the [x1, y1, x2, y2] corners the text-prompt
responses use. Documented nowhere, as far as I can tell; found by staring
at impossible numbers until the shape of the mistake became obvious.
Attempt 4 detail. Frames 0–290 returned ball-sized boxes tracing clean flight arcs, propagated in both directions from the single frame-240 prompt. Track death: box collapses to a frozen, oversized shape once the ball leaves play around frame 300. Dropping any box wider than 0.05 (normalized) cleaned that up entirely.
Open items. Player tracking: original scope was "ball and both players"; attempt 1's player masks were promising until the chunk-boundary bug ate them, and it wasn't revisited once the ball was solved. Players are bigger, slower, and hundreds of pixels tall, so box prompts should handle them too, but that's untested. The human-drawn box: still one manual click per rally to seed the track. Candidate fix, for later: text-prompt for every ball in frame, then keep whichever one is moving.
All in: about sixty cents, four tries, two dead ends. Next question: now that the pipeline can find the ball in a picture, can it work out where that ball sits on the court itself? A different kind of problem — less about winning an AI's cooperation, more plain geometry homework.