Teaching a Computer to Watch Tennis

A courtside volunteer can watch a tennis match and write down every shot by hand — serve, rally, how the point ends — in a shorthand built just for that job. I want to find out if a computer can learn to write the same shorthand from nothing but the broadcast video. This devlog is where I write down what happens, including the parts that don't work.

This is the first entry in a series called Court Vision — a public devlog, not one write-up at the end, because projects I don't write about are projects I quietly abandon, and the wrong turns are usually more interesting than the wins. Expect short entries, real artifacts (clips, plots, dead ends included), and no pretending I already know how this turns out. Recurring ideas get collected once, in tennis terms, at Court Vision in Plain English.

The shorthand

Start with what a Match Charting Project volunteer actually does. They watch a match, live or on tape, and type out every shot as it happens — not in full sentences, because no one types that fast, but in a shorthand where a few characters stand in for a whole shot. Think of it as a tennis version of a chess player's move notation, except each entry is a whole shot instead of a whole move.

Here's what one point looks like once it's written down: s5b2f1f2f2x@. Six shots, six little chunks of code, one line of text. Decoded, chunk by chunk: s5 is a serve into the body. b2 is the backhand return, back up the middle. f1 is a forehand into an outer third of the court. f2 is a forehand up the middle. f2 again — another forehand up the middle. And x@ closes the point: the ball lands wide and long at the same time, missing the court in both directions at once, ruled an unforced error, rally over. Six characters, one full point, no ambiguity about what happened.

That code is called Match Charting Project notation — MCP for short — a compact, shot-by-shot shorthand volunteers hand-write for a match, point by point, while they watch it. Thousands of matches are charted this way already, freely available online. It's the reason nearly every deep tennis stat you've ever read exists at all — serve-placement splits, rally-length breakdowns, how often a player misses long versus wide on break point — all traced back to someone typing strings like the one above.

The moonshot this whole devlog is chasing: instead of a person watching live and typing, feed a computer the broadcast video and have it produce that exact same string automatically, for every point — broadcast video in, notation out. If that works even roughly, it attacks the dataset's real bottleneck: not enthusiasm, not tooling, but the plain fact that charting one match costs a volunteer hours of undivided attention. That's why big names and big tournaments get charted, and almost nobody else does.

Why this looks possible now

For years, the honest answer to "why doesn't this exist" was that the vision part was the hard part — you'd need detectors custom built and trained just to find the ball and players reliably, a research project of its own before reaching the charting logic.

Late in 2025, Meta released SAM 3, the third version of their Segment Anything Model, with the trained model published for anyone to use. Give it a video and a plain-English prompt — literally the words "tennis ball" — and it finds, outlines, and follows every matching object, frame by frame, no custom training or hand-labeled footage required. The part of this project that used to require a research career is now, on paper, one API call (a request to software running elsewhere).

"On paper" is the catch, and the reason this devlog exists. Broadcast tennis is a rough test: the ball is a handful of pixels moving faster than 200 kilometers an hour, smeared by motion blur; players block each other; the camera cuts between angles mid-point; replays splice in with live points. I don't yet know where this breaks. Finding out is the project.

The plan, in order

Five things have to work, roughly in this order, before broadcast video turns into a finished chart:

1. Follow the ball and both players through one whole point, without losing track of who's who — the first checkpoint, and everything downstream depends on it.
2. Convert "where on your TV screen" into "where on the actual court." The camera views the court at an angle, so the far side looks squashed; a charting diagram is flat and to scale. That conversion is the homography — a recipe built from the court's own painted lines. (More: primer's homography section.)
3. Read hits and bounces out of the ball's flight path. The path is smooth curves broken by sudden changes in direction, and those breaks are exactly where the hits and bounces are. (More: primer's hits-and-bounces section.)
4. Turn that sequence of events into notation — the shorthand decoded above, generated automatically instead of typed by hand.
5. Take the test. Chart a match a human already charted, compare the two point by point, and publish the result — flattering or not. This step matters most: it separates a highlight reel from a real number, grading the machine like a rookie line judge, against calls somebody else already made.

Every deep tennis stat you've ever read started as one person, watching alone, writing shorthand by hand — this project is a bet that a computer can learn to write the same shorthand from tape.

For the technical reader

The plan tracks five milestones, referenced by these labels in later posts:

• M0 — Track one rally. Run SAM 3 on one broadcast rally clip; get clean tracks for the ball and both players. Feasibility gate for everything else.
• M1 — Find the court. Detect court lines, compute the homography, convert pixel positions into real court coordinates.
• M2 — Detect hits and bounces. A ball's trajectory is smooth arcs interrupted by discontinuities; every discontinuity is a hit or bounce — the raw material of charting.
• M3 — Chart points. Segment rallies, classify shots, emit MCP notation.
• M4 — Take the test. Chart a match a human volunteer already charted, diff the two, publish the agreement rate — whatever it is.

First question: can the tracking even hold onto the ball at broadcast speed, frame after frame, without losing it? That's what the next entry finds out.