The Machine Grows Ears

That clip up top: a 15-shot rally from the Roland Garros final, with the soundtrack drawn as a strip along the bottom. Green ticks mark the moments the video pipeline detected a hit. Yellow spikes mark the moments the microphone heard an impact. The red zones are the seconds where the ball tracker recorded nothing at all — and the yellow spikes keep firing right through them.

Today the software gets a second sense: when its eyes blink, can its ears carry the point?

Previously. The ball tracker — the tiny specialist network that finds the ball in every frame (the zero-dollar specialist) — has one chronic weakness: it blinks. On clay especially, its track is full of holes, and every downstream call (hits, bounces, rally length, wide-or-deep) inherits the blindness. Last post, physics recovered the landing spots the eyes never saw. This post goes after the holes themselves, with a witness that doesn't have eyes at all.

The witness that never blinks

Think about how you experience tennis from the cheap seats, or the next room. You can follow the rhythm of a rally without seeing it: pock — pock — pock. The strike of strings on a ball is one of the most distinctive sounds in sport — sharp, short, and unlike anything else on a tennis court. Every broadcast microphone picks it up, no matter where the ball is, how small it looks on screen, or whether it just left the frame. Crowd roar rises and falls slowly; commentary murmurs along; the pock is a spike.

That's the whole idea: onset detection — scan the soundtrack for sudden bursts of energy in the frequencies where racquet impacts live, and mark each one's exact moment. No AI, no training: the software slices the sound into 11-millisecond snippets, watches how much new energy each snippet brings, and flags the spikes that tower over their neighborhood. Notice the unit: 11 milliseconds. A video frame is 33. The ears, when they're right, are three times more precise than the eyes can ever be.

First problem: the tapes were silent

One discovery before any listening could happen: every video this project has ever downloaded is silent. The staging pipeline grabbed video-only streams from day one — it never needed sound until today. So the soundtracks had to be fetched fresh, audio-only, from the same uploads. For the Roland Garros match the source was known. For the US Open match it had to be re-found — and verified the way this project verifies everything: by receipt. The candidate video's length matched the match reel to a tenth of a second, and then the real proof: after lining up the two recordings, a single fixed time-shift (95 milliseconds on one match, 310 on the other) snapped hundreds of audio spikes onto video-detected hits at once. A wrong recording can't fake that — there is no single shift that aligns someone else's match.

What the ears heard

Two matches, both already graded against human charts: the US Open final (hard court, where the eyes work well) and the Roland Garros final (clay, where they don't).

Question	US Open (hard)	Roland Garros (clay)
Video hits confirmed by an audio spike within 2 frames	282 of 645 — about 4 in 9	37 of 76 — about half
...within 3 frames	57%	66%
Impacts heard during the eyes' blind time	156 events in 67 seconds	196 events in 78 seconds

Read that last row again, because it's the headline. Across the clay match, the ball tracker spent 78 seconds of rally time seeing nothing — and the microphone logged 196 impacts inside exactly those seconds. The witness that never blinks works best precisely where the eyes fail worst. That's what the yellow-spikes-in-red-zones moments in the clip up top are showing.

And a small piece of physics showed up in the data, unasked. When the detector is set strict — only the loudest spikes — the number of impacts per shot settles at almost exactly two. Two impacts per shot is precisely what a court hears: the racquet's pock, then the bounce's thud. The microphone is telling us it hears both, cleanly enough to count.

The honest limit

Set looser, the ears hear too much. A tennis broadcast is full of things that go pock: line calls, grunts, the umpire's microphone thump, a ball kid's bounce, applause spikes. At the settings that catch the most real hits, the detector logs about three impacts per shot — the extra one is chatter. Asked to count rally lengths by ear alone (spikes divided by two), it lands within one shot of the human chart only 3 times in 10 on hard court, worse on clay.

So the verdict, stated precisely: audio v1 is a witness, not a counter. It confirms moments and fills blind time with candidate events; it can't yet tell a forehand from a line call. Teaching it the difference — pock versus thud versus everything else — is a small classification job, and the training data is free: thousands of audio moments already labeled by the video hits they align with. That's the same flywheel this project keeps returning to — one sense teaching another, no human labeling anything.

The takeaway: the machine's eyes blink, but the court always makes the same two sounds per shot — and now the machine can hear them, timed finer than its own eyes can see.

For the technical reader

The module is experiments/audio_hits.py (report: outputs/diag/audio_hits_report.txt; run with PYTHONPATH=.:experiments uv run python experiments/audio_hits.py). Pure numpy — no librosa/scipy dependency.

Method: mono 22.05 kHz wav (yt-dlp bestaudio of the same uploads; identity by duration fingerprint — g1 5039.40 s vs reel 5039.45 s [known ID]; t6 2412.22 s vs 2412.13 s [condensed-match upload found by search]). STFT 1024/256 hann; log-magnitude spectral flux over 1–8 kHz; normalized by sliding 2 s median/MAD; peaks above a z-score THRESH with 0.10 s minimum separation. A/V offset per match by grid search (±0.5 s, 5 ms steps) maximizing video hits matched within 40 ms: +95 ms (t6, 187/645 matched in search), +310 ms (g1, 34/76).

THRESH protocol: swept on t6 only (the calibration match), frozen before grading g1. Sweep: z=5 → 44% hits matched @2fr, 3.27 onsets/true-shot; z=8 → 38%, 2.53; z=12 → 32%, 2.00 (the racquet+bounce ratio); z=16 → 26%, 1.50; z=20 → 25%, 1.25. Frozen at 5.0 (highest within 5% of peak match rate). Naive rally length (onsets/2, serve-to-last-hit window) within ±1 of MCP truth: 39/128 (30%) t6, 2/21 (10%) g1.

Hole metric: ball-track gaps >3 frames between serve and last hit + 1 s; onsets falling inside them after offset correction: 156 in 67 s (t6), 196 in 78 s (g1).

Staging finds, on the record: (1) extract.py cuts clips with -an and ingest always fetched video-only — the whole corpus is silent by construction; audio staging is now a first-class ingest step. (2) g1's clip cutter (the experiments/-grade score-bug pipeline) never persisted its reel windows; recovered because every clip measures exactly 1.6 s longer than its alignment window (verified on four clips), and the unknown pre/post pad split folds into the calibrated A/V offset. Cutters must persist their windows — noted for the package promotion.

Precedent: audio hit detection on broadcast tennis goes back to APSIPA 2012 (Australian Open footage, F-score 68–77% audio-only); Sony AI's table-tennis impact work (arXiv 2409.11760) reports millisecond precision in controlled capture. Audio-visual fusion gains on broadcast sports: arXiv 2011.04258. The v2 plan (racquet/bounce/ other mel-CNN on video-hit-aligned pseudo-labels) is blueprint roadmap item 8's transcript-alignment flywheel applied to sound. Hero renderer: experiments/render_ears_hero.py (g1_point_05, 15 shots, 4.0 s of holes, 12 onsets inside them).

Session cost: $0.00 — the audio tracks were free, and everything runs on the laptop. Project total: ~$16.

Next question: the ears confirm the eyes and cover their blinks, but both witnesses still hand their evidence to a court fit that a human clicked into place. Four corners, every match, by hand — the last big manual knob on the machine. The soccer world already builds courts that fit themselves and, more importantly, know when they've failed. Time to steal that too.