The technical reference

The scorecard says where every number stands. The devlog tells the story in plain English. This page is the third register: a technical account of the computer vision behind Court Vision — written as a working introduction to the field, grounded entirely in this system — followed by a per-component changelog precise enough that another engineer could reconstruct what changed in the model, what it measured, and why that choice was made over its alternatives.

One interpretive rule governs everything: a metric only ever moves for one of three reasons — the model improved, the test got harder, or the ground truth got corrected. All three appear below, labeled.

Everything here has a receipt: a devlog post, the build log (LOG.md), or the research-and-architecture document (blueprint).

Part 1 — The computer vision, through this system​

1.1 Projective geometry: the ground-plane homography​

The broadcast camera and the court diagram are related by a planar homography: a 3×3 matrix H such that p_court ~ H · p_image in homogeneous coordinates, valid for points on the court plane. H has 8 degrees of freedom, so 4 non-collinear point correspondences determine it (solved by direct linear transformation); this is why clicking the court's 4 corners is sufficient, and why the ITF court dimensions (10.97 m × 23.77 m) plus any 4 identified line intersections calibrate the entire ground plane.

Quality is measured by reprojection residuals — project the court model through H⁻¹ and measure the pixel distance to the observed painted lines. The first fit validated at 0.0–0.8 px on held-out lines (singles sidelines, service lines, center line — never used in fitting), which is the correct discipline: residuals on the fitting points alone can be arbitrarily low for a wrong fit (the m1 post).

The structural limitation that shapes half this project: the homography maps the plane, not the space above it. An airborne ball projects to the intersection of its camera ray with the ground — its "shadow" — which overstates depth (motion along the camera axis) far more than width. Two consequences: player feet transform truthfully, and the ball's transform is truthful only at bounce instants. The wide-vs-deep blindness and its fix (§2.7) both follow directly from this geometry.

1.2 Classical structure finding: color masks, morphology, Hough​

The first-generation court fitter is a classical pipeline: (1) isolate the court surface as the largest contour in an HSV color band (per-broadcaster config — the band ported across matches within a broadcaster family); (2) extract thin bright structures with a top-hat morphological filter (image minus its opening — isolates line-width bright ridges); (3) run a probabilistic Hough transform, in which each candidate pixel votes for the line parameters that could pass through it; (4) cluster segments into line families, take family extremes as baselines/sidelines, intersect for corners. The net cord — white, near-horizontal, and sagging — is excluded structurally: it can never be a family extreme.

Two hard lessons came out of this stage. First, fit RMS cannot choose between self-consistent wrong fits; candidate line labelings must be scored against the full observed line mask (this scorer later became the auto-fit judge in §2.1). Second, the classical approach has a surface ceiling: at Roland Garros the far baseline is brushed faint and the stands are near-court-colored, and the fitter failed at 74 px RMS — the event that motivated the learned detector.

1.3 Learned keypoint detection: heatmap regression​

The replacement court detector is a TrackNet-style encoder-decoder that takes a 640×360 frame and emits 15 heatmap channels — one per court keypoint (14 line intersections + center). A keypoint estimate is the argmax of its channel, refined classically. Trained on 8,841 broadcast frames across hard/clay/grass (a public dataset), it reaches ~1.8 px median error on its validation split.

Heatmap regression is the field's standard for small point targets, and the reasons hold here: a heatmap represents spatial uncertainty and multimodality (two plausible locations appear as two modes rather than being averaged into a wrong middle), and dense per-pixel supervision trains better than regressing two coordinates directly.

Fitting from 14 keypoints uses the detector author's configuration search: try 12 predefined 4-point subsets, fit H from each, and select by cross-validated prediction error on the keypoints not used in the fit — model selection by held-out evidence, the same principle as §1.1. Measured against the 8 human fits: 2–16 px landmark agreement on 7 plates, including the clay plate that broke the classical fitter (8 px mean). On the dim night feed only 2/14 keypoints clear confidence, no configuration is complete, and the fitter abstains — behavior we keep, because a silently wrong homography corrupts every downstream number (the cv-20 post).

1.4 Background subtraction and temporal medians​

Player detection is classical: the per-pixel temporal median over a clip's frames is a robust background estimate (each moving object occupies any given pixel only briefly, so the median votes it out), and thresholded frame-minus-background differences yield foreground blobs, one kept per court half. Before any differencing, frames are aligned with ECC (enhanced correlation coefficient) registration, because the "static" broadcast camera measurably drifts — unaligned differencing smears everything (the cv-07 post).

Known failure modes are inherent to the method: anything stationary joins the background (a line judge at parade rest erases himself), anything moving near the court is foreground (ball kids), and a blob is a centroid + bounding box, not a body — which is why stroke classification from blob geometry plateaued (§2.6) and why the pose upgrade exists (§1.7).

1.5 Detection vs tracking, and the specialist ball detector​

The project's most instructive architecture decision: the ball is found by per-frame detection stitched post-hoc, not by a stateful tracker. The detector (WASB-SBDT, BMVC 2023) is an HRNet-backbone heatmap network over short frame stacks, pretrained on broadcast tennis; it answers "where is the ball in this frame?" independently ~30×/s. Stateful trackers carry memory, and memory propagates error: one occlusion or camera cut and a tracker is confidently wrong forever after. A detector cannot lose the plot because it never has one. Measured head-to-head on night-hard footage: the free specialist covered 87% of frames vs 82% for the large promptable segmentation model it replaced — at $0 vs ~$12/match (the cv-12 post).

Post-processing treats the track as evidence with provenance, not truth: physically impossible jumps end a segment (a >3 m/frame court-plane step is ~90 m/s — beyond any real ball; found when a rendered overlay showed a "ball" teleporting 37 m in 3 frames), and holes are recorded rather than silently interpolated, because downstream confidence calibration needs to know which samples were observed.

1.6 Events from signals: trajectory analysis and audio onsets​

Hits and bounces are read from the track's shape: a hit reverses ball direction; a bounce doesn't. The naive implementation — hunting for curvature cusps with tuned thresholds — was replaced by a constraint-first reformulation: in a live rally the ball crosses the net on every shot, so net crossings partition the rally and each partition contains exactly one hit. The fragile question ("is this kink a shot?") becomes a well-posed one ("where within this partition is the shot?"), and rally-length accuracy moved 42% → 61% when the constraint replaced the thresholds (the cv-14 post).

A second, independent event sensor now exists on the bench: audio onset detection. Racquet impacts are broadband transients; the detector computes a short-time Fourier transform (1024-sample windows, 256-sample hop ≈ 11.6 ms at 22.05 kHz), band-limits to 1–8 kHz, takes half-wave rectified log-spectral flux, normalizes by a sliding median/MAD, and peak-picks. The audio and video clocks are reconciled by a single constant per-file offset found by grid search maximizing onset↔hit matches (+95 ms and +310 ms on the two staged matches — and the sharpness of that peak doubles as proof the fetched soundtrack is the same edit as the video). Audio timing is ~3× finer than the 33 ms video frame, and the microphone keeps working during track holes: 196 impacts were logged inside 78 s of clay-court blind time (the cv-19 post).

1.7 Pose estimation​

The stroke-classification upgrade path replaces blobs with top-down 2D pose: detect persons, crop, and run RTMPose (a real-time pose model using coordinate classification rather than heatmap decoding) to get COCO-17 keypoints + confidences per player per frame — CPU-real-time locally. The far player is the acid test at ~90 px tall: measured keypoint confidences hold ~0.62–0.65 at the shoulders and degrade to ~0.45–0.57 at the wrists, matching the literature's expectation that coarse body structure survives low resolution while extremities get noisy — so features lean on shoulders, hips, and ball-relative geometry rather than wrist detail.

Identity assignment is geometric, not appearance-based: a candidate skeleton counts only if its ankle midpoint, projected through the homography, stands in the striker's half within court bounds. This constraint exists because the audit render caught the naive nearest-to-blob matcher grading a ball kid — rogue blob boxes (a known upstream disease) had dragged the match to the wrong human (the cv-21 post).

1.8 From perception to notation: structured prediction​

The end product is a symbol sequence in a formal grammar (per point: serve digit, then letter+direction tokens, then an ending symbol). Two sources of information exist: per-shot pixel likelihoods from the perception stack, and a notation prior trained on the Match Charting Project corpus — 11,646 human-charted matches, 1.85M points — which knows what shots follow what (next-shot side: 72.9% vs a 50.9% base rate, match-disjoint validation).

The architecture decision, made deliberately and recorded with its trigger for reversal: hybrid fusion, not end-to-end. The plan is shallow fusion under grammar constraints — a beam search over the MCP grammar where each candidate token scores log P_prior(token | context) + λ · log P_pixel(token | features), so invalid strings are unrepresentable and every decision decomposes into "what tennis expected" plus "what the pixels saw." The evidence: in speech recognition (the closest mature analog — perception model + language model → symbol sequence), hybrid systems dominate when aligned training data is scarce; end-to-end wins only at large paired-data scale. This project has ~8 fully aligned matches; the closest published end-to-end system (F3ED, trained on 114 annotated broadcast matches with GPUs) still reaches only ~40% exact-event F1 / 75–80% per-element. The pre-registered trigger to revisit: 100+ aligned matches and a hybrid plateau below F3ED's per-element numbers for two consecutive benchmark rounds.

1.9 Evaluation, calibration, and selective prediction​

The task metric is token-level Levenshtein distance against the human chart, with refusals costing exactly what errors cost — the metric a human editor experiences, immune to component-level gaming. Points are "perfect" at ≤1 edit, "usable" at ≤5.

Every learned element is validated match-disjoint: leave-one-match-out (LOMO) for the confidence model, match-disjoint splits for corpus models, and a standing rule that constants are tuned on one named match, frozen, and graded untouched on the rest. The confidence flag is a logistic regression over ~11 pipeline self-signals (track coverage, refusal fractions, crossing consistency, serve plausibility…) plus three mechanistic gates, calibrated LOMO to predict P(≤5 edits): currently 92% precision at 44% coverage on 511 points. A stricter "sign-off" tier has failed held-out validation at every attempted n and is deliberately not shipped. This is selective prediction as product architecture: the system's contract is calibrated abstention — auto-accept what it can guarantee, route the rest to a human — and the same abstention pattern now recurs in the court fitter (§1.3) and the skeleton matcher (§1.7).

Part 2 — The changelog, by component​

Each entry: version history with measured movement, then the decision record. Grades and current numbers live on the scorecard; n = 511 points / 8 matches unless stated.

2.1 Court calibration​

Decision record. Learned keypoints over more classical tuning because the failure was representational (faint lines don't yield corners), and the field's evidence says points-and-lines methods lead broadcast-calibration benchmarks. The proposer/judge split was chosen so the existing hand-tuned scorer demotes from fitter to verifier — no trust transfer to the new net. Open item: on one clay match the net's fit hugs the paint where the human fit is visibly inset — the "truth" may be wrong, and the line-mask judge will arbitrate. License caveat on the public weights (none published): research-only until a self-trained replacement, for which every hand-fit match already auto-generates 14-keypoint labels through its known H.

2.2 Ball​

Decision record. Heatmap-family detector over YOLO-style box regression: every credible cross-dataset result for tiny fast balls comes from the heatmap + temporal-window family, and heatmaps natively express uncertainty. Per-frame detection over stateful tracking: no error propagation (§1.5). The load-bearing surprise, kept on the record: the generalist's track-death habit had been the de facto end-of-point detector, so the "upgrade" initially regressed point segmentation — the bug was load-bearing. Bench: clay fine-tuning (~hundreds of corrected frames) is the planned domain-shift fix; no clay-specific public model exists.

2.3 Point segmentation & alignment to truth​

Decision record. The score bug was chosen as the point ID because it is the only broadcast element with a contract (present, constant within a point, changes at point end). LCS over greedy matching because OCR glitches arrive out of order and only an order-preserving optimal alignment survives deuce recurrences. This automated the last by-eye staging step and is the scaling unblock for corpus training; the reader generalizes across broadcasters, the crop geometry doesn't (per-feed config).

2.4 Serve​

Decision record. Per-surface variants over one universal detector because the failure modes are physical (white ball on white grass lines refuted the ball path on grass — measured at coin-flip and rolled back). Serve zone remains the weakest call (40%): the corpus-trained random forest (31.7% → 59.7% LOMO, top-quartile confidence 71.7%) is proven on the bench and is the wiring priority, because zone is where hand-written geometry hit its ceiling first.

2.5 Shot direction​

Decision record. The jump was a sign fix, found because the errors were systematically mirror-images — a signature worth teaching: consistent mirroring means the measurement works and a convention is flipped. The signal ladder (near-half landing 86% > receiver's next contact 77% > net-crossing extrapolation 67% > far-half landing 47%, each rung measured) commits the best available witness per shot; refusal is reserved for no-witness cases because under the edit-distance metric a 77% guess strictly beats a refusal.

2.6 Shot type (forehand/backhand)​

Decision record. The counterfactual experiment matters as method: three alternative anchors were evaluated against truth before any rebuild, and all failed to move committed calls — proof the ceiling was informational (blobs carry no swing), not parametric. The planned lift is a keypoint-graph classifier over shot windows (skeletons + ball + court nodes), chosen on few-shot evidence: at this project's aligned-data scale, keypoint-graph models beat raw-pixel classifiers ~2–3× on the closest public benchmark, and every input node is a module this pipeline already computes and measures independently.

2.7 Point endings​

Decision record. The reframe follows §1.1's geometry: the landing spot is inferred from flight, never observed — the same principle commercial ball-tracking systems use. Three versions are on the record including both failures found by rendered overlays (a teleport-poisoned fit; a target-half bug inherited from the striker column — the racer now reads direction from the flight itself and trusts no upstream stage). The naive integration (racer outputs as features in a flat 4-class forest) was tested and refuted — base rates dominate because the depth-inflation bias makes in-balls extrapolate out too. Conclusion with architectural consequence: the racer supplies a conditional likelihood P(wide vs deep | out) to the fusion decoder; out-ness itself must come from independent evidence (net-death, observed landing, audio/pixel spotters).

2.8 Confidence & routing​

Decision record. Precision is sacrificed never, coverage first — a promotion rule written down before it was needed. The recurring failure mode is feature blindness: feeds whose diseases are invisible to the self-signals (editor cuts, late ball acquisition) under-deliver, which is why routing thresholds are calibrated per-surface in the roadmap (conformal prediction, giving distribution-free per-group error guarantees, is the planned upgrade).

2.9 Ground truth itself​

Decision record. Truth is versioned, corrections are published rather than silently patched, and experiment exports are hash-frozen so pipeline changes can never regrade a completed human study. The unmeasured ceiling worth stating: MCP charts are single-volunteer labels with no inter-annotator agreement estimate, so an unknown share of the residual gap is label ambiguity — most plausibly on letters and endings.

Part 3 — How choices get made​

1. Constraint-first, then learned, then hand-tuned — in that order. Rules of tennis (crossings partition rallies; players swap ends; balls don't teleport) are free, exact priors. Where constraints run out, train on the corpus. Hand-tuned thresholds are a last resort and are treated as technical debt with a measured ceiling.
2. Prove ceilings before rebuilding. The letter plateau and the serve zone were declared dead by counterfactual experiments against truth, not by frustration.
3. Independent witnesses over stronger single sensors. Physics, audio, keypoints, and priors each earn their place where existing sensors are blind; fusion happens late, with per-source confidence.
4. Abstention is a feature. Court fits, skeleton matches, and point flags all prefer "I don't know, route to human" over silent errors — because a calibrated abstention is priceable and a silent error poisons everything downstream.
5. Render everything. Four bugs in one week (teleporting track, double-scaled coordinates, a lying striker column, a ball-kid skeleton) were invisible in metrics and obvious in one rendered frame each. Every experiment ships with an overlay renderer.
6. Match-disjoint or it didn't happen. Every published number is held-out; tuned matches are practice courts, permanently.
7. External evidence is verified before adoption. The architecture choices above cite research that was adversarially fact-checked (citations and verdicts in the blueprint) — including checks that a recommended model's weights, license, and local feasibility actually exist as claimed.

Numbers on this page are frozen at writing (n = 511, 8 matches, 2026-07-23) and drift as the model does — the scorecard is always current. When a changelog table and the scorecard disagree, the scorecard wins and this page needs an edit.