System — how a game becomes analytics

Every box is a real pipeline stage (module names in the corner). Solid chips are shipped; dashed amber chips are the improvement queue. Run a demo at the bottom to watch a video move through the chain and inspect what each model produced — all model outputs are persisted per job, so inspecting never re-runs anything.

Pipeline

media.py · court.py · audio.py

1 · Ingest, proxy & court geometry

ffmpeg · classical line/corner detection · Gemini corner fallback · RMS audio energy
  • Multi-clip uploads ordered by recording time, normalized, joined
  • 480p/12fps analysis proxy (keeps ambient audio) + 1080p vision proxy for detection
  • Court corners: detected → Gemini fallback → user-drawn; homography = image ↔ court meters
  • New: ambient-audio energy series + impact-like peaks stored per job
GAPS → NEXT
  • One calibration per job — camera motion/zoom breaks it (per-segment calibration queued)
rally.py · gemini.py

2 · Rally segmentation & selection

Gemini video understanding (structured JSON), fallback model ladder
  • Prompt: rally = serve struck → point clearly over; “strictly exclude dead time”; intensity 1–5 + note
  • Clean: clamp to video, drop < min length, merge overlaps
  • Reel pick: rank by (duration, intensity), cap count + total seconds; pad ±
GAPS → NEXT (2026-07-12 audit)
  • Boundaries are whole-second and model-judged only — no fused signal check
  • No in-play/no-play sub-segmentation INSIDE a rally (shuttle pickup survives in long rallies)
  • “Longest = best highlight” is unvalidated → highlight score v2 (audio peaks + shot speed + length)
runpod_worker · vision_local.py

3 · Vision workers (GPU / CPU)

TrackNetV3 shuttle · YOLO26L-pose · BoT-SORT + ReID ids · racquet chain
  • Per-rally 30fps clip → TrackNetV3 points; players+17 keypoints at 6 Hz, imgsz 1920
  • Tracker persists across the rally, resets between rallies; court-polygon gating kills spectators
  • Honest per-rally quality: coverage × longest-gap × teleport penalties
  • New: verbatim worker output persisted (vision_raw.json)
GAPS → NEXT
  • 180-frame cap → long rallies sample below 6 Hz
  • Whole-frame pose is weak on far players → per-player crop pose (queued)
  • No singles/doubles contract yet (match_type, queued)
track.py · smooth.py

4 · Tracking & filtering (honest-data layer)

innovation-scored confidence · Hampel outliers · One-Euro pose smoothing
  • Measured shuttle confidence: fwd+bwd constant-velocity innovation, static-run + >500 km/h gates, provenance
  • ONE filtered track feeds camera, Studio, baked marker and 3D (no divergence)
  • Stable player ids: fragment merge, near player = P1; One-Euro keypoint smoothing
  • Public contracts: gap-preserving 12.5 Hz shuttle track; dropouts stay visible gaps
GAPS → NEXT
  • No predicted/inpainted gap-fill provenance yet (consumers would need to trust-tier it)
court.py · rally3d.py

5 · Interpolation layer — 2D → 3D court space

isolated module — iterate models here freely
plane calibration (Zhang) · ray casting · drag-ballistic LM fit · physical gates
  • Known court dims (6.1×13.4 m) + homography → camera focal, rotation, position
  • Players: ankle/foot pixel → ground-plane meters (movement series, heatmaps)
  • Shuttle: per-shot ballistic fit (a = g − k·|v|·v), multi-start, gated on floor/bounds/net/contact/speed/continuity
  • Speeds: at impact (|v0|) + radar-comparable at-net
GAPS → NEXT
  • Hits inferred from 2D direction turns only → fuse audio peaks + wrist/racquet proximity
  • Player “3D” is a 2.5D billboard, not lifted pose
track.py · render.py · validate.py

6 · Virtual camera & render

FocusPath planning · keep-in-frame containment · validation ladder
  • Camera path from player tracks + filtered shuttle (POV clips: gentle motion fallback)
  • 9:16 crop per frame; camera_path exported so the Studio can invert the bake
  • Validate motion/framing → trim → safe re-render → drop (never ship a broken clip)
stitch.py · coach.py · analysis.py

7 · Outputs & analytics

xfade stitch + music bed · Gemini coach notes · analysis.json export
  • Reel (beat-synced, ambient audio under music), thumbnail, gallery, Studio editor
  • New — analysis.json: play/no-play timeline, hit markers with speeds, shuttle-flight segments, audio peaks, per-player court-space movement series
  • Built once from stored results, cached; ?refresh=1 after Studio edits

Stored per job (nothing gets re-run to answer a question)

result.jsonanalysis.json gemini_rallies_raw.jsonvision_raw.json proxy.mp4 / vision_proxy.mp4reel.mp4 · thumb.jpg

Improvement queue (from the rally-break / in-play audit)

in-play mask v1: shuttle-flight ∪ audio peaks ∪ player kinetics highlight score v2 (audio + shot speed + length) match_type singles|doubles per-player crop pose TrackNet A/B + fine-tune on labelled bench per-segment camera calibration

Run a demo

or inspect an existing job: