SimTrack3D — supplementary material

Gino Chang

Supplementary material · 2026

← back to SimTrack3DDemoRelated workMethodObservabilityExperiments3D viewer

Supplementary material

These four analyses did not fit the eight-page submission. None of them is needed to follow the main results; each answers a question a careful reader is entitled to ask. Numbering and the bibliography are shared with the main page.

• S1. Closing the loop: self-training on the estimator’s own output
• S2. Synthetic benchmark: absolute accuracy by stroke type
• S3. Venues the model has never seen through any camera
• S4. Robustness to tracker noise and dropout
• S5. Real tournament footage: the guarantee, and what the repair costs
• S6. Extended related work

S1. Closing the loop: self-training on the estimator’s own output

Everything reported in §5 comes from a model trained on synthetic flights alone. This section describes what happens if that rule is relaxed and the estimator's own repaired output is fed back as supervision. It is reported because it works, and separated because a sim-to-real claim that needs real labels to hold is not a sim-to-real claim.

The estimator of §3.4 runs at test time and returns a possible flight for every stroke. That is useful in itself, but it also produces something the training pipeline never had: a 3D label for real footage. The label is not ground truth — it is the network's own proposal moved onto the feasible set — yet it differs from the proposal in exactly the way physics says it must, and it is available on every rally we can track.

Training on one's own predictions is self-training, and its failure mode is well known: a confident mistake becomes a confident label, and the error is reinforced rather than corrected. That risk is acute here, because §4 shows the network is most assertive precisely where the image is least informative. Two things make the loop safe enough to be worth closing.

The first is that the corrector is independent of the model's confidence. The court's rules are not learned, not tuned, and not correlated with anything the network believes; a reconstruction that passes below the cord is wrong no matter how certain the network was. Projection therefore removes an entire class of error that self-training on raw predictions would preserve.

The second is that the projection reports its own reliability. Two numbers fall out of the solve at no extra cost and need no ground truth: the reprojection residual of the fitted flight, and the price of feasibility, the extra residual the projection had to accept. A stroke whose flight explains its pixels and needed no help is a stroke worth learning from; a stroke that the physics could only satisfy by abandoning the image is not. We keep strokes below the median residual and discard the rest.

The fine-tuning itself is deliberately plain, so that any gain is attributable to the labels and not to the recipe: the same architecture and loss, initialised from the synthetic model, with each batch drawn one quarter from the accepted real pseudo-labels and three quarters from synthetic data so the prior is refreshed rather than overwritten.

The evaluation is the point. Measuring the fine-tuned model with the constrained solve still attached would be circular, since the solve enforces the property we are testing for. We therefore measure the raw network, with no solve at test time, on venues held out in their entirety — every rally from those matches excluded from the pseudo-label set, so the model has never seen that camera's depth prior. The question is whether the fraction of physically impossible reconstructions falls. If it does, physics has moved from a post-hoc correction into the prior itself, which is the difference between a filter and a model that has learned the game.

It does fall, by about a third (§5.7): on two venues held out entirely, the raw network's impossible fraction goes from 27.4% to 17.6% after twelve epochs on 2,588 accepted pseudo-labels, at a cost of 0.006 m on the synthetic benchmark. Physics can be distilled into the prior; it cannot yet replace the safety net.

S2. Synthetic benchmark: absolute accuracy by stroke type

Claim. A ray-depth lifter trained only on anchored synthetic flights recovers metric 3D on held-out strokes, and its error is governed by how much the image can say about each stroke.

Held-out strokes are partitioned by real anchor, so no replayed flight appears on both sides. Badminton is tactically stratified, and so is the error: the table lists every type with at least thirty held-out strokes, ordered by observability.

Through cameras the model never saw. The model is trained on 51,672 strokes rendered through the 25 training cameras and tested on 11,653 held-out strokes, 2,525 of them through the 7 held-out cameras. Its per-frame median is 0.325 m through cameras it never saw, against 0.330 m through the cameras it trained on. Per stroke type through unseen cameras the ordering is unchanged (per-stroke medians): lift 0.271, clear 0.273, smash 0.352, net shot 0.392, drop 0.417, block 0.533 m. The camera is not what the network learns.

Baselines on the same held-out set: assuming the shuttle is on the floor gives 11.8 m, a constant depth to the court centre gives 1.99 m. The stress sweep of §5.8 shows this figure degrades gracefully with detection noise and dropout rather than collapsing.

Conclusion. The claim holds, with a qualification the next sections build on. For lifts and clears the model sits just above the Cramér–Rao bound, as an honest estimator must. For every fast or short stroke it sits below the bound, by up to 4.6× on drives. An unbiased estimator cannot do that; the network is answering those strokes from the distribution of real play it was trained on, not from the image. §4 and §6.1 take that up.

S3. Venues the model has never seen through any camera

Claim. A model that never saw a venue's camera lifts through it as well as through the cameras it was trained on.

The twenty-two TrackNetV3 venues are the test: they are calibrated from broadcast alone (§3.1), five of them are held out of training entirely, and their per-frame 2D truth removes the tracker from the comparison. On their 2,592 interior strokes the raw network's reconstructions are physically impossible for 17.9% (cord 9.5%, no side change 6.9%, floor 2.2%, hall 0.0%), against 11.6% on the ShuttleSet corpus (§5.5). Per venue the rate runs from 9.6% to 30.1%, and correlates only weakly with camera height (−0.31) or distance (−0.22), so venue difficulty rather than camera geometry sets the spread. The two corpora are not directly comparable — these venues are calibrated from broadcast alone and their strokes are segmented by a different pipeline — so the number to read here is the split by camera below, not the gap between corpora. The constrained solve pays a median price of 0.00 px on these venues (p90 23 px), and returns possible flights for all of them.

The split by camera is what matters, and it has almost closed. Of the 22 venues, five were held out of training entirely (§5.1); on those the impossible fraction is 18.4%, against 17.8% on the 17 whose cameras rendered training data — a gap of six tenths of a point. Seeing a venue's camera in training is worth almost nothing, which is the claim this section exists to test. Per venue the range is 9.6% to 24.5%, and the two extremes are both training-camera venues, so venue difficulty rather than camera novelty sets the spread. On the 150 strokes solved with the rectifier the price of feasibility is 0 px at the median and 20.3 px at the 90th percentile.

One control belongs here. The model trained on random launch states (§5.9) breaks a rule on 24.8% of these strokes while being 1.8× less accurate on held-out synthetic ones, so on these venues the more accurate model is also the more legal one. That is not something a rule count can be relied on to show. Plausibility and accuracy are separate axes, and the evidence for that does not come from this control: the cord test is decided by a 1% depth error (§5.6), and on the rally-final strokes where the true 3D landing is known, adding the floor constraint drives the landing height error to zero while increasing the horizontal error.

Closing the loop (§3.5). The constrained solve labelled 6,878 real ShuttleSet strokes from eight venues; two venues (1,702 strokes) were held out entirely, and of the 5,176 trainable strokes the residual gate kept the 2,588 below the median. Fine-tuning the 25-camera model for twelve epochs on batches drawn one quarter from these labels and three quarters from synthetic data, and then scoring the raw network with no solve at test time, the fraction of physically impossible reconstructions falls from 14.2% to 6.1% on the trained venues and from 27.4% to 17.6% on the two it never saw, while held-out synthetic error moves from 0.267 to 0.273 m. A third of the real-footage failures were learnable from the court's rules alone, without a single 3D label; the remaining two thirds are what the second tier is for.

Conclusion. The claim holds on both sides of the sim-to-real boundary. On synthetic data accuracy is flat across camera geometry (a 0.010 m spread, an order of magnitude below the error; only the height component grows with camera height, 0.063 to 0.127 m), and on real footage unseen venues are indistinguishable from seen ones (18.4% vs 17.8% impossible on the TrackNetV3 venues, 71.8% vs 72.9% possible on the held-out ShuttleSet venues). The camera is not what the network learned.

S4. Robustness to tracker noise and dropout

Claim. The model degrades gracefully with the two failures real trackers actually have.

Training injects σ = 2 px detection noise and 5% random dropout. A sweep over noise level and the number of consecutively dropped frames measures the margin. Each cell is 1,908–2,000 held-out strokes; the figure is the median per-stroke 3D error in metres.

Conclusion. Quintupling detection noise costs 25%; dropping eight consecutive frames costs 45–60% depending on the noise level, and the two effects are close to separable. Real trackers fail by losing the shuttle, not by jittering (9–92% detection per venue on our footage), so it is the length of the gap, not the precision of the hits, that governs what can be recovered.

S5. Real tournament footage: the guarantee, and what the repair costs

Claim. A model trained on synthetic flights alone, with no real 3D label and no fine-tuning on real video, produces flights that a badminton court permits, and the repair that guarantees this is cheap enough to be invisible.

The estimator runs on 6,553 real strokes from ten venues, through the cameras measured in §3.1, with the anchors of §3.5 as tokens to the network and the court's rules in the rectifier. Nothing here was trained on real data.

Two thirds of the strokes need no repair at all: the launch state that best explains the network's depths already breaks no rule. The rectifier moves another 28.9% into the feasible set, and it does so almost for free — the median repaired stroke gives up no measurable reprojection accuracy, and the 90th percentile gives up 0.76 px, well inside the tracker's own noise. For the remaining 4.0% no legal flight is found and none is returned: the stroke is flagged. Forcing one would trade an honest gap for a plausible-looking wrong answer, and it would break the property that makes the output usable — that every trajectory in it is one the game allows.

What the sim-to-real work bought. The same architecture trained without the tracker-noise model of §3.3 — clean 2 px Gaussian noise and random dropout, the standard synthetic recipe — is worse on real video by every measure that matters, although it scores better on the clean synthetic benchmark (0.244 against 0.271 m). Modelling the real tracker's failures is worth 19% of the arrival error.

Endpoint error deserves its own qualification. The reference is the hitting player's annotated ground position, itself a median 0.75 m from the racket, so a 0.6 m arrival error is close to the floor this reference can resolve; the improvement from 0.73 to 0.59 m is therefore larger in relative terms than it looks. On the 530 rally-final strokes the landing spot read from the stationary shuttle gives a second, independent check, and it is the hardest case in the corpus: those strokes are the fastest and shortest, and no receiver stands at their end.

Conclusion. Feasibility is not accuracy, and the two are reported separately for that reason. What the guarantee provides is that every returned flight is one the game allows, obtained in 31 ms per stroke, from a model that has never seen a real 3D label.

S6. Extended related work

This section is organised by the problem each literature solves rather than by venue, because the difficulty here is that no single community owns it: the tracker comes from sports vision, the depth ambiguity from monocular geometry, the training data from the sim-to-real literature, the calibration from sports field registration, and the guarantee from constrained estimation.

2.1 Tracking a small, fast object in 2D

The 2D half of this problem is solved well enough to build on. TrackNet^[1] established the recipe still in use: a heatmap network consuming several consecutive frames so that motion, not appearance, localises an object too small to have texture. Successors extend it with trajectory-aware refinement and rectification^[2] and with motion attention^[3], and the same architecture family has been retargeted to soccer^[4], table tennis^[5] and to a sport-agnostic baseline^[6]. Speed-accuracy trade-offs for ball detection are surveyed in^[7], and the general difficulty of small-object detection and tracking in^[8].

What matters for us is that a broadcast shuttlecock is not a point but a streak. Motion blur is treated as a nuisance to be removed in general vision^[9], but in racket sports it carries information: BlurBall^[10] estimates ball position and blur jointly and shows the streak improves localisation, and blur-aware detection has been applied to broadcast sport directly^[11]. Our own measurements on 9,312 annotated shuttlecock positions agree: the streak's long axis grows linearly with image speed while its short axis is constant to 0.20 px, which is what makes exposure self-calibrating from the video alone. Badminton-specific detectors continue to appear^[12][13][14], including real-time smash-speed estimation on device^[15].

Multi-object tracking metrics and player tracking are adjacent but distinct problems^{[16][17][18][19]}; we use annotation for stroke segmentation rather than tracking players.

2.2 Monocular 3D trajectory reconstruction in racket sports

Reconstructing 3D ball flight from one view is an established goal with a consistent structure: fit a physical flight model to a 2D track through a calibrated camera. Early table-tennis systems reconstructed trajectories for performance analysis^[20][21][22]; robot table tennis needs the same estimate in real time and solves it with filtering^[23][24][25]. Multi-camera systems avoid the ambiguity entirely^[26] and are therefore not comparable.

The closest physics-fitting work to ours is MonoTrack^[27], which reconstructs badminton shuttle trajectories from monocular video by fitting a per-shot launch state and drag coefficient under a reprojection loss. We re-implement it faithfully as our baseline. The closest learned counterpart is TrajTrans^[28], a thesis-chapter extension of SynthNet that trains a transformer only on synthetic shuttle flights, with cameras sampled around the 47 calibrated ShuttleSet22 cameras, and regresses six launch parameters from the pixel track and four court corners; on real video it is evaluated only by 2D reprojection and cross-camera consistency, with no real 3D. In table tennis the same idea has been pushed further: TT3D^[29] uses the bounce on a calibrated table as a hard geometric anchor, TT4D^[30] builds a pipeline and dataset for 4D reconstruction from monocular video, and Uplifting Table Tennis^[31] trains a transformer on synthetic trajectories to lift 2D tracks to 3D with spin. Related efforts recover 3D localisation from 2D labels using physics^[32], estimate spin from broadcast footage^[33], reconstruct a ballistic shot from a monocular broadcast^[34], and localise a soccer ball in real time from a single camera^[35][36].

Two of these deserve separate mention because they share our strategy. SynthNet^[37] and "Where Is The Ball"^[38] both train on synthetic trajectories and represent the input in a camera-independent form; the latter reports an ablation showing that a canonical ray representation outperforms domain-randomised pixel input, which is independent support for the parameterisation we adopt. The gap we address is not that these methods lift badly. It is that every one of them validates on real footage with a proxy — reprojection error, a landing tile, a bounce or net consistency check — and none reports whether that proxy can pass while the 3D is wrong.

Badminton has also been studied without 3D reconstruction: shuttlecock trajectory prediction from player context, stroke and skill modelling with transformers^[39], self-play simulation of the game^[40], and event-camera reconstruction of swing dynamics^[41]. Earlier work estimated shuttle trajectory from motion blur in a single image^[42], which is the same physical signal we exploit for exposure.

2.3 Sim-to-real transfer: what the field actually does

Training on synthetic data and deploying on real data is a mature field with a small number of distinct mechanisms, and it is worth being precise about which one we use.

Domain randomisation makes the simulator's nuisance parameters so variable that the real world looks like one more sample^[43]. It underlies CAD2RL^[44] and drone racing^[45], has been given structure by conditioning on scene context^[46], analysed theoretically^[47], benchmarked^[48], and applied to object detection for robotics^[49]. Its known failure mode is that the randomisation range must bracket the real value; when it does not, no amount of variety helps.

Real-to-sim, or system identification, takes the opposite route: measure the real system and set the simulator to match. SimOpt closes the loop by updating the simulation distribution from real rollouts^[50], sim-to-sim canonicalisation trains a mapping to a canonical rendering^[51], residual physics learns the part of the dynamics the model misses^[52], and differentiable simulators fit parameters by gradient^{[53][54][55][56]}. Sensor-level realism can be identified the same way^[57][58].

Domain adaptation aligns features or pixels using unlabelled real data: adversarial feature alignment^[59][60], entropy minimisation^[61], self-training and co-training^[62][63][64], statistical alignment^[65], frequency-domain transfer^[66], and standard benchmarks^[67]. We use none of it. Our model never sees real video during training, which is a limitation we state rather than a design virtue.

Image-level translation refines synthetic images towards realism^[68][69], optionally with a task-consistency term^[70]. Photorealism is the competing school: photorealistic synthetic datasets^[71][72][73], physically based rendering pipelines^{[74][75][76][77]}, and applications from spacecraft pose^[78] to street scenes^[79]. Reviews of synthetic-data generation^[80][81] and of how a transfer budget should be spent^[82] survey the trade-offs. Mixing a little real data in is often the cheapest win of all^[83].

Our approach sits in the real-to-sim family and is unusually literal about it: rather than randomising camera parameters over an assumed range, we measure real broadcast cameras (thirty-two in all, twenty-five of them used for training) and sample only from those, and rather than sampling flights from a prior, we solve boundary-value problems anchored to 13,129 annotated strokes from real matches. What we do not do is close the loop — our identification is one-shot, not iterative as in SimOpt.^[50]

2.4 Synthetic data for sports vision

Synthetic data has reached sport recently and mostly for detection rather than 3D: SoccerSynth-Detection renders soccer scenes to train player detectors^[84] and SoccerSynth Field does the same for field detection^[85]. Virtual environments have long been used to study sports skill itself^[86]. The 3D-trajectory works of §2.2 that train on synthetic flights^[37][31][38] are the closest precedents for what we build, and they differ from us mainly in how the camera distribution is chosen.

2.5 Camera calibration for broadcast sport

A planar court gives a homography, and a homography does not determine focal length: this is the classical degeneracy of plane-based calibration^[87], and it is why single-view calibration usually recruits extra structure such as vanishing points^{[88][89][90][91]} or a learned prior^[92]. Sports field registration is now a field of its own, with keypoint- and line-based methods^{[93][94][95][96][97]}, graph-based homography estimation^[98], sequential Bayesian estimation across a video^[99], and a benchmark protocol^[100].

Almost all of this work registers the ground plane, which at broadcast tilt constrains focal length only weakly. Our contribution is to recruit the net cord — the only rigid off-plane structure a badminton court offers — and, separately, to gate each calibration on the cord's catenary sag, a physical invariant the fit never sees.

2.6 Physically constrained reconstruction

Enforcing physics on a monocular reconstruction is well established for humans: scene constraints resolve pose ambiguity^[101], gravity constrains human-object reconstruction^[102], trajectory optimisation enforces physical consistency on pose sequences^[103], and physical plausibility can be built into prediction^[104]. Physics-informed learning more broadly is surveyed in^{[105][106][107][108]}.

The distinction that matters here is soft versus hard. Most of the work above adds physics as a loss term, which can always be traded away. Enforcing constraints exactly is a separate literature: constrained Kalman filtering projects the unconstrained estimate onto a feasible set^[109], differentiable optimisation layers embed a constrained solve inside a network^[110], and architectures exist that satisfy hard constraints by construction^[111]. We take the projection route, applied per stroke to a six-parameter flight, and we report what feasibility costs in reprojection error rather than only that it was achieved.

2.7 Observability, degeneracy and estimation bounds

That a monocular view leaves depth weakly determined is not news; what is useful is to quantify it per stroke. Bundle adjustment has long treated the null space of the normal equations as a first-class object — gauge freedom, handled by minimum-norm steps^[112]. In filtering, the observability-constrained EKF identifies the same kind of unobservable subspace in order to protect it from spurious information^[113], and observability-aware trajectory optimisation plans motions that make states observable^[114]. Visual-inertial estimators must handle the same degeneracies^[115].

We use that subspace differently: not to protect it and not to plan around it, but as the place to look for a physically feasible solution, on the argument that moving there is nearly free in image terms. We pair this with a Cramér–Rao bound on the six launch parameters, which turns "this stroke is hard" into a number that can be compared against what a model actually achieves.

2.8 Does in-distribution error predict real-world performance?

Our central negative result runs against a well-supported regularity. "Accuracy on the Line" reports a strong linear relationship between in-distribution and out-of-distribution accuracy across many benchmarks^[116], and where that relationship holds, a synthetic test set would be a fair guide. But it is not universal: in-distribution and out-of-distribution performance are sometimes inversely correlated^[117], and underspecification explains why two models with identical validation loss can behave differently under shift^[118]. Metrics designed for the sim-to-real gap itself have begun to appear^[119].

The closest precedent for our diagnosis is in human pose: physical plausibility can be measured without ground truth by simulating the result^[120], and that work also finds its own plausibility proxy does not track accuracy reliably — the same shape of result we obtain for net clearance. Reference-free consistency assessment is being explored for generated video as well^[121]. Our contribution here is the sharper, rank-order version of the claim, measured on 6,553 real strokes: across stroke types, held-out synthetic error carries no usable signal about real-footage error, and neither does the image-side observability number.

2.9 Shuttlecock aerodynamics

The flight model we integrate rests on a well-characterised object. The shuttlecock's drag coefficient, terminal velocity and strongly asymmetric trajectory have been measured in wind tunnels and free flight^{[122][123][124][125][126][127][128]}, simulated^[129][130], and summarised for the sport as a whole^[131]. A terminal velocity near 6.8 m/s and a drag-dominated, markedly asymmetric arc are what make a badminton flight identifiable from a short observation at all, and they are also why a ballistic parabola is not an adequate model.

These sections cite no additional work; the bibliography is on the main page.