Error-signal replay
Each small dot is a tracked point, coloured by its own error at the current frame: white is rigid, redder is more error. For the rigid residuals, each large blue ring is that point's rigid prediction, where a rigid motion of its link would put it, and a line joins the ring to the dot. A dot inside its ring is moving rigidly. The legend on the replay names every mark.
Loading…
Layers
Per frame: the 90th percentile over the link's scored points. Click or drag on the chart to move through time; click a point in 3D to add its own curve.
The four error signals
Open one to see what it measures, how it is derived from MoSE3's outputs, and how the per-frame and per-video values are computed.
Rigid residualfraction of link length
What it measures. How far each tracked point is from where MoSE3's own rigid motion puts it. MoSE3's rigidity head chooses which pixels form one rigid body; its track head says where each point went. If the link stays rigid, every track lands on that body's motion.
How it is derived. For point i, MoSE3's SE(3) (Ri(t), ti(t)) is fitted
exactly as upstream fit_per_pixel_se3 does:
- Draw 12,000 random pixels from the whole first frame (the same pool for every point of a video).
- Weight each pool pixel j by softmax of cos(ei, ej) / 0.01, using the first-frame rigidity embeddings; pixels with nearly the same embedding dominate.
- Fit the weighted rigid motion (Horn's closed form) that carries the pool's first-frame positions to their frame-t positions from MoSE3's tracks.
Pi(t) = Ri(t) xi(0) + ti(t)
How it is measured. ri(t) = ‖Pi(t) − xi(t)‖ / L, with xi(t)
the point's own MoSE3 track and L the link's length. Points count when MoSE3 sees them at frame t,
the fit is well conditioned (se3_valid), and they are not on a depth edge or a depth outlier.
Each frame's value is the 90th percentile over at least 5 such points; a video's value is the largest frame
value after frame 0.
In the replay. Blue rings are Pi(t), dots are xi(t), and the line between them is the residual. AUROC (severe vs none): Link2 , Link5 .
Rigid residual, visible refitfraction of link length
What it measures. How far the link's visible points are from moving as a single rigid body, with no use of MoSE3's rigidity embedding or its SE(3).
How it is derived. At each frame, take this link's scored points that MoSE3 sees, and fit one rotation and translation by least squares (Kabsch) from their first-frame positions to their frame-t positions:
Pi(t) = R(t) xi(0) + t(t)
Every visible point weighs the same; occluded points are left out of that frame's fit; at least 5 are needed. Every point is also part of the fit it is measured against.
How it is measured. The same ri(t) = ‖Pi(t) − xi(t)‖ / L, 90th percentile per frame, and largest frame value per video.
In the replay. Blue rings show the single fitted motion. AUROC: Link2 , Link5 . The method section below compares the rigid residuals in detail.
Rigid residual, dense mask fitfraction of link length
What it measures. The same question as the visible refit, how far the link's points are from moving as one rigid body, but the rigid motion comes from every pixel of the link's mask instead of from the points being scored.
How it is derived. MoSE3 tracks every pixel of the first frame, so the link's frame-0 SAM mask is followed into every frame. The mask is resized to MoSE3's 294 × 518 grid (a pixel is in when the mask covers at least half of it). At each frame t:
- Keep mask pixels more than 2 pixels inside the mask border, not on a MoSE3 depth edge, whose frame-0 depth is not an outlier, and that MoSE3 sees at frame t. A fit uses a median of about 4,850 pixels on Link2 and 6,060 on Link5, against roughly 60–200 PDI points.
- Fit one rotation and translation from their frame-0 positions to their frame-t positions by weighted least squares (Kabsch), reweighting 10 times with w = 1 / (1 + (r / 3·median r)2). Pixels far off the common motion, for example where the mask spills onto another surface, lose their pull; a median 96% of pixels keep a weight of at least 0.5.
- Leave the frame without a value when fewer than 200 pixels or 30% of them are visible, or the visible ones span less than half the link.
Pi(t) = R(t) xi(0) + t(t)
The scored PDI points are only 1–3% of the fitted pixels, so a point barely moves the fit it is measured against.
How it is measured. The same ri(t) = ‖Pi(t) − xi(t)‖ / L at the same scored PDI points, with L measured on the fitted pixels instead (a median 0.99 of the PDI-point length). 90th percentile per frame, largest frame value per video.
In the replay. Blue rings show the dense fit's single motion. AUROC: Link2 , Link5 . It ranks the videos almost exactly as the visible refit does (rank correlation 0.97–0.99), and the AUROC difference from the refit is within noise: Link2 −0.03 (95% interval −0.08 to +0.01), Link5 +0.00 (−0.05 to +0.06).
Rigidity discrepancy1 − cos
What it measures. Whether MoSE3 still treats the link as one rigid body: how far each point's rigidity vector is from the link's typical one. Pixels on one rigid part share nearly the same vector, so a part that starts to move on its own drifts away from the rest.
How it is derived. MoSE3 predicts a 16-dimensional rigidity embedding for every pixel of every frame. For point i at frame t, read frame t's embedding at the point's tracked pixel and scale it to unit length: ei(t). The link's typical vector m(t) is the normalized mean over its points that are visible and inside the frame.
di(t) = 1 − cos(ei(t), m(t))
How it is measured. 0 means the same direction as the link's mean. Each frame's value is the 90th percentile of di(t) over at least 5 such points; a video's value is the largest frame value after frame 0. Unlike the residuals, frame 0 is not zero by construction.
In the replay. No rings: this signal has no rigid prediction. AUROC: Link2 , Link5 .
All videos
Method: the three rigid residuals
All three residuals ask the same question: how far is each tracked point from where a rigid motion of its link would put it? They differ only in which rigid motion is used.
What all three share
ri(t) = ‖ Pi(t) − xi(t) ‖ / L
- xi(t): MoSE3's 3D track of PDI point i at frame t, in the first camera's coordinates and MoSE3's scale. Frame 0 is the query frame.
- Pi(t): the rigid prediction, i.e. where a rigid motion moves the point's frame-0 position xi(0). This is the only part that differs.
- L: the link length. Take the first principal direction of the scored points' frame-0 positions, project the points onto it, and use the distance between the 2nd and 98th percentiles. The dense mask fit does the same with its fitted pixels.
- Scored points: the link's PDI points that do not touch a MoSE3 depth edge, whose frame-0 depth is not an outlier, and that MoSE3 marks visible at frame t (visibility logit > 0).
- Per frame: S(t) is the 90th percentile of ri(t) over at least 5 scored points; this is the chart. Per video: the maximum of S(t) over frames 1…T−1; this is the index and the AUROC. Frame 0 is 0 for all three, because Pi(0) = xi(0).
Where they differ
| Rigid residual (MoSE3 SE(3)) | Rigid residual, visible refit | Rigid residual, dense mask fit | |
|---|---|---|---|
| Rigid prediction | Pi(t) = Ri(t) xi(0) + ti(t), MoSE3's per-point SE(3) | Pi(t) = R(t) xi(0) + t(t), one transform for the link at frame t | Pi(t) = R(t) xi(0) + t(t), one transform for the link at frame t |
| How the transform is made | Upstream fit_per_pixel_se3, evaluated at the point: a weighted Horn fit (closed-form quaternion) of a pixel pool's frame-0 positions to their frame-t positions |
Kabsch (unweighted least squares by SVD) of the link's visible scored points: frame-0 positions to frame-t positions | Kabsch with robust weights (Cauchy, 10 reweightings) of the link's visible mask pixels: frame-0 positions to frame-t positions |
| Which samples define it | 12,000 pixels drawn at random from the whole query frame (the same pool for every point of a video), not just the link; 420–670 of them fall on a link (median 470 for Link2, 580 for Link5) | Only this link's PDI points that are visible at frame t (≈60–200); at least 5 are required | Every grid pixel of the link's frame-0 mask more than 2 px inside its border, off depth edges, not a depth outlier and visible at frame t (median about 4,850 for Link2, 6,060 for Link5) |
| Weights | softmax over the pool of cos(ei, ej) / 0.01, using frame-0 rigidity embeddings only (the point's own embedding is interpolated at its pixel). A cosine gap of 0.05 changes a weight about 150×, so the fit comes from pixels with nearly the same embedding: MoSE3's rigid body for that point | Equal weight for every visible link point; no embedding is used | w = 1 / (1 + (r / 3·median r)2): near equal on a rigid link, near zero for pixels far off the common motion; no embedding is used |
| Occluded samples | Count at full weight: MoSE3 still predicts their positions | Excluded at that frame | Excluded at that frame |
| Region | Defined by the embedding. It can include neighbouring parts or background that MoSE3 groups with the link, and it ignores the SAM mask | Defined by the SAM mask, through the PDI points sampled in it | Defined by the SAM mask directly, shrunk by 2 px |
| Is the point itself in the fit? | Effectively no: the transform comes from the pool, so the point is checked against a body it barely influences | Yes: every scored point helps fit the transform it is then measured against (in-sample) | Barely: the scored points are 1–3% of the fitted pixels |
| Validity | Requires se3_valid (Horn's top two eigenvalues distinct) |
Requires at least 5 visible points | Requires at least 200 visible pixels and 30% of the fitted ones, spanning half the link |
| It measures | Whether MoSE3's track head agrees with the rigid body its rigidity head chose | How far the link's visible points are from moving as any single rigid body | The same, with a motion that the scored points barely influence |
| AUROC (max over clip) | Link2 · Link5 | Link2 · Link5 | Link2 · Link5 |
| C-index with moderate kept | Link2 · Link5 | Link2 · Link5 | Link2 · Link5 |
When they disagree
- Self-occlusion. Occluded pool pixels keep full weight in MoSE3's fit, so its transform can drift when much of the link is hidden. The refit uses only visible points.
- The embedding over-merges. If MoSE3 gives the link the same embedding as a neighbouring link or as static background, its SE(3) is the motion of that larger group. The residual then reflects the mismatch between the link and the group, not only the link's own shape change. The refit stays inside the link.
- The embedding splits the link. If MoSE3 already separates a deforming end, each piece gets its own SE(3) and the MoSE3 residual drops. The refit still sees two motions inside one link. The rigidity discrepancy flags this case directly.
- A few bad tracks. The refit is an unweighted least-squares fit, so a few large track errors pull the whole fit and raise residuals across the link. MoSE3's transform comes from hundreds of pool pixels and is barely affected by one point's track.
- Sample size. MoSE3's transform and the dense mask fit use dense pixels. The refit uses only PDI points, which cluster at textured spots near mask borders. All three score the same PDI points.
- Dense mask fit vs visible refit. On this run they agree closely. The dense fit can differ where the PDI points are few (Link5 has about 80 scored points per video), where a few bad tracks would pull an unweighted fit, or where the mask spills onto another surface. Its pixels come from one network's smooth output, so neighbouring pixels' errors are correlated: thousands of pixels are not thousands of independent measurements.
On a rigid link, every point lands on its prediction in all three versions (all rings filled). They diverge only through the cases above. These are diagnostics on 35 labelled videos per link (43 and 45 with moderate kept), not validated detectors.