Spherical Interpolation for Backward-Compatible Multimodal Representations

Abstract

Contrastive vision-language models map visual and textual representations into a shared normalized embedding space, making cosine similarity the natural metric for cross-modal retrieval. A practical challenge arises during model upgrades: independently trained models generally produce incompatible representation spaces, so replacing a deployed model typically requires recomputing embeddings for the entire gallery, which is prohibitively expensive at scale. Orthogonal post-hoc alignment can partially mitigate this problem by mapping new-model queries into the old-model gallery space. However, because independently trained models can differ in fine-grained representation structure, the orthogonal alignment remains approximate, leaving a residual angular discrepancy between the old-model query and the aligned new-model query. We study whether interpolation along the spherical geodesic between these two normalized query representations can improve retrieval without re-indexing the gallery. We characterize when this path contains an interior query direction closer to an idealized retrieval-optimal direction than either endpoint, and connect this characterization to Recall@K through a local margin-based certification result. Experiments across multiple benchmarks and model families show that post-alignment spherical interpolation improves over orthogonal alignment alone, recovering backward-compatibility in most evaluated settings. Consistent with our geometric characterization, per-query oracle analysis shows that retrieval-favorable interior points occur frequently in practice.

Method

The gallery stays encoded by the old model \(\phi_{\text{old}}\). Only queries change: each query is encoded by both models, the new embedding is rotated into the old space, and the two are blended along the sphere.

Because \(R^\star\) is a rotation, it keeps the new model's inner products intact. Independently trained models still differ in fine-grained structure, so a residual angle \(\theta\) remains between u and v. SLERP moves at constant angular speed, \(\angle(u,q_\alpha)=\alpha\theta\), so \(\alpha\) means the same fraction of the way from u (\(\alpha=0\), old query) to v (\(\alpha=1\), SVD) for every query.

When can an interior point help?

Let \(q^\ast\) be an idealized retrieval-optimal query direction (used only for analysis), and let \(p\) be its projection onto the plane spanned by \(u\) and \(v\).

Theorem 1. For \(\|p\|>0\), there is an \(\alpha^\ast\in(0,1)\) with \(\angle(q_{\alpha^\ast},q^\ast)<\min\{\angle(u,q^\ast),\angle(v,q^\ast)\}\) if and only if \(p/\|p\|\) lies strictly inside the arc from \(u\) to \(v\). Otherwise the best point on the arc is an endpoint.

Appendix F connects this angular gap to Recall@K through a top-K margin bound. Empirically (Appendix M, Flickr30k, CLIP ViT-L/14 → ViT-B/32), the similarity to a relevant caption peaks strictly inside the arc for 99.87% of the 31,014 image queries, and 1,408 queries reach Recall@1 only at interior weights.

Explore the interpolation path

Recall along the geodesic for the CLIP ViT-L/14 → CLIP ViT-B/32 upgrade. The gallery is always encoded by ViT-B/32 (old). \(\hat\alpha\) maximizes CC3M Recall@1 for the selected support modality and direction, and is then reused unchanged on every test set.

Results

Five upgrades (CLIP ViT-L/14 → ViT-B/32, CLIP ViT-H/14 → ViT-B/32, SigLIP2 → SigLIP1, SigLIP2 → CLIP ViT-B/32, SigLIP1 → CLIP ViT-H/14) on Flickr30k, COCO and NoCaps. An upgrade is backward-compatible when new-to-old retrieval beats the old system, \(M_{\text{new}\to\text{old}} > M_{\text{old}\to\text{old}}\).

Recall@1 (%) with text-only Procrustes support, from Tables 1 and 2. ✓ / × mark backward compatibility against the old model. α̂ is selected on CC3M (I2T / T2I) and fixed across test sets. Image-only and joint supports, the CLIP ViT-H/14 → ViT-B/32 pair, Recall@5/10, the test-set oracle α* and zero-shot classification are in the paper.

Where the gains come from

• Combining the two endpoints does most of the work. Even a fixed midpoint of the old and aligned queries improves mean Recall@1 by +4.75 over SVD and raises compatibility from 45 to 72 of 90. Choosing α̂ on the support set brings it to 85 of 90 (Appendix N).
• The weight transfers. α̂ chosen on CC3M lands close to each benchmark's own optimum α*, with no test-set tuning.
• Cheap at deployment. The overhead is on the query side only. With the two encoders on separate GPUs, or a cached old-model query, latency is within about 0.3 ms of a fully re-indexed system (10⁷-item gallery, Appendix P).
• Useful during migration. On Flickr30k (ViT-L/14 → ViT-B/32), SLERP reaches 48.00 R@1 without touching the gallery. Partial backfilling needs to re-encode 90% of the captions to reach 47.98.
• Beyond compatibility. SLERP often improves over the new model even when the gallery is re-indexed, and also helps zero-shot classification with fixed old-model text prototypes.

Implementation

The method consists of two functions in PyTorch (see slerp_bc/method.py).

def procrustes(V_new, U_old):
    # argmin_R ||V_new R - U_old||_F  s.t.  R orthogonal  (closed form)
    P, _, Qt = torch.linalg.svd(V_new.T @ U_old, full_matrices=False)
    return P @ Qt

def slerp(u, v, alpha, eps=1e-7):
    u, v = F.normalize(u, dim=-1), F.normalize(v, dim=-1)
    theta = torch.acos((u * v).sum(-1, keepdim=True).clamp(-1 + eps, 1 - eps))
    q = (torch.sin((1 - alpha) * theta) * u + torch.sin(alpha * theta) * v) / torch.sin(theta)
    return F.normalize(q, dim=-1)

d = max(V_support.shape[1], U_support.shape[1])
pad = lambda x: F.pad(x, (0, d - x.shape[-1]))   # zero-pads the old space when d_new > d_old
R = procrustes(V_support, pad(U_support))      # new -> old, fitted once on the support set
q = slerp(pad(old_query_emb), new_query_emb @ R, alpha=alpha_hat)
scores = q @ pad(old_gallery).T                # = q[:, :d_old] @ old_gallery.T: the gallery is untouched

Zero-padding keeps \(R^\star\) square when the new model has more dimensions (e.g. 768 → 512). Scores only depend on the original old-model coordinates.

Limitations

• Both encoders (or a cached old-model query embedding) are needed at query time. This overhead does not grow with gallery size.
• The interpolation weight must be chosen on data, either a support set or a small labeled subset of the target distribution (Appendix K).
• Performance is bounded by the two endpoints and the Procrustes map: interpolation cannot add information that neither model has.

Citation

@inproceedings{ricci2026spherical,
  title     = {Spherical Interpolation for Backward-Compatible Multimodal Representations},
  author    = {Ricci, Simone and Biondi, Niccol{\`o} and Pernici, Federico},
  booktitle = {Advances in Neural Information Processing Systems},
  volume    = {39},
  year      = {2026}
}