arXiv · 2611.21372
Learning What Pairs Miss for Higher-Order Multimodal Alignment
Abstract
Contrastive learning beyond two modalities now scores the whole tuple with multilinear products, volumes, and fusion heads, critics that can represent the joint distribution. In practice these critics still behave like bags of pairs. Their negatives break at least two of the three pairs, a single pairwise check rejects them, and when most samples can be solved by pairs the higher-order term receives almost no gradient. To solve this problem, we propose CHORD (Contrastive Higher-Order Residuals with Decoys), which trains a residual against decoys drawn from a calibrated pairwise model, negatives that the pairwise model finds plausible, so that rejecting them often needs all three modalities. Drawing decoys costs one inner-product query over a memory bank, and the optimal residual is the log-ratio between the data and the pairwise model, so the decoy loss also bounds what that model misses. On M3-Mix, where one in ten training samples needs all three modalities, our CHORD improves accuracy on those samples by 21.2 percentage points over the strongest of eight baselines. CHORD also gives the best two-to-one results on every real dataset, and its residual improves other retrievers without retraining.
Explore related subjects
Keep this discovery
Explore connections, maps & timelines
Kunyu Zhang, Shenghui Feng, Xiangjie Shi, Zeqing Wang, Yingxu Wang. 2026-11-16. Learning What Pairs Miss for Higher-Order Multimodal Alignment. https://searcharxiv.com/abs/2611.21372
Citations retain the original authors and DOI and include this site’s record URL. If this discovery service contributed to your research, you can also cite Searcharxiv by Indiciae: BibTeX · RIS.