FactEmbed factorizes MLLM embeddings for composed image retrieval
A KAIST AI paper makes an MLLM imagine the requested image edit through factorized latents before building retrieval embeddings, and reuses those latents for cheap reranking.
The task and the gap
Composed Image Retrieval (CIR) aims to retrieve a target image given a reference image and a modification text: a picture plus an instruction describing how the wanted image should differ. A paper from KAIST AI (arXiv 2611.05193, 4 authors, published 2026-11-03) argues that recent approaches built on Multimodal Large Language Models (MLLMs) leave much of the model’s compositional understanding unused.
The authors trace this to how the query is handled. Current methods map the composed query directly into a retrieval embedding via contrastive supervision alone, without explicitly resolving the intended visual modifications. The model is trained to produce a vector that matches the target, but it is never asked to work out what the modified target should contain.
How FactEmbed works
Factorized MLLM Embeddings, or FactEmbed, decouples compositional query comprehension from building discriminative embeddings. Instead of projecting the composed query into a single embedding without internal reasoning, it uses a set of factorized latents. These latents simulate the intended visual modifications through an intermediate target imagination objective. According to the abstract, this encourages the latents to ground into target semantics before the model forms discriminative representations for retrieval.
A second step carries these factorized representations through the rest of the model. The authors optimize a factor-level contrastive loss and, at the same time, distill the decomposed representations into a deeper attention layer of the MLLM. The point is that the factorized understanding does not stay isolated in the latents but shapes the representations the model uses for retrieval.
Two embeddings from one pass
For efficient inference, FactEmbed keeps the [EOS] token as a global-level embedding built on top of the factorized latents. The mean-pooled factorized latents get a second job: they rerank a short candidate list. Because both are produced by the same forward pass, the abstract says the reranking comes at negligible cost.
For anyone running a retrieval system, this is the practical part of the design. The global embedding does the search, and the latent-based score refines only a short list of candidates, with no second model and no extra forward pass.
What the abstract leaves open
The abstract states that extensive evaluations show FactEmbed outperforms prior works across CIR benchmarks. That is the authors’ own claim, and the abstract gives it without figures.
It does not name the CIR benchmarks, report scores or margins over prior works, or identify the MLLM that FactEmbed is built on. It also does not say how many factorized latents are used, which deeper attention layer receives the distilled representations, or how the cost of reranking was measured beyond calling it negligible. The paper lists its code repository: FactEmbed on GitHub.
Source: Factorized MLLM Embeddings for Composed Image Retrieval (arXiv 2611.05193)