New Framework Enhances Text-to-Image Retrieval by Mitigating Visual Hallucinations

Researchers have introduced Diffusion-aware Multi-view Contrastive Learning (DMCL), a framework designed to enhance Diffusion-Augmented Interactive Text-to-Image Retrieval (DAI-TIR) systems by mitigating visual 'hallucinations'—misleading cues generated during the diffusion process that can degrade retrieval performance.

The team, comprising Zhuocheng Zhang from Hunan University, Kangheng Liang and Paul Henderson from the University of Glasgow, along with Guanxuan Li, Richard Mccreadie, and Zijun Long, empirically demonstrated that these hallucinated cues significantly reduce retrieval accuracy.

DMCL addresses this issue by optimizing representations of both the query intent and target images, effectively filtering out these misleading signals. The framework introduces two key training objectives: a Multi-View Query-Target Alignment objective, which emphasizes shared cues across views while filtering out inconsistencies, and a Text-Diffusion Consistency objective, which enhances agreement between text and diffusion queries, reducing sensitivity to mismatches and generative hallucinations.

Attention visualization and geometric embedding-space analyses corroborate the model's ability to discern and disregard irrelevant visual information. Across five standard benchmarks, DMCL consistently improved multi-round retrieval accuracy, achieving gains of up to 7.37% over existing methods.

The authors acknowledge that their current implementation uses a simple additive fusion scheme for query integration and plan to explore more sophisticated fusion techniques in future research. They have also released a large-scale DAI-TIR training dataset to facilitate further investigation in this area.