Image reranking is effective for improving the performance of a text-based image search. However, existing reranking algorithms are limited for two main reasons: 1) the textual meta-data associated with images is often mismatched with their actual visual content and 2) the extracted visual features do not accurately describe the semantic similarities between images. Recently, user click information has been used in image reranking, because clicks have been shown to more accurately describe the relevance of retrieved images to search queries. However, a critical problem for click-based methods is the lack of click data, since only a small number of web images have actually been clicked on by users. Therefore, we aim to solve this problem by predicting image clicks. We propose a multimodal hypergraph learning-based sparse coding method for image click prediction, and apply the obtained click data to the reranking of images. We adopt a hypergraph to build a group of manifolds, which explore the complementarily of different features through a group of weights. Unlike a graph that has an edge between two vertices, a hyperedge in a hypergraph connects a set of vertices, and helps preserve the local smoothness of the constructed sparse codes. An alternating optimization procedure is then performed, and the weights of different modalities and the sparse codes are simultaneously obtained. Finally, a voting strategy is used to describe the predicted click as a binary event (click or no click), from the images’ corresponding sparse codes. Thorough empirical studies on a large-scale database including nearly 330K images demonstrate the effectiveness of our approach for click prediction when compared with several other methods. Additional image reranking experiments on real world data show the use of click prediction is beneficial to improving the performance of prominent graph-based image reranking algorithms.

Image reranking, click, manifolds, sparse codes

[1] Xiaogang Wang, Ke Liu et.al, “Web Image Re-Ranking UsingQuery-Specific Semantic Signatures”, IEEE Transactions on Pattern Analysis and Machine Intelligence Volume: 36 , Issue: 4 April 2014.
[2] Xinmei Tian, Dacheng Tao et.al, “Active Re-ranking for Web Image Search”, IEEE Transactions on Image Processing, Vol. 19, No. 3, March 2010.
[3] J.Cui, F. Wen, et.al, “Real time Google and live image search reranking”, The 16th ACM international conference on Multimedia, Pages 729-732, 2008.
[4] X. Tang, K. Liu, J. Cui, et. a, “Intent Search: Capturing User Intention for One-Click Internet Image Search”, IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 34, No.7 pages 1342 – 1353, July 2012.
[5] Y. Rui, T. S. Huang, M. Ortega, et.al, “Relevance feedback: a power tool for interactive Content-based image retrieval”, IEEE Transactions on Circuits and Systems for Video Technology, 1998.
[6] N. Rasiwasia, P. J. Moreno, et.al, “Bridging the gap: Query by semantic example”, IEEE Transactions. On Multimedia, vol. 9, no. 5, pages.923 -938, August 2007.
[7] Xin Jin, JieboLuo,Jie Yu et. al, “Reinforce Similarity Integration in Image Rich Information Network”, IEEE Transactions on Knowledge & Data Engineering, vol.25, Issue No.02, Feb 2013.
[8] E. Bart and S. Ullman. Single-example learning of novel classes using representation by similarity. In Proc. BMVC, 2005.
[9] D. Tao, X. Tang, X. Li, and X. Wu. “Asymmetric bagging and random subspace for support vector machines-based relevance feedback in image retrieval”, IEEE Trans. on Pattern Analysis and Machine Intelligence, 2006.
[10] A.W.M. Smeulders, M. Worring, S. Santini, et. al, “Content-Based Image Retrieval,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 22, no. 12, pp. 1349-1380, Dec. 2000.