Xirong Li

Content-Based Visual Search Learned from Social Media

In a world with increasing amounts of digital pictures, content-based visual search is an important and scientifically challenging problem in ICT research. This thesis tackles the problem by learning from social media. The fundamental question addressed in this thesis is:
what is the value of socially tagged images for visual search?

To that end, we propose the neighbor voting algorithm (Chapter 2) and its multi-feature variant (Chapter 3) to verify whether what people spontaneously say about an image is factually in the pictorial content. The two algorithms are used to find high-quality positive examples for learning automated image taggers. To obtain negative training examples without manual verification, we go beyond the classical random sampling approach by introducing informative negative bootstrapping (Chapter 4). For answering complex visual searches, we introduce the notion of bi-concepts as a retrieval method for unlabeled images in which two concepts are co-occurring (Chapter 5). Finally, as users have their own associations with image semantics, we propose personalized image tagging by jointly exploiting personal tagging history and content-based analysis, optimized through Monte Carlo sampling (Chapter 6).

On the basis of the reported theories, algorithms, and experiments, this thesis has revealed the value of socially tagged images for content-based visual search, providing a basis for uncovering universal knowledge on images and semantics. With the methodologies established, this thesis opens up promising avenues for image search engines which provide access to the semantics of the visual content, but without the need of manual labeling.