Geo-Temporal Mining and Searching of Events from Web-based Image Collections

Abstract

The proliferation of Web- and social media-based photo-sharing applications have not only opened many possibilities but also resulted in new needs and challenges. They have resulted in a large amount of personal photos being available for public access. One of the most interesting characteristics of these data is that they are surrounded by 1) textual annotations, also called tags, which are intended to describe and categorize, by collective user efforts, the uploaded resources 2) temporal information referring to when a pictures has been taken and often by 3) a locational information describing where the picture has been taken. Despite the recent developments and technological advances in Web-based media-sharing applications, the continuously increasing amount of available information has made the access to the photos a demanding task. In general, we can address this challenge by allowing photo collections to be organized and browsed through the concept of events. We also believe most users are familiar with searching photo collections using events as starting points.

Aiming at supporting the detection and search of event-related photos, this thesis proposes a novel framework for extracting pictures related to real-life events from a collection of Web-images by leveraging on their temporal geographical and textual annotations and comparing the proposed approach with existing related state-ofthe- art approaches. Second, a set of geographical features is proposed describing the characteristics of the geographical profile of query terms deriving concepts from exploratory analysis. Third, the thesis provides two different tag-based search framework to improve the effectiveness of searching images related to events. The first framework is based on temporal and geographical proximity of query terms to the temporal neighbors of a given timestamped query, while the second framework is based on a novel machine-learning based query expansion method combining the heterogeneous textual, temporal and geographical similarity between query terms and candidate expansion terms for the selection of the expansion terms given a free text textual query.

All the proposed methods have been evaluated by performing extensive experiments on real data gathered from media-sharing applications on the Web. Where possible, comparison with related techniques has been performed to reinforce the validity of the presented approaches. The proposed methods have shown promising results in both the extraction and clustering of event-related images and searching eventrelated pictures by using metrics from the state of the art.