Context
The need for collaborative indexing and more complexes indexing tools is rapidly increasing in multimedia indexing domain. Many indexes can not be generated by a simple tool and need to be produced by a sequence of more or less sophisticated tools in a given order. These tools are developed and distributed by different research teams with in general really specialized skills on a single media. This makes the exploitation of these tools for cross-media analysis a really difficult challenge and a hard lock for potential research activities on multimedia index mining.
For example, in order to make the transcription of the speech in an automatic way, we may first analyze the audio content type (Speech/Music/Noise), then we may identify the language of speaker in case of speech content, and then we can try to transcribe the speech into text with the corresponding indexer. The integration of these tools into cooperative applications requires the development of an intelligent platform in which the interoperability of different indexing tools is defined.
Figure 1: Example of an indexing chain generated for the Klimt Project (ITEA)
We have developed concepts for a dynamic multimedia indexing platform that allows dynamic tools composition. These tools are modeled as processes with some different potential customized executions according to the client needs. A composite tool is modeled by a tree. The latter defines the order of execution among the nodes or different processes. The goal of this platform is to generate an index in a completely dynamic way as it is explained in the following scenario: given an index, the chaining algorithm must find a subset of tools and their interactions to realize the desired index on any type of source (audio, video and textual contents). Input data is not defined by the user’s request. All the documents available on the platform is potential candidate to be indexed and will be selected automatically by the chaining algorithm.
Overview
An indexing tool does not provide generally the features required for easy integration on a distributed platform. Our idea is to define a wrapper able to reply to the set of standard messages used for communication between that kind of tools. This wrapper is a package of code that can be dynamically linked to the central server. It is composed of the following two generic interfaces:
1. Configuration interface
Each tool will be defined by a specification file written by the tool developer. This describes some properties such as ComponentRole, Description, ExecutablePath, InputType, OutputType, StorageInputPath, StorageOutputPath. This file must be written in XML format.
The configurator interface allows the platform to access these specifications needed for the formulation of messages transmitted between the platform and the distant tool.
2. Control interface
This interface defines the method used by the plat-form to communicate with the tool. It provides functionalities for:
ï€ Launching the distant component,
ï€ Getting component status, and
ï€ Data transferring from and to the component.
Figure 2: Example of communication services
The dynamic composition problem stands in defining and assuming the interoperability between available tools declared in the tool repository to build a given index. Our idea is to describe the Input/output data type of each indexing tool with a semantic description. For instance, the goal of an audio analyzer may be to localize segments of speech, music, noises and silences in an audio file. The integration of this tool in a dynamic chain is based only on the Input/Output data type, i.e. a .wav file as input and an AudioSegmentContent type as output. Semantic types are from the ones defined by the MPEG7 standard when it is possible.
The composition process is based on a chaining algorithm. Based on a given requested index (which is actually a descriptor asked by the user) the chaining algorithm must find a set of tools and their interactions to detect or recognize each occurrence related to this descriptor. The algorithm is recursive and it stops when a primitive multimedia content (audio, videos files) can be used as an input of the last integrated indexer.
We start with a given "Index" and we try to find the unitary indexer able to generate this descriptor type. Once such a tool is found, if the input type of this indexer is a multimedia content, we stop and the given index can be generated by application to all documents corresponding to the input data type. In the other case, if the input type of this indexer is another descriptor type corresponding to data generated by another tool, we repeat the same process recursively, integrating each time a new indexer at the top of the chain, until a multimedia content type is found.
Projects
Contributors
- Bassem Haïdar
- Philippe Joly (contact)
Main publications
Bassem Haidar, Philippe Joly, Jean-Paul Bahsoun. Distributed Opened Cross-Media Indexing Platform. Dans / In : 11th Annual Scientific Conf. on Web Technology, New Media, Communications and Telematics Theory, Methods, Tools and Applications (EUROMEDIA'2005), Toulouse, 11/04/2005-13/04/2005, M Al-Akaidi, L Rothkrantz (Eds.), EUROSIS, B-9052 Ghent-Zwijnaarde, Belgium, ISBN 90-77381-17-1, p. 178-182, avril / april 2005.
Bassem Haidar, Philippe Joly, Siba Haidar. A Graph Based Approach to Automatically Chain Distributed Multimedia Indexing Services. Dans / In : Ninth IASTED Int. Conf. on Internet and Multimedia Systems and Applications, Grindelwald, Switzerland, 21/02/2005-23/02/2005, M.H Hamza (Eds.), ACTA Press, CH-8053 Zurich, Switzerland, ISBN 0-88986-484-5, p. 336-342, février / february 2005.
Bassem Haidar, Philippe Joly, Jean-Paul Bahsoun. An Open Platform for Dynamic Multimedia Indexing. Dans / In : 5th Int. Workshop on Image Analysis for Multimedia Interactive Services (WIAMIS'2004), Lisboa, Portugal, 21/04/2004-23/04/2004, IST/Lisboa-Suvisoft, avril / april 2004.
Bassem Haidar. Services d'indexation multimedia distribués. Thèse de doctorat, Université Paul Sabatier, septembre / september 2005.