One approach to acquiring biological data of insects, fish and animals in
their natural setting is to use embedded video cameras to observe the
presence and behaviour of the organisms. This presents three problems for
the biologists: 1) the quantities of acquired raw data can approach 1-2 gigabyte
per hour per camera, even with substantial compression,
2) extracting information from the data can require substantial
computer programming skills and 3) adapting the programs as scientific
questions evolve or new questions arise can be quite time-consuming.
These three problems sit at the heart of the motivations for the
Intelligent Information Management workprogram focus.
As a practical framework for focussing and integrating research into
issues of Intelligent Information Management,
this project will investigate methods for capture, storage, analysis
and query of multiple video streams of undersea environmental videos.
The video streams are being generated as part of an ecological monitoring
effort and also form a resource base for marine biologists.
The cameras (e.g. 10) generate on the order of 20 gigabytes per
hour of video data and, with 12+ hours of usable daylight, this
could lead to on the order of 100 terabytes of data per year.
This project will investigate methods for how a combination
of computer vision, semantic web, database storage and query
and workflow methods can be used to extract
useful information and make it accessible to non-programming scientists.
A typical image from one of the monitored areas is:
This marine monitoring scenario is an application framework that motivates and
unifies the scientific research of this project, and which will supply
the data for evaluation of that research.
Complicating the issue is the 3000 different species of fish that have been
observed in the area.
However, the core goal of the project is to investigate research issues,
in particular, into knowledge and data representation, database indexing,
flexible data processing workflow architectures, computer vision
based video analysis and query answering.
The real issue underlying the project is how to extract useful scientific
information from the enormous amount of data provided by the video cameras,
and, in fact, see if this enormous amount of data can actually allow marine
scientists to answer new sorts of scientific questions.
This point is made eloquently in the recent electronic
book "The Fourth Paradigm: Data-Intensive
Scientific Discovery", Tony Hey, Stewart Tansley, and Kristin Tolle (Eds),
The book argues that the next wave of scientific development will be data-driven,
as contrasted with observation, theory or simulation driven,
and new tools are needed to enable this new wave.
While we are not addressing this big issue in general, we are
investigating tools that turn raw data into analysable, conceptual units
that can be accessed flexibly through a knowledge-driven user interface
and efficiently using high-performance computing resources.
And the goal is to provide this without requiring the target user scientist
(marine biologists here) to also be a programmer.
The particular questions that will be investigated are:
- What are the appropriate forms of ontologies and vocabularies that:
These ontologies and vocabularies include both
computer and human-level descriptions, and are the primary mechanism
to ensure that the different project components can interface with
a consistent understanding.
- describe organisms like the observed fish,
their behaviours, the environmental conditions seen in the videos?
- allow formulation of the questions that the marine biologists
are interested in?
- describe the video data itself?
- describe the computer algorithm capabilities and processing resources
available for use when question answering?
- What storage representations allow for:
- Massive reduction in stored data (e.g. only record the extracted
fish and a background frame, or perhaps only a description of each
fish and a single view of it)?
- Efficient indexing to fish images and segments within the videos?
- Effective re-analysis of previously acquired video data to
answer new scientific queries?
- What computer vision methods are most effective for describing
fish shape, appearance, movement and behaviour? What descriptions are
computable for the environmental conditions seen in the videos?
What machine learning methods are beneficial for discovering optimal
fish detection and categorisation.
- What are the most effective methods for accessing information
previously extracted from the videos, combining user text-based query terms,
computer vision based descriptions and a semantic ontology linking
How can the user generated queries be used to compile new workflow
sequences for extracting new information from the video data?
Does user feedback during database retrieval provide a benefit here?
- What computer processing structures will allow continuous capture,
analysis and storage of the video data without a backlog of unanalysed videos?
How might irrelevant information be removed and redundancy reduced to make the
recorded data tractable in size, and enable the possibility of increased
numbers of cameras in the future?
- How can the processing components be organised to allow
flexible real-time reconfiguration of the processing workflow, as users formulate
new scientific questions?
How can new scientific questions be quickly translated into new processing
workflows that extract the answers to the questions in a manner that
does not require the user to be a scientific programmer?
How can processing resources be redeployed between processing of
live versus pre-stored data processing?
- Which sorts of queries are most effectively answered, such as
fish and behaviour identification or counting? How generally can
we allow users to frame queries, e.g. using some sort of a logical
specification query language? What are the types of query primitives
that can be computed accurately?
To help answer the scientific questions listed above, we will
undertake research into: 1) computer vision methods (for fish detection,
species identification and behaviour recognition),
2) ontology driven semantic user interfaces (to allow non-programming
biologist users to specify queries about the video content),
3) ontology and planning driven automatic workflow construction (for on-line
automatic construction of computational sequences
that answer the biological questions) and 4) computer
hardware and software architecture organisations that can acquire
and process the massive amount of video data efficiently.
We will integrate the individual research components into a
publically usable demonstration system.
This system will allow a marine biologist to formulate a query
in his/her vocabulary (by the semantic interface),
which will be translated into a sequence of image processing
and database access operations (by the workflow compiler), which
are then executed using the database host and query execution machine.
One novel aspect of this project is the location where the data analysis
will take place. In the past, scientists downloaded the datasets or bought
copies on CDs/DVDs. With up to 10E+12 pieces of data, this is now impossible
for most projects. Even with new 1T portable disks, these databases would require
dozens, if not hundreds of drives. So, instead, this project takes the scientist's
questions to the data. In this case, this means the use of an intelligent
web-based user interface, which leads to queries over the database, executed on
a HPC computing system co-located with the data.
A schematic architecture for the integrated project and system can be seen here:
At the centre of the diagram are the three data repositories to be developed and
hosted by NCHC: a repository of previously recorded video data in a computationally
compressed format, a metadata repository of RDF triples recording
previously extracted information from the videos, plus an XML database
of the videos and associated summaries. The content producers for these
repositories are shown at the left side, where image data is captured from
multiple cameras, fish are detected and tracked in the video
(which will require coping with potential problems like: murky water,
algae on the lens, moving background plants and changing lighting conditions).
From the tracked fish, the project will develop processes for recognising
fish species and distribution, and for inferring interactions between individual fish.
The video data could be live or previously captured, depending on the
user requests. Partners UCATANIA and UEDIN will be responsible for the
image and video analysis research.
At the right of the diagram are mainly the user response processes.
A marine biologist will be able to phrase a broad range of fish-related queries
in a reasonably natural way, with the domain ontologies providing re-interpretation
into the system's vocabulary. The queries will lead to either previously
computed results being extracted from the RDF metadata store, or new results being
computed (which could involve a mixture of processing video data - either
previously captured or live - and previously extracted information).
The new results will be generated by workflows compiled `on-the-spot' from the
user query and image processing knowledge, using a set of domain, user, fish and
capability ontologies. Partners CWI and UEDIN are responsible for the
query, planning and workflow construction processes.
The workflows would then be executed on a flexible architecture,
which may involve more that one processor (NCHC is responsible for this).
An important workflow component is the database query engine.
For this project, we do not anticipate undertaking any research into
high-performance database query. Because of the quantity of the
data, we are planning for a distributed database and distributed subqueries
over that data. However, at the moment, we propose to use multiple distributed
instances of standard query engines, i.e. SPARQL for the RDF component,
XQuery for the XML component.
As part of the project, we will develop a public web-accessable database
of the videos and their associated computed descriptors (e.g. XML and RDF).
We will develop a web interface that will allow users anywhere to
access live or previously stored videos or compose queries about the
previously stored data.
Extensive performance experiments will be performed to provide statistics
on the accuracy, ease and speed of retrieval of information from the database.
Given a successful outcome, as well as producing new research results,
we will demonstrate a potential environmental analysis tool suitable
for answering questions like:
- What species and numbers of fish appeared in the last N days?
- What unrecognised fish were detected? Do they cluster by appearance?
- Show me examples of fish from species X?
- Show me examples of a fish with description X?
- What other species were also present when species X was seen?
- Are the observed numbers of species X increasing in the past 3 years?
While the focus of the project is based on very specific marine eco-system
video data and environmental study tasks, this has been selected as a
particular use case to demonstrate the more general techniques being
developed within the project. The strength of the project is not only in
going beyond the state of the art in the expertises of the different
partners, but in combining their strengths to form an end-to-end solution
- a thin bridge over the semantic gap.
As a whole, the research undertaken here is partially
applicable to other video-based monitoring applications, e.g.
video-based monitoring of the behaviour of farm and wild land-based animals,
people in shops, secure area surveillance, etc.
Furthermore, several of the core technologies being investigated and
developed here should have wider applicability, such as the knowledge
driven expert query interfaces, computational workflow compilation
from user task ontologies, hierarchical object recognition, target
detection and tracking in unfavourable image data, automatic workflow
distribution over multiple processors.
While we do not expect that the project results will generalise directly,
it is at least clear that the technologies listed just above address
research issues encountered in other applications.