Technical Approach - Visual and Acoustic Surveillance and Monitoring
- Site-model based image stabilization.
Prior research in collaboration with ARL on image stabilization
for automatic target acquisition (ATA) revealed
that high-accuracy image alignment is
very important for subsequent detection and tracking tasks. For some applications,
even subpixel misalignment between image frames may
cause ATA algorithms to fail.
In wide-area surveillance,
a person or small vehicle in a surveillance image may be only several pixels across and
the camera platform may vibrate due to wind and/or strong impacts on
the ground. We expect a high-accuracy camera stabilization capability
to be even more critical in this situation. Additionally, in surveillance systems in which the
visual sensors are truly mobile, stabilization is again a critical step in the
detection of people and vehicles.
We have developed several image stabilization algorithms over the past
As part of the DARPA UGV/RSTA project, we developed
a 2D feature-based multi-resolution camera motion estimation
These stabilization algorithms will be extended through the use of site models for
surveillance applications. Using a 3D site model for image stabilization has the
- Camera location and
orientation can be accurately determined by camera resection based on the
image domain locations of several control points whose 3D coordinates
with respect to the site are known.
- Task-dependent information (e.g., monitoring a building entrance for
activity) can be used to choose the points and/or surfaces stabilized
by the algorithm.
In the surveillance system, the acquired video sequence will be registered to
the site model. Task-specific control points whose 3D coordinates are known from the
site model will be tracked from frame to frame and used in the registration
of the video sequence to the site model. We expect the registration
procedure to achieve high accuracy. We will develop a real-time,
high-accuracy feature tracking algorithm for video sequence to site model
- Wide area detection of people and vehicles using
visual and acoustic sensors.
Detection, recognition, and motion analysis of people and vehicles form the core
tasks of wide-area surveillance and monitoring.
For the detection task, we plan to use
- changes between video frames
to detect moving objects,
- changes between the acquired video and
reference images stored in the site model to detect slow-motion
- unexpected acoustic signals to detect potential
Recognition will involve fusing cues from the site model with information from visual and acoustic
features of the target(s), while moving people and vehicles will be detected
use 3D cues from the site model combined with feature tracking results
to estimate location, heading, and speed.
- Narrow-area recognition of activities.
Our research on narrow-area recognition of human activities
addresses the problems of:
- Detection, segmentation and tracking of people (and their
parts) in color video and
- Developing structural models for single-person and multi-person
activities, with emphasis on entering, exiting, carrying and
Our proposed research on detection and tracking of people is based
on segmenting that part of each image predicted to contain the moving
person, and finding as many natural body parts as possible using a combination
of motion-based tracking, and shape and color analysis.
We plan to employ a novel hierarchical, region-based background subtraction
method to focus on that part of the image predicted to be the
person. Current versions of
these programs currently operate at 10-15 frames per second on a PC system.
Our proposed research involves integration of this segmentation process into a tracking
framework that employs a generic model of the human body and its parts to find (through
limited searches over the combinatorial space defined by the hierarchical segmentations)
body parts and to track them both through the image sequence and, when possible, in 3-D.
Tracking will be based on identifying short-term stable features on the surfaces of the
visible body parts, dynamically updating this set as the sequence progresses. Tracking will
employ models of the dynamics of the motion being observed (walking, bending,
reaching, etc.) under control of the high-level system described below.
- Recognizing human activity.
We propose to develop a mixed statistical and structural approach to the
of human actions.
The models will be grounded in statistical primitive action
models, in which the movements of individual body parts in a body-centered
frame of reference are associated with ``primitive'' body part actions.
These primitive action models are augmented with a theory of
attachment that is used to determine how and when a person is moving with
an object (as opposed to a chance visual coincidence between the
instantaneous views of a human and an object in the scene). We will
specifically be concerned with three types of attachment corresponding,
roughly, to objects with handles (briefcases, suitcases) that are carried
in one hand, bags and boxes that are carried with both arms and hands, and
stick-like objects that are carried in one or two hands. The theory will
also model how such objects are ``picked up'' and ``put down.''
Structural models are compositional activities involving the
coordinated action of many body parts and sequential constraints on
primitive or constituent composite body actions.
Primitive body actions will be recognized using a robust estimation
algorithm for indexing into databases of such linear models and
simultaneously extrapolating motion descriptions to previously unencountered
The theory of attachment, used to recognize carrying activities,
will be based both on constraints on body part
motion that result from different types of carrying activity, and on the
recognition of ``objects'' in a time-varying image that move with a person
and whose position and motion are consistent with the hypothesized type of
Our computational approach will
be strongly motion- and body-pose-based, and will not rely on any general
vision capabilities for finding and tracking objects that humans
might be carrying. Instead, we will hypothesize, from body pose and motion,
the type of carrying action, and then search in the image sequence for collections
of features (regions, markings, etc.) that move with the body in a manner
consistent with the hypothesized carrying action.
- High-level system for control and operator interaction.
a wide variety of control knowledge that should be represented in a general way
in order to control the activities of the surveillance system, and the interactions
between the surveillance system and human operators. This control knowledge
will, generally, make reference to both spatial and temporal attributes of the
surveillance site being monitored. We propose to represent this control knowledge
using temporal logic programs, taking advantage of the general database capabilities of
logic programming to
- support the insertion of ancillary data that can
be integrated into situation assessments by the surveillance system,
- specify the conditions
under which control passes from wide-area surveillance to narrow-area surveillance to
requests for human assistance through queries posed to the temporal logic programming
This high-level system will also draw upon the research being conducted under
a DARPA AASERT grant.