Moving Vistas: Exploiting Motion for Describing Scenes

Nitesh Shroff, Pavan Turaga and Rama Chellappa, CVPR 2010
Dynamic Scene Dataset 

Abstract

  • Scene recognition in an unconstrained setting is an open and challenging problem with wide applications. In this paper, we study the role of scene dynamics for improved representation of scenes. We subsequently propose dynamic attributes which can be augmented with spatial attributes of a scene for semantically meaningful categorization of dynamic scenes. We further explore accurate and generalizable computational models for characterizing the dynamics of unconstrained scenes. The large intra-class variation due to unconstrained settings and the complex underlying physics present challenging problems in modeling scene dynamics. Motivated by these factors, we propose using the theory of chaotic systems to capture dynamics. Due to the lack of a suitable dataset, we compiled a dataset of in-the-wild dynamic scenes. Experimental results show that the proposed framework leads to the best classification rate among other well-known dynamic modeling techniques. We also show how these dynamic features provide a means to describe dynamic scenes with motion-attributes, which then leads to meaningful organization of the video data

Problem Definition and Motivation

  • Dynamic Scene Recognition

  • Dynamics of scene reveals further information !!

  • Motion of scene elements improve or deteriorate classification?

  • How to expand the scope of scene classification to videos?

Modeling Scene Dynamics

  • What makes it difficult?

    Scenes are unconstrained and in-the-wild – Large variation in scale, view, illumination, background, etc.
    Underlying physics of motion – too complicated or very little is understood of them.

  • Is there a Ray of hope?

    Underlying process not entirely random but has deterministic component.

  • Can we characterize motion at a global level ?

    Yes, using dynamic attributes and chaotic invariants.

Dynamic Attributes

  • Degree of Busyness: Amount of activity in the video.

    Highly busy: Sea-waves or Traffic scene –high degree of detailed motion patterns.
    Low busyness: Waterfall – largely unchanging and motion typically in a small portion

  • Degree of Flow Granularity of the structural elements that undergo motion.

    Coarse: falling rocks in a landslide.
    Fine: waves in an ocean.

  • Degree of Regularity of motion of structural elements.

    Irregular or random motion: chaotic traffic
    Regular motion: smooth traffic

Modeling Dynamics Using Chaotic Invariants

  • Requires No assumptions

  • Purely from the sequence of observations.

  • Fundamental notion – all variables in a influence one another.

  • Constructs state variables from given time series

  • Estimate embedding dimension and delay

  • Reconstruct the phase space. Then characterize it using invariants

Chaotic Invariants

  • Lyapunov Exponent: Rate of separation of nearby trajectories.

  • Correlation Integral: Density of phase space.

  • Correlation Dimension: Change in the density of phase space

Dataset

  • Dynamic Scene Dataset consisting of 13 classes with 10 videos per class.

  • In the figure above, top 4 rows show an example from 12 out of the 13 classes in the order Avalanche, Iceberg Collapse, Landslide, Volcano eruption, Chaotic traffic, Smooth traffic, Forest fire, Waterfall, Boiling water, Fountain, Waves and Whirlpool.

  • The bottom row shows frames from 4 videos of the 13th class Tornado. Notice the large intra-class variation in the dataset. Large variations in the background, illumination, scale and view can be easily seen. Similar variations are present in each of the classes.

References

  • A.Oliva and A. Torralba. Modeling the shape of the scene: A holistic representation of the spatial envelope. IJCV, 2001

  • M. Perc. The dynamics of human gait. European journal of physics, 26(3),525 - 534, 2005

  • G. Doretto, A. Chiuso, Y. Wu, and S. Soatto. Dynamic textures, IJCV, 2003

  • S. Ali, A. Basharat, and M. Shah. Chaotic Invariants for Human Action Recognition. ICCV, 2007.

CVPR Poster

Dynamic Scene Poster