CAVIAR Test Case Scenarios

Video clips created July 11, 2003 and January 20, 2004

 

Introduction

For the CAVIAR project a number of video clips were recorded acting out the different scenarios of interest. These include people walking alone, meeting with others, window shopping, entering and exitting shops, fighting and passing out and last, but not least, leaving a package in a public place.

The first section of video clips were filmed for the CAVIAR project with a wide angle camera lens in the entrance lobby of the INRIA Labs at Grenoble, France. The resolution is half-resolution PAL standard (384 x 288 pixels, 25 frames per second) and compressed using MPEG2. The file sizes are mostly between 6 and 12 MB, a few up to 21 MB.

A typical frame from the image sequences is below. It shows three individual boxes (yellow) and one group box (green). There are several people in the video sequence that are not boxed because they do not move over the course of the sequence.

 

The second set of data also used a wide angle lens along and across the hallway in a shopping centre in Lisbon. For each sequence, there are are two time synchronised videos, one with the view across and the other along the hallway. The resolution is half-resolution PAL standard (384 x 288 pixels, 25 frames per second) and compressed using MPEG2. The MPEG file sizes are mostly between 6 and 12 MB, a few up to 21 MB.

All data is publicly available and on this page they can be downloaded in MPEG2 format or split into JPEGs. If you publish results using the data, please acknowledge the data as coming from the EC Funded CAVIAR project/IST 2001 37540, found at URL: http://homepages.inf.ed.ac.uk/rbf/CAVIAR/. Usability is Creative Commons BY-SA:

 

The ground truth for these sequences was found by hand-labeling the images, as in the example shown above. The JAVA programs for the interactive labeller can be found here (unsupported). This is the userguide.

The finite state automata that describes the allowable roles, activities and sequence of situations in each context is here.

The XML grammar for the ground truth labeling files is here. Note that these XML files are large (eg. averaging 1.5MB), so viewing one of the files in a browser window can take a minute or more to load up because of the default formatting of the XML. The groundtruth XML is based on the CVML language (CVML- An XML-based Computer Vision Markup Language), which was presented at IPCR. CVML is described in more detail at mindmakers.org. The CVML detail includes a C++ based CoreLibrary which supports reading and writing the XML (amongst other things).

A discussion about the INRIA datasets was presented at PETS04. The ground truth labelling notation discussed in the paper has changed to XML and some minor details have changed, but most of the concepts and discussion are still useful.

Some tracked targets in 17 of the sequences have had additional information about their head, gaze direction, hand, feet and shoulder positions added. (Not all because we have no more time and money at the moment.)

INRIA Sequences
Datafile         Sequence            Comments

fomdgt2.xml  Fight_OneManDown        only targets marked: 0,1,4
mc1gt.xml    Meet_Crowd              all marked
mwt2gt.xml   Meet_WalkTogether2      all marked
fcgt.xml     Fight_Chase             all marked
mws1gt.xml   Meet_WalkSplit          all marked
ms3ggt.xml   Meet_Split_3rdGuy       only targets marked: 0
fra1gt.xml   Fight_RunAway1          only targets marked: 7
fra2gt.xml   Fight_RunAway2          only targets marked: 4

Lisbon Sequences
Datafile         Sequence            Comments

c2es1gt.xml  TwoEnterShop1cor        no gaze directions annotated
fsa1gt.xml   ShopAssistant1front     all marked
csa1gt.xml   ShopAssistant1cor       all marked
c3ps1gt.xml  ThreePastShop1cor       all marked
f3ps1gt.xml  ThreePastShop1front     all marked
f2es1gt.xml  TwoEnterShop1front      all marked
fosow2gt.xml OneShopOneWait2front    all marked
cosow2gt.xml OneShopOneWait2cor      all marked
cosme2gt.xml OneStopMoveEnter2cor    only targets marked: 0,2,11
c3ps2gt.xml  ThreePastShop2cor       all marked
f3ps2gt.xml  ThreePastShop2front     all marked

An example of a marked up frame with heads, gaze, hands, feet and shoulders is:

Clips from INRIA (1^st Set)

 

Six basic scenarios were acted out by the CAVIAR team members. Most clips start with one member showing in body sign language the scene number. This can be used for calibration or be removed at will.

For people interested in the ground plane homography, the mapping can be computed from this information. The image pixel positions will depend on your image scaling. Here the image sizes are the jpg 384x288. For the image just below the corresponding pixel positions are:

Point	(Col,Row) (pixels)	(X,Y) (cm)
1	(64,88)	(0,671.5)
2	(211,40)	(1116,670)
3	(349,184)	(1545,190)
4	(39,187)	(0,0)

 

Walking

One person walking – straight line	Walk1.mpg (7 Mb) Ground truth XML version: wk1gt.xml JPEGS: Walk1_jpg.tar.gz (13 Mb)
One person walking – straight line and return	Walk2.mpg (12 Mb) Ground truth XML version: wk2gt.xml JPEGS: Walk2_jpg.tar.gz (22 Mb)
One person walking – B-line	Walk3.mpg (16 Mb) Ground truth XML version: wk3gt.xml JPEGS: Walk3_jpg.tar.gz (29 Mb)

 

Browsing

Person browsing back and forth	Browse1.mpg (12 Mb) Ground truth XML version: br1gt.xml JPEGS: Browse1_jpg.tar.gz (22 Mb)
Person browsing and reading for a while	Browse2.mpg (10 Mb) Ground truth XML version: br2gt.xml JPEGS: Browse2_jpg.tar.gz (18 Mb)
Person browsing and reading with back turned	Browse3.mpg (11 Mb) Ground truth XML version: br3gt.xml JPEGS: Browse3_jpg.tar.gz (19 Mb)
Person browsing reception desk	Browse4.mpg (13 Mb) Ground truth XML version: br4gt.xml JPEGS: Browse4_jpg.tar.gz (24 Mb)
Person browsing while waiting short	Browse_WhileWaiting1.mpg (9 Mb) Ground truth XML version: bww1gt.xml JPEGS: Browse_WhileWaiting1_jpg.tar.gz (16 Mb)
Person browsing while waiting long	Browse_WhileWaiting2.mpg (21 Mb) Ground truth XML version: bww2gt.xml JPEGS: Browse_WhileWaiting2_jpg.tar.gz (39 Mb)

 

Resting, slumping or fainting

Person resting in chair	Rest_InChair.mpg (12 Mb) Ground truth XML version: ricgt.xml JPEGS: Rest_InChair_jpg.tar.gz (21 Mb)
Person slump on floor	Rest_SlumpOnFloor.mpg (11 Mb) Ground truth XML version: rsfgt.xml JPEGS: Rest_SlumpOnFloor_jpg.tar.gz (19 Mb)
Person wiggle on floor	Rest_WiggleOnFloor.mpg (15 Mb) Ground truth XML version: rwgt.xml JPEGS: Rest_WiggleOnFloor_jpg.tar.gz (28 Mb)
Person fall down immobile	Rest_FallOnFloor.mpg (12 Mb) Ground truth XML version: rffgt.xml JPEGS: Rest_FallOnFloor_jpg.tar.gz (21 Mb)

 

Leaving bags behind

Person leaving bag by wall	LeftBag.mpg (17 Mb) Ground truth XML version: lb1gt.xml JPEGS: LeftBag_jpg.tar.gz (31 Mb)
Person leaving bag at chairs	LeftBag_AtChair.mpg (13 Mb) Ground truth XML version: lb2gt.xml JPEGS: LeftBag_AtChair_jpg.tar.gz (24 Mb)
Person leaving bag behind chairs	LeftBag_BehindChair.mpg (13 Mb) Ground truth XML version: lbbcgt.xml JPEGS: LeftBag_BehindChair_jpg.tar.gz (23 Mb)
Person leaving box	LeftBox.mpg (10 Mb) Ground truth XML version: lbgt.xml JPEGS: LeftBox_jpg.tar.gz (18 Mb)
Person leaving bag but then pick it up again	LeftBag_PickedUp.mpg (16 Mb) Ground truth XML version: lbpugt.xml JPEGS: LeftBag_PickedUp_jpg.tar.gz (29 Mb)

 

People/groups meeting, walking together and splitting up

Two people meet and walk together	Meet_WalkTogether1.mpg (8 Mb) Ground truth XML version: mwt1gt.xml JPEGS: Meet_WalkTogether1_jpg.tar.gz (15 Mb)
Two other people meet and walk together	Meet_WalkTogether2.mpg (9 Mb) Ground truth XML version: mwt2gt.xml JPEGS: Meet_WalkTogether2_jpg.tar.gz (17 Mb)
Two people meet, walk together and split	Meet_WalkSplit.mpg (7 Mb) Ground truth XML version: mws1gt.xml JPEGS: Meet_WalkSplit_jpg.tar.gz (13 Mb)
Two people meet, walk, split with third person	Meet_Split_3rdGuy.mpg (11 Mb) Ground truth XML version: ms3ggt.xml JPEGS: Meet_Split_3rdGuy_jpg.tar.gz (19 Mb)
Crowd of four people meet, walk and split	Meet_Crowd.mpg (6 Mb) Ground truth XML version: mc1gt.xml JPEGS: Meet_Crowd_jpg.tar.gz (10 Mb)
Two people enter walk and split	mpg missing Ground truth XML version: spgt.xml JPEGS: Split_jpg.tar.gz (11 Mb)

 

Two people fighting

Two people meet, fight and run away	Fight_RunAway1.mpg (6 Mb) Ground truth XML version: fra1gt.xml JPEGS: Fight_RunAway1_jpg.tar.gz (12 Mb)
Two other people meet, fight and run away	Fight_RunAway2.mpg (6 Mb) Ground truth XML version: fra2gt.xml JPEGS: Fight_RunAway2_jpg.tar.gz (12 Mb)
Two people meet, fight, one down, other runs away	Fight_OneManDown.mpg (11 Mb) Ground truth XML version: fomdgt1.xml fomdgt2.xml fomdgt3.xml JPEGS: Fight_OneManDown_jpg.tar.gz (20 Mb)
Two people meet, fight and chase each other	Fight_Chase.mpg (5 Mb) Ground truth XML version: fcgt.xml JPEGS: Fight_Chase_jpg.tar.gz (9 Mb)

 

Clips from Shopping Center in Portugal (2^nd Set)

 

In the second set of experiments each clip was recorder from two different points of view. The first one shows a view of the corridor, while the second shows a frontal view of the scenario. The two video sequences should be time synchronised frame by frame. However, each video set may start at a slightly different time, so you need to figure out the frame correspondences. The time-code in the upper left of the images gives the necessary information. During capture and digitisation, a frame segmentation error means that some frames are duplicated and the expected next frame is missing. Thus, the overall rate is correct, but occasionally 2 consecutive frames are identical.

This set of sequences are longer (1500 frames on average), containing more individuals and groups than the first set. Example synchronized images are shown here:

For people interested in the ground plane homography, the mapping can be computed from this information. The image pixel positions will depend on your image scaling. Here the image sizes are the jpg 384x288. For the image just below the corresponding pixel positions are:

Point	(Col,Row) (pixels)	(X,Y) (cm)
1	(91,163)	(000,975 )
2	(241,163)	(290,975 )
3	(98,266)	(000,-110 )
4	(322,265)	(290,-110 )
5	(60,153)	(000,000 )
6	(359,153)	(000,975 )
7	(50,201)	(382,098)
8	(367,200)	(382,878)

	CORRIDOR VIEW	FRONT VIEW
Couple walking along corridor browsing, persons going inside and coming out of stores	WalkByShop1cor.mpg (14 Mb) Ground truth XML version: cwbs1gt.xml JPEGS: WalkByShop1cor.tar.gz (48 Mb)	WalkByShop1front.mpg (14 Mb)  Ground truth XML version: cwbs1gt.xml JPEGS: WalkByShop1front.tar.gz (56 Mb)
Two persons cross paths at the entrance of a store, couple walking on the corridor	EnterExitCrossingPaths1cor.mpg (2  Mb)  Ground truth XML version: ceecp1gt.xml JPEGS: EnterExitCrossingPaths1cor.tar.gz (8 Mb)	EnterExitCrossingPaths1front.mpg ( 2 Mb)  Ground truth XML version: feecp1gt.xml JPEGS: EnterExitCrossingPaths1front.tar.gz (10 Mb)
Two persons cross paths at the entrance of a store	EnterExitCrossingPaths2cor.mpg ( 3 Mb)  Ground truth XML version: ceecp2gt.xml JPEGS: EnterExitCrossingPaths2cor.tar.gz (10 Mb)	EnterExitCrossingPaths2front.mpg ( 3 Mb)  Ground truth XML version: feecp2gt.xml JPEGS: EnterExitCrossingPaths2front.tar.gz ( 12 Mb)
Person goes outside a store. Visible on corridor view only: Three persons walking together in the corridor	OneLeaveShop1cor.mpg (2 Mb)  Ground truth XML version: cols1gt.xml JPEGS: OneLeaveShop1cor.tar.gz ( 6 Mb)	OneLeaveShop1front.mpg (2 Mb)  Ground truth XML version: fols1gt.xml JPEGS: OneLeaveShop1front.tar.gz ( 8 Mb)
Person goes outside a store. Visible on corridor view only: Four persons walking together in the corridor	OneLeaveShop2cor.mpg (6 Mb)  Ground truth XML version: cols2gt.xml JPEGS: OneLeaveShop2cor.tar.gz ( 23 Mb)	OneLeaveShop2front.mpg (6 Mb)  Ground truth XML version: fols2gt.xml JPEGS: OneLeaveShop2front.tar.gz ( 29 Mb)
Person comes out of store and later reenters. Visible on corridor view only: Person coming out of store and walking on the corridor	OneLeaveShopReenter1cor.mpg ( 2 Mb)  Ground truth XML version: colsr1gt.xml JPEGS: OneLeaveShopReenter1cor.tar.gz ( 8 Mb)	OneLeaveShopReenter1front.mpg (2 Mb)  Ground truth XML version: folsr1gt.xml JPEGS: OneLeaveShopReenter1front.tar.gz ( 10 Mb)
Person comes out of store and later reenters. Visible on corridor view only: Four persons walk on the corridor.	OneLeaveShopReenter2cor.mpg ( 3 Mb)  Ground truth XML version: colsr2gt.xml JPEGS: OneLeaveShopReenter2cor.tar.gz ( 12 Mb)	OneLeaveShopReenter2front.mpg (3 Mb)  Ground truth XML version: folsr2gt.xml JPEGS: OneLeaveShopReenter2front.tar.gz ( 15 Mb)
Couple walking on the corridor, one goes inside a store, the other waits outside, later they rejoin and leave together.	OneShopOneWait1cor.mpg ( 8 Mb)  Ground truth XML version: cosow1gt.xml JPEGS: OneShopOneWait1cor.tar.gz ( 30 Mb)	OneShopOneWait1front.mpg ( 8 Mb)  Ground truth XML version: fosow1gt.xml JPEGS: OneShopOneWait1front.tar.gz ( 10 Mb)
Similar to OneShopOneWait1front, but contains various groups of persons walking along the corridor.	OneShopOneWait2cor.mpg ( 5 Mb)  Ground truth XML version: cosow2gt.xml JPEGS: OneShopOneWait2cor.tar.gz ( 32 Mb)	OneShopOneWait2front.mpg (8 Mb)  Ground truth XML version: fosow2gt.xml JPEGS: OneShopOneWait2front.tar.gz ( 38 Mb)
Couple goes inside store	OneStopEnter1cor.mpg ( 9 Mb)  Ground truth XML version: cose1gt.xml JPEGS: OneStopEnter1cor.tar.gz ( 31 Mb)	OneStopEnter1front.mpg (9 Mb)  Ground truth XML version: fose1gt.xml JPEGS: OneStopEnter1front.tar.gz ( 38 Mb)
Person browses stores and goes inside and out, couple of walker along the corridor Visible on corridor view only: Two persons go inside stores.	OneStopEnter2cor.mpg ( 16 Mb)  Ground truth XML version: cose2gt.xml JPEGS: OneStopEnter2cor.tar.gz ( 56  Mb)	OneStopEnter2front.mpg (16 Mb)  Ground truth XML version: fose2gt.xml JPEGS: OneStopEnter2front.tar.gz ( 71 Mb)
Person stops outside store, goes inside and out of store. Five groups of people walking along the corridor.	OneStopMoveEnter1cor.mpg (9  Mb)  Ground truth XML version: cosme1gt.xml JPEGS: OneStopMoveEnter1cor.tar.gz ( 37 Mb)	OneStopMoveEnter1front.mpg (9 Mb)  Ground truth XML version: fosme1gt.xml JPEGS: OneStopMoveEnter1front.tar.gz ( 42 Mb)
Person goes inside and out of a store twice. Visible on corridor view only: Group of 4 people comes out store	OneStopMoveEnter2cor.mpg (13 Mb)  Ground truth XML version: cosme2gt.xml JPEGS: OneStopMoveEnter2cor.tar.gz ( 46 Mb)	OneStopMoveEnter2front.mpg (13 Mb)  Ground truth XML version: fosme2gt.xml JPEGS: OneStopMoveEnter2front.tar.gz ( 58 Mb)
Person stops out of store goes inside and out of store Visible on corridor view only: Couple come out of store	OneStopMoveNoEnter1cor.mpg (10  Mb)  Ground truth XML version: cosmne1gt.xml JPEGS: OneStopMoveNoEnter1cor.tar.gz ( 35 Mb)	OneStopMoveNoEnter1front.mpg (10 Mb)  Ground truth XML version: fosmne1gt.xml JPEGS: OneStopMoveNoEnter1front.tar.gz ( 42 Mb)
Person goes inside store, browses, and leaves store. Visible on corridor view only: Couple walking along the corridor	OneStopMoveNoEnter2cor.mpg (6  Mb)  Ground truth XML version: cosmne2gt.xml JPEGS: OneStopMoveNoEnter2cor.tar.gz ( 21 Mb)	OneStopMoveNoEnter2front.mpg (6 Mb)  Ground truth XML version: fosmne2gt.xml JPEGS: OneStopMoveNoEnter2front.tar.gz ( 27 Mb)
Person stops outside an store and continues walking along the corridor	OneStopNoEnter1cor.mpg (4  Mb)  Ground truth XML version: cosne1gt.xml JPEGS: OneStopNoEnter1cor.tar.gz ( 15 Mb)	OneStopNoEnter1front.mpg (4 Mb)  Ground truth XML version: fosne1gt.xml JPEGS: OneStopNoEnter1front.tar.gz ( 18 Mb)
Person stops outside a store and continues walking, another person goes inside a store, browses and then leaves the store.	OneStopNoEnter2cor.mpg (9  Mb)  Ground truth XML version: cosne2gt.xml JPEGS: OneStopNoEnter2cor.tar.gz ( 31 Mb)	OneStopNoEnter2front.mpg (9 Mb)  Ground truth XML version: fosne2gt.xml JPEGS: OneStopNoEnter2front.tar.gz ( 37 Mb)
Person goes inside a store and browse, another person joins and they leave together the store	ShopAssistant1cor.mpg (10  Mb)  Ground truth XML version: csa1gt.xml JPEGS: ShopAssistant1cor.tar.gz ( 26 Mb)	ShopAssistant1front.mpg (10 Mb)  Ground truth XML version: fsa1gt.xml JPEGS: ShopAssistant1front.tar.gz ( 43 Mb)
Person goes inside a store, browse, another person joins later they split,   3 people walking together along the corridor	ShopAssistant2cor.mpg (21  Mb)  Ground truth XML version: csa2gt.xml JPEGS: ShopAssistant2cor.tar.gz ( 69 Mb)	ShopAssistant2front.mpg (21 Mb)  Ground truth XML version: fsa2gt.xml JPEGS: ShopAssistant2front.tar.gz ( 101 Mb)
3 persons walking in the corridor	ThreePastShop1cor.mpg (10  Mb)  Ground truth XML version: c3ps1gt.xml JPEGS: ThreePastShop1cor.tar.gz ( 36 Mb)	ThreePastShop1front.mpg (10 Mb)  Ground truth XML version: f3ps1gt.xml JPEGS: ThreePastShop1front.tar.gz ( 42 Mb)
Another 3 persons walking in the corridor	ThreePastShop2cor.mpg (9  Mb)  Ground truth XML version: c3ps2gt.xml JPEGS: ThreePastShop2cor.tar.gz ( 33 Mb)	ThreePastShop2front.mpg (9 Mb)  Ground truth XML version: f3ps2gt.xml JPEGS: ThreePastShop2front.tar.gz ( 39 Mb)
Couple goes inside a store and later comes out	TwoEnterShop1cor.mpg (10  Mb)  Ground truth XML version: c2es1gt.xml JPEGS: TwoEnterShop1cor.tar.gz ( 35 Mb)	TwoEnterShop1front.mpg (10 Mb)  Ground truth XML version: f2es1gt.xml JPEGS: TwoEnterShop1front.tar.gz ( 43 Mb)
Couple goes inside a store and later comes out	TwoEnterShop2cor.mpg (9  Mb)  Ground truth: c2es2gt.txt Ground truth XML version: c2es2gt.xml JPEGS: TwoEnterShop2cor.tar.gz ( 35 Mb)	TwoEnterShop2front.mpg (9 Mb)   Ground truth: f2es2gt.txt Ground truth XML version: f2es2gt.xml JPEGS: TwoEnterShop2front.tar.gz ( 42 Mb)
Two couples go inside store and later one comes out	TwoEnterShop3cor.mpg (7  Mb)  Ground truth XML version: c2es3gt.xml JPEGS: TwoEnterShop3cor.tar.gz ( 24 Mb)	TwoEnterShop3front.mpg (6 Mb)  Ground truth XML version: f2es3gt.xml JPEGS: TwoEnterShop3front.tar.gz ( 28 Mb)
Couple leaves a store while browsing	TwoLeaveShop1cor.mpg (8  Mb)  Ground truth XML version: c2ls1gt.xml JPEGS: TwoLeaveShop1cor.tar.gz ( 28 Mb)	TwoLeaveShop1front.mpg (8 Mb)  Ground truth XML version: f2ls1gt.xml JPEGS: TwoLeaveShop1front.tar.gz ( 35 Mb)
A couple leaves a store	TwoLeaveShop2cor.mpg ( 4 Mb)  Ground truth XML version: c2ls2gt.xml JPEGS: TwoLeaveShop2cor.tar.gz ( 13 Mb)	TwoLeaveShop2front.mpg (4 Mb)  Ground truth XML version: f2ls2gt.xml JPEGS: TwoLeaveShop2front.tar.gz ( 16 Mb)