Measuring success in the Map Task.

To evaluate communicative success in dialogue an independent method is
required to assess task performance.  Scores on tests of linguistic
attributes can then be compared with how well the participants
completed the task.  

The route produced on the Follower's map provides a useful,
non-linguistic indicator of dialogue performance: successful
communication strategies ought to result in an accurate replication of
the Giver's route on the Follower's map.
However, developing an appropriate, non-subjective rating of Follower
maps is not as straight-forward as might at first appear.
We offer, here, a method  of scoring Follower maps.

Incorrect Entity Score

The Incorrect Entity -- or IE -- score focuses on feature negotiation
rather than absolute accuracy.  It is based on the number of features
which fail to be negotiated correctly: the higher the score, the more
inaccurate the map. 
Features are viewed as route `pivots', the primary elements around
which the route is constructed.  Since dialogue participants are told
"to avoid obstacles" then failing to navigate around one of the
landmarks is clearly an error, and the dialogue is consequently
penalised.
Marks for errors vary along two independent dimensions:
the sharedness of the feature and the grossness of the mistake.

Sharedness
Features can be Shared (S), or Unshared, with unshared features
appearing on either the Giver's (UG) or the Follower's map (UF).
The IE scoring system is weighted so that more penalties are incurred
if an S feature is incorrectly navigated than if a UF feature is
incorrect: it should be easier to negotiate a feature that both
participants know about.  Similarly, more points are incurred if a UF
feature is incorrect than if a UG feature is incorrect: it is easier
to work out where the route goes when you have the missing feature
(you have the feature if not the route) than when trying to draw
round a feature which you don't have, as you have direct knowledge of
neither feature nor route.

Size of Error
The size of the IE score depends on how badly the Follower negotiates
the landmark, or feature.
Failure to make any sort of attempt to avoid a feature receives a
maximum penalty.
Attempts at avoiding the obstacle which are nevertheless classified as
incorrect are subdivided into `Bad' and `Good' misses.
  Bad Miss:
  Features are scored as incorrect if the route drawn goes through the
  landmark (feature) or if the route is taken to the wrong side of the
  feature.  (The feature label is not considered a part of the
  landmark, so it is allowable for the route to go through writing.)
  Good Miss:
  Features are scored as `Good' misses if the drawn route clips the
  edge of the feature it is meant to go around, or if the route is
  taken too widely, where too widely is defined as more than half the
  height or width of an average landmark.


The following table depicts the weighted IE scores for features of
different information sharedness which are replicated on the
Follower's map with varying levels of inaccuracy.


Sharedness	No Attempt	         Attempt
				Good Miss	Bad Miss

SHared		3		2		2
UFollower	2		1		1
UGiver		1		.5		1