Treasure Hunting Domain
(The blue dot indicates the user's gaze fixation, which was not shown to the user during the study)
In this domain, the user's task is to find some treasures that are hidden in a 3D castle. The user can walk around inside the castle and move objects. The user needs to consult with a remote "expert" (i.e., an artificial agent) to find the treasures. The expert has some knowledge about the treasures but can not see the castle. The user has to talk to the expert for advice about finding the treasures.
Collected Data
- The collected data includes users' speech and accompanying gaze fixations (with possibly selected objects identified by the system).
- Users' speech was transcribed.
- Each speech-gaze instance (speech and its accompanying gaze fixations) was annotated whether any word in the speech refers to a fixated object in the accompanying gaze fixations.
- Download data used for Languge Processing and Acquisition (data description):
- data w/o audio (28.1M)
- data w/ audio (564M)
- Download data used for Reference Resolution (data description):
- data w/o audio (11.8M)
