AMI Corpus Scenario Explained
from http://dis.tpd.tno.nl/papers/mccowan-ami-mb2005.pdf
In our meeting elicitation scenario, the participants
play the roles of employees in an electronics company that
decides to develop a new type of television remote control
because the ones found in the market are not user friendly,
as well as being unattractive and old-fashioned. The
participants are told they are joining a design team whose
task, over a day of individual work and group meetings, is
to develop a prototype of the new remote control. We
chose design teams for this study for several reasons.
First, they have functional meetings with clear goals,
making it easier to measure effectiveness and efficiency.
Second, design is highly relevant for society, since it is a
common task in many industrial companies and has clear
economic value. Finally, for all teams, meetings are not
isolated events but just one part of the overall work cycle,
but in design teams, the participants rely more heavily on
information from previous meetings than in other types of
teams, and so they produce richer possibilities for the
browsing technology we are developing.
Within this context, each participant in the elicitation is given a different role to play. The project manager (PM) coordinates the project and has overall responsibility. His job is to guarantee that the project is carried out within time and budget limits. He runs the meetings, produces and distributes minutes, and produces a report at the end of the trial. The marketing expert (ME) is responsible for determining user requirements, watching market trends, and evaluating the prototype. The user interface designer (UI) is responsible for the technical functions the remote control provides and the user interface. Finally, the industrial designer (ID) is responsible for designing how the remote control works including the componentry. The user interface designer and industrial designer jointly have responsibility for the look-and-feel of the design. For this elicitation, we use participants who are neither professionally trained for design work nor experienced in their role. It is well-known that expert designers behave differently from novices. However, using professional designers for our collection would present both economic and logistical difficulties. Moreover, since participants will be affected by their past experience, all those playing the same role should have the same starting point if we are to produce replicable behaviour. To enable the participants to carry out their work while lacking knowledge and experience, they are given training for their roles at the beginning of the task, and are each assigned a (simulated) personal coach who gives sufficient hints by e-mail on how to do their job. Our past experience with elicitations for similar non-trivial team tasks, such as for crisis management teams, suggests that this approach will yield results that generalise well to real groups. We intend to validate the approach for this data collection both by the comparisons to other data already described and by having parts of the data assessed by design professionals.
The structure of the elicited data distinguishes four phases in the design process:
We use these phases to structure our elicitation, with one meeting per design phase. In real groups, meetings occur in a cycle where each meeting is typically followed by production and distribution of minutes, the execution of actions that have been agreed on, and preparation of the next meeting. Our groups are the same, except that for practical reasons, each design project was carried out in one day rather than over the usual more extended period, and we included questionnaires that will allow us to measure process and outcomes throughout the day.
Communication with participants is controlled by the scenario controller which emails participants at appropriate times through the day, providing information and resources to help with the task. This PDF file (8MB) contains all the emails sent to participants by the scenario controller and this tarred gzip file (3MB) contains information on how when and what is sent to participants (documents in their original formats).
Familiarity and Roles
Although TNO and Edinburgh both tried to organize groups in which the subjects didn't know each other, we know that in practice this is violated, and we didn't always manage to gather information from them about who knew whom - so you can't assume anything about familiarity in these groups. For IDIAP-based groups, I suspect most people know each other at least a bit.
In our pilots, we tried randomly assigning roles, but subjects were unhappy with roles that didn't fit them, and the teams performed poorly. So instead, at least at TNO and Edinburgh, we asked who wanted to do what. Most of the time this was easy for the groups to agree.
Participants and Roles
Within this context, each participant in the elicitation is given a different role to play. The project manager (PM) coordinates the project and has overall responsibility. His job is to guarantee that the project is carried out within time and budget limits. He runs the meetings, produces and distributes minutes, and produces a report at the end of the trial. The marketing expert (ME) is responsible for determining user requirements, watching market trends, and evaluating the prototype. The user interface designer (UI) is responsible for the technical functions the remote control provides and the user interface. Finally, the industrial designer (ID) is responsible for designing how the remote control works including the componentry. The user interface designer and industrial designer jointly have responsibility for the look-and-feel of the design. For this elicitation, we use participants who are neither professionally trained for design work nor experienced in their role. It is well-known that expert designers behave differently from novices. However, using professional designers for our collection would present both economic and logistical difficulties. Moreover, since participants will be affected by their past experience, all those playing the same role should have the same starting point if we are to produce replicable behaviour. To enable the participants to carry out their work while lacking knowledge and experience, they are given training for their roles at the beginning of the task, and are each assigned a (simulated) personal coach who gives sufficient hints by e-mail on how to do their job. Our past experience with elicitations for similar non-trivial team tasks, such as for crisis management teams, suggests that this approach will yield results that generalise well to real groups. We intend to validate the approach for this data collection both by the comparisons to other data already described and by having parts of the data assessed by design professionals.
The structure of the elicited data distinguishes four phases in the design process:
- Project kick-off, consisting of building a project team and getting acquainted with each other and the task.
- Functional design, in which the team sets the user requirements, the technical functionality, and the working design.
- Conceptual design, in which the team determines the conceptual specification for the components, properties, and materials to be used in the apparatus, as well as the user interface.
- Detailed design, which finalizes the look-and-feel and user interface, and in which the result is evaluated
We use these phases to structure our elicitation, with one meeting per design phase. In real groups, meetings occur in a cycle where each meeting is typically followed by production and distribution of minutes, the execution of actions that have been agreed on, and preparation of the next meeting. Our groups are the same, except that for practical reasons, each design project was carried out in one day rather than over the usual more extended period, and we included questionnaires that will allow us to measure process and outcomes throughout the day.
Communication with Participants
Communication with participants is controlled by the scenario controller which emails participants at appropriate times through the day, providing information and resources to help with the task. This PDF file (8MB) contains all the emails sent to participants by the scenario controller and this tarred gzip file (3MB) contains information on how when and what is sent to participants (documents in their original formats).
Familiarity and Roles
Although TNO and Edinburgh both tried to organize groups in which the subjects didn't know each other, we know that in practice this is violated, and we didn't always manage to gather information from them about who knew whom - so you can't assume anything about familiarity in these groups. For IDIAP-based groups, I suspect most people know each other at least a bit.
In our pilots, we tried randomly assigning roles, but subjects were unhappy with roles that didn't fit them, and the teams performed poorly. So instead, at least at TNO and Edinburgh, we asked who wanted to do what. Most of the time this was easy for the groups to agree.