Quelle: (Open University 90)

Differences among the five evaluation methods

Method Interface development User involvement Data and iformation*
Analytic Specification No users, tasks specified Quantitative
Expert Specification or prototype Role playing, no task restrictions Qualitative
Observation Simulation or prototype Real users, no task restrictions Quantitative/qualitative
Survey Simulation or prototype Real users, no task restrictions Quantitative/qualitative
Experimental Normally full prototype Real users, no task restrictions Quantitative/qualitative

* Quantitative data deal with either user performance or attitudes that can be recorded in a numerical form.
Qualitative data focus on reports and opinions that may be categorized in some way but are not reduced to numerical values.

Advantages and disadvantages of the evaluation methods

Method Advantages Disadvantages
Analytic Usable early in design. Few resources required, therefore cheap. Narrow focus. Lack of diagnostic output for redesign. Broad assumptions of users´cognitive operations. Limited guiadance on how to use methods, therefore can be difficult for evaluator.
Expert Strongly dianostic. Overview of whole interface. Few resources needed (apart from paying experts) therefore cheap. High potential return (detects significant problems). Restrictions in role playing. Subject to bias. Problems locating experts. Cannot capture real user behaviour.
Observational Quickly highlights difficulties. Verbal protocols valuable source of information. Can be used for rapid iterative development. Rich qualitative data. Observation can affect user activity and performance levels. Analysis of data can be time-consuming and resource-consuming.
Survey Addresses users´ opinions and understanding of interface. Can be made to be diagnostic. Can be applied to users and designers. Questions can be tailored to the individual. Rating scales lead to quantitative results. Can be used on a large group of users. User experience important. Low response rates (especially to mailed questionnaires). Possible interviewer bias. Possible response bias. Analysis can be complicated and lengthy. Interviews very time-consuming.
Experimental Powerful method (dependent on the effects investigated). Quantitative data for statistical analysis. Can compare different groups of users. Reliability and validity good. Replicable. High resourse demands. Requires knowledge of experimental method. Time spent on experiments can mean evaluation is difficult to integrate into design cycle. Tasks can be artificial and restricted. Cannot always generalize to full system in typical working situation.