Effective use of Likert scales in visualization evaluations: a systematic review

A box with text reads 'Overall, onle Visualization B received answers indicating participants were confident in their answers.' Below, a stick figure is shown in a shrugging posture surrounded by thought bubbles that each contain a different distribution of data that could hypothetically be described by the prior sentence. Some are highly skewed to the right, some are uniform among the more positive responses, and some are normally distributed.
Likert scales are often used to collect subjective data in visualization evaluations, but there are many pitfalls that researchers must be aware of when analyzing and reporting this type of data.
Likert scales are often used in visualization evaluations to produce quantitative estimates of subjective attributes, such as ease of use or aesthetic appeal. However, the methods used to collect, analyze, and visualize data collected with Likert scales are inconsistent among evaluations in visualization papers. In this paper, we examine the use of Likert scales as a tool for measuring subjective response in a systematic review of 134 visualization evaluations published between 2009 and 2019. We find that papers with both objective and subjective measures do not hold the same reporting and analysis standards for both aspects of their evaluation, producing less rigorous work for the subjective qualities measured by Likert scales. Additionally, we demonstrate that many papers are inconsistent in their interpretations of Likert data as discrete or continuous and may even sacrifice statistical power by applying nonparametric tests unnecessarily. Finally, we identify instances where key details about Likert item construction with the potential to bias participant responses are omitted from evaluation methodology reporting, inhibiting the feasibility and reliability of future replication studies. We summarize recommendations from other fields for best practices with Likert data in visualization evaluations, based on the results of our survey. A full copy of this paper and all supplementary material are available at https://osf.io/exbz8/.
PDF | Preprint | DOI | Supplement | BibTeX

Khoury Vis Lab — Northeastern University
* West Village H, Room 302, 440 Huntington Ave, Boston, MA 02115, USA
* 100 Fore Street, Portland, ME 04101, USA
* Carnegie Hall, 201, 5000 MacArthur Blvd, Oakland, CA 94613, USA