Odds and insights: decision quality in exploratory data analysis under uncertainty

Visualization of three confidences intervals with a shared y-axis labeled "profit", with buttons to predict which will be really profitable given the sample information.
Comparing multiple purported effects using confidence intervals. As the number of comparisons increases, the rate of false discoveries might go up.
Abstract
Recent studies have shown that users of visual analytics tools can have difficulty distinguishing robust findings in the data from statistical noise, but the true extent of this problem is likely dependent on both the incentive structure motivating their decisions, and the ways that uncertainty and variability are (or are not) represented in visualisations. In this work, we perform a crowd-sourced study measuring decision-making quality in visual analytics, testing both an explicit structure of incentives designed to reward cautious decision-making as well as a variety of designs for communicating uncertainty. We find that, while participants are unable to perfectly control for false discoveries as well as idealised statistical models such as the Benjamini-Hochberg, certain forms of uncertainty visualisations can improve the quality of participants' decisions and lead to fewer false discoveries than not correcting for multiple comparisons. We conclude with a call for researchers to further explore visual analytics decision quality under different decision-making contexts, and for designers to directly present uncertainty and reliability information to users of visual analytics tools.
Materials
PDF | Preprint | Supplement | Blog post | BibTeX | CHI 2024 Best Paper Honorable Mention!
Authors
Abhraneel Sarma
Xiaoying Pu
Yuan Cui
Eli T. Brown
Matthew Kay
Citation
Thumbnail image for publication titled: Odds and insights: decision quality in exploratory data analysis under uncertainty
Odds and insights: decision quality in exploratory data analysis under uncertainty

Abhraneel Sarma, Xiaoying Pu, Yuan Cui, Eli T. Brown, Michael Correll, and Matthew Kay. Proc. CHI Conference on Human Factors in Computing Systems—CHI. 2024.

PDF | Preprint | Supplement | Blog post | BibTeX | CHI 2024 Best Paper Honorable Mention!


Khoury Vis Lab — Northeastern University
* West Village H, Room 302, 440 Huntington Ave, Boston, MA 02115, USA
* 100 Fore Street, Portland, ME 04101, USA
* Carnegie Hall, 201, 5000 MacArthur Blvd, Oakland, CA 94613, USA