Odds and insights: decision quality in exploratory data analysis under uncertainty

@InProceedings{Sarma2024OddsInsightsDecision,
  author    = {Sarma, Abhraneel and Pu, Xiaoying and Cui, Yuan and Brown, Eli T. and Correll, Michael and Kay, Matthew},
  booktitle = {Proc.\ CHI Conference on Human Factors in Computing Systems},
  title     = {Odds and insights: decision quality in exploratory data analysis under uncertainty},
  year      = {2024},
  note      = {CHI 2024 Best Paper Honorable Mention! Preprint at \url{https://osf.io/qa8dj/}. Supplemental material at \url{https://osf.io/xtsfz/}.},
  series    = {CHI},
  abstract  = {Recent studies have shown that users of visual analytics tools can have difficulty distinguishing robust findings in the data from statistical noise, but the true extent of this problem is likely dependent on both the incentive structure motivating their decisions, and the ways that uncertainty and variability are (or are not) represented in visualisations. In this work, we perform a crowd-sourced study measuring decision-making quality in visual analytics, testing both an explicit structure of incentives designed to reward cautious decision-making as well as a variety of designs for communicating uncertainty. We find that, while participants are unable to perfectly control for false discoveries as well as idealised statistical models such as the Benjamini-Hochberg, certain forms of uncertainty visualisations can improve the quality of participants' decisions and lead to fewer false discoveries than not correcting for multiple comparisons. We conclude with a call for researchers to further explore visual analytics decision quality under different decision-making contexts, and for designers to directly present uncertainty and reliability information to users of visual analytics tools.},
}