Multiple Forecast Visualizations (MFVs): trade-offs in trust and performance in multiple COVID-19 forecast visualizations

Stimuli from experiment 1 and 2 shown stacked vertically. In the top 2 rows, Experiment 1's stimuli is made up of individual line charts showing incident deaths from COVID. These charts truncate at the far right and are followed by a grid of possible prediction lines. From right to left, a mean line roughly horizontal around halfway up the graph with a 95% confidence interval around it, just the same mean line without a confidence interval, the minimum and the maximum predictions together on a graoh (forming a funnel shape that spans almost the whole vertical space of the graph), 3 predictions filling in the same funnel space as the previous panel, 4 predictions filling in the same space, and so on until 15 predictions fill the space. These predictions are in multiple colors, all chosen to be as a distinguishable from each other as possible. In the row below these line charts, are the same exact charts in grayscale. In the next row, the stimuli from Experiment 2 are shown. These are the same as the previous experiment, but all in color and with a tighter funnel of predictions which fill in the space that would be occupied by a 95% confidence interval. The row below, shows the same stimuli with a worst case scenario added on to each prediction. The row below, is the final row and shows the same stimuli but with a best case scenario added instead of the worst case.
Stimuli used in Experiments 1 and 2 showing COVID-19 mortality forecasts for November 13, 2021 in the US. Each line depicts a different group's forecast, and the experiments examined the impact of the number of forecasts shown on trust and predictions of the COVID-19 trends. Each participant was shown the 16 stimuli in one row of this figure in a randomized order.
The prevalence of inadequate SARS-COV-2 (COVID-19) responses may indicate a lack of trust in forecasts and risk communication. However, no work has empirically tested how multiple forecast visualization choices impact trust and task-based performance. The three studies presented in this paper (N=1299) examine how visualization choices impact trust in COVID-19 mortality forecasts and how they influence performance in a trend prediction task. These studies focus on line charts populated with real-time COVID-19 data that varied the number and color encoding of the forecasts and the presence of best/worst-case forecasts. The studies reveal that trust in COVID-19 forecast visualizations initially increases with the number of forecasts and then plateaus after 6–9 forecasts. However, participants were most trusting of visualizations that showed less visual information, including a 95% confidence interval, single forecast, and grayscale encoded forecasts. Participants maintained high trust in intervals labeled with 50% and 25% and did not proportionally scale their trust to the indicated interval size. Despite the high trust, the 95% CI condition was the most likely to evoke predictions that did not correspond with the actual COVID-19 trend. Qualitative analysis of participants' strategies confirmed that many participants trusted both the simplistic visualizations and those with numerous forecasts. This work provides practical guides for how COVID-19 forecast visualizations influence trust, including recommendations for identifying the range where forecasts balance trade-offs between trust and task-based performance.
PDF | Preprint | DOI | Preregistration | Video preview | Demo video | News | BibTeX | VIS 2022 Best Paper Award!

Khoury Vis Lab — Northeastern University
* West Village H, Room 302, 440 Huntington Ave, Boston, MA 02115, USA
* 100 Fore Street, Portland, ME 04101, USA
* Carnegie Hall, 201, 5000 MacArthur Blvd, Oakland, CA 94613, USA