Subjects, Trials, and Levels: Statistical Power in Conjoint Experiments

Conjoint analysis is an experimental technique that has become quite popular to understand people’s decisions in multi-dimensional decision-making processes. Despite the importance of power analysis for experimental techniques, current literature has largely disregarded statistical power considerations when designing conjoint experiments. The main goal of this article is to provide researchers and practitioners with a practical tool to calculate the statistical power of conjoint experiments. To this end, we first conducted an extensive literature review to understand how conjoint experiments are designed and gauge the plausible effect sizes discovered in the literature. Second, we formulate a data generating model that is sufficiently flexible to accommodate a wide range of conjoint designs and hypothesized effects. Third, we present the results of an extensive series of simulation experiments based on the previously formulated data generation process. Our results show that—even with relatively large sample size and the number of trials—conjoint experiments are not suited to draw inferences for experiments with large numbers of experimental conditions and relatively small effect sizes. Specifically, Type S and Type M errors are especially pronounced for experimental designs with relatively small effective sample sizes (< 3000) or a high number of levels (> 15) that find small but statistically significant effects (< 0.03). The proposed online tool based on the simulation results can be used by researchers to perform power analysis of their designs and hence achieve adequate design for future conjoint experiments.

Find full paper here: https://osf.io/preprints/socarxiv/spkcy/

Shiny app available here: https://mblukac.shinyapps.io/conjoints-power-shiny/