New Publication with @neugebauer.bsky.social in Sociological Science! Factorial surveys are widely used to predict real-world decisions, but are they valid? Our results raise concerns when it comes to predicting real-world decisions from factorial survey responses (1)
This is an extremely relevant and interesting study that shows that behavioral intentions measured in survey experiments should not be mistaken for a measure of actual behavior. And it is a nice example of studies that replicate themselves by using different research designs!
We varied the topic sensitivity and tested whether behavioral predictions were more successful after filtering out respondents who gave socially desirable answers or did not exert sufficient effort when answering FS vignettes. (3)
We took hiring decisions as a use case. In a field experiment, we sent out ~3000 applications to job vacancies in Germany and measured employer responses. Eight weeks later, we presented nearly identical applicant profiles to the same employers as a part of a factorial survey (2)
Across conditions, the FS results did not correspond well with the real-world benchmark. We conclude that researchers must exercise caution when using factorial surveys to study (hiring) behavior (4)