PhD student at Columbia University working on human-AI collaboration, AI creativity and explainability. prev. intern @GoogleDeepMind, @AmazonScience
asaakyan.github.io
Arkadiy Saakyan
Loading...
See paper for more analyses and practical takeaways!
Thanks to my amazing hosts Charvi Rastogi and Lora Aroyo, and Aida Davani, Alicia Parrish, @vinodkpg.bsky.social, Ding Wang, Shachi Dave, @sydneylevine.bsky.social, @williamis.bsky.social, Dee Cattle and others for crucial feedback!
We experiment using LLMs as surrogates and triage tools for pluralistic annotation. We find that they struggle to emulate safety ratings of all cultural quadrants, but show potential in prioritizing culturally sensitive items for human annotation.
We propose a method to empirically identify and quantify culturally sensitive items. Our analysis reveals that roughly 10% of the items in datasets we examined would be misclassified as safe without adequate cultural representation.
Via multilevel modeling, we show that both culture and demographics are predictive of safety ratings.
Crucially, cultural zone membership explains variance in safety ratings beyond standard demographics (p < 0.05 across 6 datasets).
Out of 1000+ search results, only 8 studies report both demographic and geo-cultural rater information, revealing a systemic lack of geo-cultural reporting in safety research.
Datasets varied by geographic coverage and methods for analyzing cultural disagreement.
Excited to share #ICML2026 paper from my internship @ Google DeepMind!
AI models are deployed globally, but AI safety datasets are largely geographically homogenous. What is the impact of culture on AI safety ratings? Is there any impact beyond standard demographics like age, gender, and ethnicity?
We made traversle.io, a new daily word game! The goal is to traverse from a start word to a target word through a network of related words. (Our motivating question: is it possible to construct a network that allows human navigation?)
Excited to see MIGRATE recognized in the IPUMS awards! Huge thanks to @emmapierson.bsky.social, @nkgarg.bsky.social, and our coauthors. Our work primarily aims to make spatiotemporal data more trustworthy and accessible to researchers, just like IPUMS. Read the paper to request data access!
We are co-hosting the EAAMO colloquium next Monday (12pm EST) with Professor Rachel Franklin. Come hear her talk about spatial inequality and the smart city and feel free to share with colleagues!
Register below to get the Zoom link:
www.eaamo.org/colloquium/r...
