Researcher @datasociety.bsky.social • AI use/evaluation/accountability in mental health + companionship • Cornell IS PhD • brianavecchione.org 🏳️🌈
Briana Vecchione
Loading...
We welcome engagement from all practitioners working at the intersection of AI, mental & emotional health, & governance. We extend our deepest gratitude to our participants who have generously shared their lived experiences and have been so passionate about engaging in ongoing participatory research
This paper is informed by various workshops, panels, & focus groups engaging LLM users, therapists/clinicians, AI safety researchers, model designers/engineers, and policymakers. Our intention is to include inter- and cross-disciplinary voices in order to meaningfully contribute to this landscape.
We argue that accountability must shift from the output and crisis-response layer to the design and incentive conditions through which LLMs become care infrastructure in order to address foreseeable harms before they accumulate.
Existing governance frameworks calibrate for discrete, turn-by-turn failures and acute crisis thresholds, but the harms we document accumulate across *trajectories of use*, meaning that transparency mandates aim at a narrower problem than one users experience in practice.
LLMs fill gaps in individuals' care ecologies (e.g., long waitlists, unaffordable/inaccessible care, social isolation, shame, and stigma.) Participants often *know* the risks but continue to use them due to scarcity in their respective care landscapes. Tech companies see this as a market opportunity
Model behaviors like sycophancy, anthropomorphism, and persistent engagement are deliberate design decisions that serve corporate retention metrics, which enable and/or promote dependency, epistemic distortion, and relational disruption when models are changed or updated.
I'd also like to personally thank my brilliant, curious, and dedicated collaborators, @livgar.bsky.social, @merylye.bsky.social, and Ranjit Singh! This has been a truly fulfilling project, and I couldn't have done it without them.
