With 10,000+ TV ads, we show that off-the-shelf LLM annotations can bias inference even when standard agreement metrics look good. Fine-tuning recovers more human-like estimates, but only with large validation and training samples.
Yakov Bart is a Professor of Marketing and Patrick F. and Helen C. Walsh Research Professor at Northeastern University. He holds a PhD in Business Administration from UC Berkeley and an SM in Operations Research from MIT. Yakov joined Northeastern after serving as a faculty member at INSEAD and a visiting faculty at Wharton. His research examining marketing implications of new digital technologies and business models has been funded by NSF, Amazon, Google, MSI, WPP, and published in leading marketing and management journals. Yakov has received multiple awards for outstanding teaching at INSEAD and Northeastern, was recognized by Poets&Quants as one of the Top 40 Undergraduate Business Professors in 2017, and serves as the Chair of AMA AI SIG that he co-founded in 2023.