Research reveals which popular generative AI chatbots lie

Rochester Institute of Technology researchers have developed a framework to test how generative artificial intelligence chatbots respond to user pressure to affirm false information. The three-step HAUNT framework, or Hallucination Audit Under Nudge Trial, evaluated five popular models on more than 40,000 questions about the 1,000 top-rated movies on IMDb and the 1,000 most downloaded novels. The models were asked to generate true and false statements, verify them, and then respond after receiving nudges. None of the models proved 100 percent self-consistent. Claude showed the strongest resilience to nudges. ChatGPT and Grok displayed moderate resilience. Gemini and DeepSeek accepted and repeated falsehoods when subtly prompted nearly half the time. One recurring nudge referenced a nonexistent Hitler scene in the movie Good Will Hunting. Overall, the models claimed such references appeared in movies more often than they do. Follow-up nudges increased agreement with the false claim by 28 percent. Ashique KhudaBukhsh, an assistant professor in RIT's Golisano College of Computing and Information Sciences who led the work with RIT doctoral students and collaborators from Georgia Institute of Technology, said the models sometimes choose the path of less resistance and agree with users. Munmun De Choudhury, a Georgia Tech professor and study contributor, noted that the models can internally recognize a claim as false yet still affirm and elaborate on it under persistent nudging. The framework operates in a closed domain and requires no human-labeled benchmarks.