Self-supervision of Hallucinations in Large Language Models: LLteaM


  • Sofía Correa Busquets Wird
  • Lucas Maccarini Llorens Wird



large language model, hallucination, Chain-of-Thought prompting, self-supervision, retrieval-augmented generation


Large language models like GPT and Claude have revolutionized the tech industry over the past year. However, as generative artificial intelligence, they are prone to hallucinations. A large language model hallucinates when it generates false or nonsensical text. As these models improve, these hallucinations become less obvious and more dangerous for users. This research explores the phenomenon in the context of automated email response for customer service. First, it proposes a taxonomy of hallucinations in large language models based on their linguistic nature, and second, a multi-agent system that allows for the self-supervision of such hallucinations. This system generates email responses but prevents their delivery if hallucinations are detected, thus reducing the risks of generative AI in productive environments. Experiments with various state-of-the-art language models reveal that the only successful model’s operating costs currently exceed those viable for operational deployment. Moreover, a drastic performance drop after a recent update to GPT-3.5-turbo suggests likely shortcomings in industrial applications driven by retrieval-augmented generation. Overall, the research advocates for a Machine Linguistics to analyze the outputs of large language models, suggesting that such a collaboration between Linguistics and Artificial Intelligence could help mitigate the social risks of hallucination.


Download data is not yet available.


Anthropic. "Claude." 2023. Acceso el 13 de septiembre.

Austin, John Langshaw. 1975. How to Do Things with Words. Cambridge: Harvard University Press.

Bai, Yuntao, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. 2022. "Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback." arXiv preprint.

Bohannon, Molly. 2023. "Lawyer Used ChatGPT in Court-And Cited Fake Cases. A Judge Is Considering Sanctions." Forbes. Última actualización el 8 de junio.

Chase, Harrison. 2022. "LangChain." Acceso el 13 de septiembre.

Chui, Michael, Mena Issler, Roger Roberts y Lareina Yee. 2023. "McKinsey Technology Trends Outlook 2023." McKinsey Insights. Última actualización el 20 de julio.

Google. 2023. "Bard." Última actualización el 13 de julio.

Grice, H. P. 1975. "Logic and Conversation." In Syntax and Semantics, vol. 3: Speech Acts, edited by Peter Cole and Jerry L. Morgan, 41-58. New York: Academic Press.

Ji, Ziwei, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto y Pascale Fung. 2023. "Survey of Hallucination in Natural Language Generation." ACM Computing Surveys 55 (12): 1-38.

Johnson, Jeff, Matthijs Douze y Hervé Jégou. 2017. "Billion-scale Similarity Search with GPUs." arXiv preprint.

Liu, Jerry. 2022. "LlamaIndex." Acceso el 6 de septiembre. doi:10.5281/zenodo.1234

Microsoft. 2023. "Microsoft and OpenAI extend partnership." Acceso el 6 de septiembre.

Maynez, Joshua, Shashi Narayan, Bernd Bohnet y Ryan McDonald. 2020. "On Faithfulness and Factuality in Abstractive Summarization." arXiv preprint.

OpenAI. 2022. "ChatGPT." Acceso el 15 de septiembre.

OpenAI. 2023a. "GPT-4 Technical Report." arXiv preprint.

OpenAI. 2023b. "Chat Completion API." Acceso el 16 de junio.

OpenAI. 2023c. "Chat Completion API." Acceso el 22 de agosto.

de Saussure, Ferdinand. 1916. Cours de Linguistique Générale. Taiwán: Payot.

Searle, John. 1979. Expression and Meaning: Studies in the Theory of Speech Acts. New York: Cambridge University Press.

Sivasubramanian, Swami. 2023. "Announcing New Tools for Building with Generative AI on AWS." AWS Machine Learning Blog. Última actualización el 13 de abril.

Tharp, Carrie y Lee Moore. 2023. "Helping Businesses with Generative AI." Google Cloud Blog. Última actualización el 7 de junio.

Touvron, Hugo, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, et al. 2023. "LLaMA: Open and Efficient Foundation Language Models." arXiv preprint.

Turing, Alan Mathison. 1950. "Computing Machinery and Intelligence." Mind LIX (236): 433-460.

Vaswani, Ashish, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser e Illia Polosukhin. 2017. "Attention Is All You Need." arXiv preprint.

Wei, Jason, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le y Denny Zhou. 2022a. "Chain-of-Thought Prompting Elicits Reasoning in Large Language Models." arXiv preprint.

Wei, Jason, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai y Quoc V. Le. 2022b. "Finetuned Language Models Are Zero-Shot Learners." arXiv preprint.