Unraveling the Mystery:
Evaluating Language Models' Grasp of Presuppositions

Presuppositions are implicit background information conveyed in a sentence that are taken for granted as true through a process called presupposition accommodation. Presuppositions are crucial in understanding meaning in natural language, as they provide context, influence interpretation, and enrich communication. The ability to recognize and accommodate presuppositions is essential for effective and precise communication. In this paper, we evaluate whether language models are able to accommodate presuppositions using a probing task. We explore how the complexity of the model (i.e. BERT vs. GPT-2 models) and stages of training (i.e. random initialization vs. pre-training vs. fine-tuning) affect the model's ability to accommodate presuppositions and how presuppositions are encoded within sentence embeddings. We find that pre-training and fine-tuning can help the language model learn to accommodate presuppositions, and the presupposition accommodation ability is captured within the later layers of the BERT model. Of all types of presuppositions, language models accommodate question presuppositions the best.

hello

I. Introduction
gello Since the launch of ChatGPT in 2022, the development, applications, and limitations of large language models have become prominent topics of discussion among the general public. It has led to a surge of interest in LLMs and AI technology, and its applications have led to advancements across various industries, including digital marketing, e-commerce, healthcare, and education (A. Shaji George, 2023).
At the same time, there has been growing debate about whether language models can genuinely acquire an understanding of language and reason about it, or if they merely mimic human responses (Emily M. Bender, 2020). Recent research has sought to determine how proficient language models are in learning meaning by evaluating their performance in understanding linguistic semantic phenomena, such as entailments or implicatures (Li et al,

2021), which implicitly contain information about the state of the world. In this project, we seek to evaluate language models’ performance on another, more complicated sentence relationship called presuppositions, which has been less explored in the current literature.
Presuppositions are implicit background information conveyed in a sentence that are either assumed to be true prior to the utterance or taken for granted as true through a process called presupposition accommodation. This enables the sentence to be contextually appropriate and understood by listeners. Presuppositions rely on and update a “common ground,” which constitutes shared background knowledge and information among participants in a conversation. This common ground is essential for effective, contextually-relevant communication (Fintel, 2008). For instance, the sentence “John’s wife is here” presupposes that John has a wife, though this fact is not explicitly stated in ...read more.

Unraveling the Mystery: Evaluating Language Models' Grasp of Presuppositions

Unraveling the Mystery:
Evaluating Language Models' Grasp of Presuppositions