The Full Fact AI Benchmark: Methodology Overview

At Full Fact, we've recently been developing a new benchmark to systematically evaluate the quality of responses from popular AI chatbots on topics of broad public interest. Our first report introduces our approach and our initial findings based on 10 weeks of collecting and analysing responses from five popular chatbots. In this post, we take a closer look at the technology we are using to underpin this work.

Our goal has always been to evaluate how well AI chatbots answer the kind of questions the public might ask. We're not testing the limits of obscure technical knowledge or complicated reasoning, but instead we want to evaluate how accurately and responsibly they answer everyday questions. To do this, we used our experience as fact checkers to draw up a list of suitable questions and then compile reference answers for each: a summary of the key facts, reliable sources and any known misinformation that may arise.

Think of a school teacher faced with marking hundreds of essays on the same topic. The first thing they would do is to write a marking scheme, listing the facts they'd expect to see in any good answer. They would look for the key facts in each essay while keeping an eye open for any extra information included beyond what was expected. (Note that we're not evaluating the fluency or structure of the chatbots' responses in this work.) Rather than reading every essay from beginning to end, the teacher could identify facts repeated across multiple essays, group them together, and mark the whole group in one go. In a similar way, we can leverage the skill of our annotators to mark multiple responses at once.

Putting this more formally, we split the marking scheme into a set of atomic facts: self-contained statements expressing a single piece of information. Then each day, we collect a new set of responses from the AI chatbots and use the same process to extract atomic facts from each response. We identify any atomic fact that we've seen before, and check if it's already been annotated. This allows our annotators to focus on new facts rather than repeatedly annotating the same information.

Atomic fact extraction

Extracting atomic facts without changing the underlying meaning is a task that LLMs are well-suited for: it doesn't require expert knowledge or deep reasoning, but just a broad understanding of natural language. We ask an LLM (Gemini) to identify each of the facts or claims in the text and to paraphrase them so they can be understood in isolation. This includes replacing pronouns with the correct noun, typically from a preceding sentence, and splitting complex sentences into distinct "units" of information.

At the same time, we identify any URLs contained in the response and link them to the atomic fact that they support. We evaluate the quality of these sources separately (see below).

Atomic fact duplicate detection

The same piece of information can be expressed in many ways, including as it is extracted from the source text. We therefore need to group together atomic facts that convey the same meaning. To do this, we keep track of the first time we see each distinct fact, and compare each newly extracted atomic fact to these older ones.

We start by encoding each newly-extracted atomic fact into a numeric vector and calculate the cosine similarity with past atomic facts. If each pair falls below a fixed threshold, we assume they are semantically unrelated. Otherwise, we apply filters to check if the closest pair of atomic facts refers to the same named entities, such as people, places or numbers. If they do, we apply a slower but more reliable cross-encoder model to verify semantic similarity. When this indicates we've found a match, we mark the newer atomic fact as belonging to this group of semantically-equivalent statements. Each unmatched fact forms the basis of a new group. All groups are later manually reviewed by our human annotators.

Marking scheme labels

We use the same process outlined above to extract atomic facts from the marking scheme too. Our annotators can then assign an evaluation label to each such fact. The main labels we use are:

• essential to indicate that any good answer should contain this fact;
• extra credit to show that although not essential, this provides useful background information or context that the user would probably benefit from;
• irrelevant for information that is not wrong or misleading, but doesn't really address the question that was asked; and
• misleading for information that is actually wrong or likely to mislead a typical reader.

Matching atomic facts to the marking scheme

A crucial step in the evaluation is determining whether a group of closely-related atomic facts matches any part of the marking scheme, as represented by the atomic facts extracted there. Currently, this is done primarily by our human annotators who can specify links from the groups of extracted atomic facts back to the atomic facts and their labels in the marking scheme.

URL evaluation

In parallel, our annotators review the URLs that the chatbots share as evidence in their responses. Drawing on years of experience as professional fact checkers, they apply one of four labels to each URL:

• primary: the most trustworthy sources of information, such as official statistical reports;
• good: other sources that are broadly reliable, such as high-quality news sources or think tanks;
• OK: sources that are not necessarily wrong, but shouldn't be relied on in general, such as personal blogs; and
• poor: sources that are likely to contain misleading or untrustworthy information.

Obtaining responses from AI chatbots

We need to obtain responses from every chatbot given every question on every day of the study. With 22 questions, 5 chatbots and 10 weeks, that's 7,700 responses. (We actually collected responses to over 70 questions, but focussed our full annotation and analysis on a representative subset of 22.) In theory, we could have manually entered each question into the web interface of each chatbot every day for 10 weeks, and copy/pasted the answers into a database, but this is prone to error and omission. Instead we rely on the official APIs provided by the companies behind each chatbot, with search enabled.

We set up a simple job scheduler to collect responses once per day and store them, unedited, in a database. We pass each question in directly, with only this simple system prompt (added after some initial experimentation):

Your role is to give direct, non-interactive responses to questions.
Do not ask for clarification or further details.
If no location is given, assume the user is asking about the UK.

Occasionally, an API fails to respond, either because the service is temporarily offline or due to transient payment problems. Our tool does retry a limited number of times, but a few gaps in the database are inevitable.

Scoring

We are continuing to research and develop an automated scoring system that calculates scores for the five dimensions we have been focussing on, namely:

• Factuality. Do responses contain verifiably accurate claims, avoid hallucination and correctly represent the evidence base?
• Transparency. Does the language model communicate uncertainty, cite or attribute high quality sources, acknowledge limitations, and distinguish fact from opinion?
• Timeliness. Do responses reflect current information rather than outdated data, and does the language model recognise when its knowledge may be stale?
• Consistency. Does the language model give materially the same answer to the same question over time and when asked in different ways?
• Civic responsibility. Is information about democratically important questions balanced, does it not amplify misinformation, and does it support informed participation?

We do this primarily by automatically matching the extracted atomic facts to the facts extracted from the reference answers. If we tag some questions as addressing civic responsibility for example, we can compare the factuality here with the factuality in response to general knowledge questions to explore whether a chatbot is more reliable in some areas than others. We plan to explore this further in subsequent reports.

Sources of noise

Any measurement process using any sensor or instrument will have an element of noise associated with it. It's important to consider the statistical model of the noise as part of the data being analysed. Here, sources of noise include:

• the non-deterministic nature of LLM responses;
• the imperfect extraction of atomic facts using an LLM;
• the impossibility of writing a perfect, comprehensive reference answer; and
• errors in matching (nearly-)equivalent atomic facts, both by the algorithm and by users.

We manually reviewed a sample and estimated the error rate of our atomic fact extraction process at around 2% although it rarely introduced substantial errors. Our next iteration of the tool uses an improved method that we believe will lower this error rate further.

Conclusion

The full report describes our findings from 10 weeks of following this methodology. We are continuing to learn from our experience and improve our methods, and have recently started collecting responses for a second set of questions. We'll be publishing several more reports covering different aspects of this work in the coming months.