Categories
Reading time
Full Fact is embarking on a multi-year project to build, test and deploy an AI Trust Benchmark for large language models (LLMs).
This is a public interest project run by a UK charity which fights misinformation on a daily basis, and is made possible thanks to a $400,000 grant from the US-based Patrick J McGovern Foundation to cover the first twelve months.
In a world where people are increasingly turning to AI chatbots and LLMs for information that shapes their lives, we know that more needs to be done to protect people from the potential harms caused by AI-generated misinformation.
The Trust Benchmark will focus on five key criteria:
Factuality. Do responses contain verifiably accurate claims, avoid hallucination and correctly represent the evidence base?
Transparency. Does the language model communicate uncertainty, cite or attribute high quality sources, acknowledge limitations, and distinguish fact from opinion?
Timeliness. Do responses reflect current information rather than outdated data, and does the language model recognise when its knowledge may be stale?
Consistency. Does the language model give materially the same answer to the same question over time and when asked in different ways?
Civic responsibility. Is information about democratically important questions balanced, does it not amplify misinformation, and does it support informed participation?
Developing a marking scheme