Our paper "Towards Reliable Latent Knowledge Estimation in LLMs: Zero-Prompt Many-Shot Based Factual Knowledge Extraction" has been accepted at WSDM 2025!
The 18th ACM International Conference on Web Search and Data Mining will take place March 10-14, 2025 in Hannover, Germany.
The Challenge
Reliably estimating factual knowledge embedded inside large language models is a challenging task. Prior approaches using prompt engineering have significant reliability concerns, as results can vary dramatically based on prompt phrasing.
Our Approach: ZP-LKE
We propose to eliminate prompt engineering entirely when probing LLMs for factual knowledge. Our approach, called Zero-Prompt Latent Knowledge Estimator (ZP-LKE), leverages the in-context learning ability of LLMs to communicate both the factual knowledge question and the expected answer format.
Key Advantage
By removing prompt sensitivity from the equation, ZP-LKE provides more reliable and reproducible estimates of what knowledge LLMs actually possess.
This is joint work in collaboration with Boston University: Qinyuan Wu (MPI-SWS), Mohammad Aflah Khan (MPI-SWS), Soumi Das (MPI-SWS), Vedant Nanda (UMD), Bishwamittra Ghosh (MPI-SWS), Camila Kolling (MPI-SWS), Till Speicher (MPI-SWS), Krishna Gummadi (MPI-SWS), and Evimaria Terzi (Boston University).