Qualifying LLM accuracy

For data extracted by large language models (LLMs), Sensible asks the LLMs to report any uncertainties about the accuracy of the extracted data. For example, an LLM can report "multiple possible answers" or "ambiguous query". These confidence signals offer more nuanced troubleshooting than confidence scores.

Note that LLMs can inaccurately report confidence signals. For more information about confidence signals, see the research paper Teaching models to express their uncertainties in words.

Sensible support confidence signals for the Query Group method. The confidence signals checkbox is enabled by default in the Sensible Instruct editor for new Query Group fields. To enable confidence signals for a field in SenseML, use the Query Group method's Confidence Signals parameter.

For more information about troubleshooting confidence signals, see the following table.

Query Group method confidence signals

For the Query Group method, Sensible returns the following messages, or "confidence signals", to qualify the LLM's confidence in the accuracy of the extracted data.

confidence signalJSON outputdescriptiontroubleshooting
Multiple possible answersmultiple_possible_answersThe LLM answers the prompt, but identifies multiple possible answers in the context that Sensible provides to the LLM.- To return multiple answers, use the List method.
- To return a single answer, ensure the context contains a single answer. For more information, see Advanced prompt configuration.
Answer might not fully answer the promptanswer_may_be_incompleteThe LLM answers the prompt, but is uncertain whether the context that Sensible provides to the LLM contains the full answer.- Simplify your prompt, for example, break it up into multiple prompts.
- See Advanced prompt configuration.
Answer not found in the contextanswer_not_foundThe LLM fails to answer the prompt, and can't find an answer to your prompt in the context that Sensible provides to the LLM.See Advanced prompt configuration.
Ambiguous queryambiguous_queryThe LLM either answers or fails to answer your prompt, and identifies ambiguities in your prompt.- Rephrase your prompt using the tips for each Instruct method. For example, see Query Group tips.
- See Advanced prompt configuration.
Incorrect answerincorrect_answerThe LLM judges the answer is incorrect given the context that Sensible provides to the LLM.See Advanced prompt configuration.
Confident answerconfident_answerThe LLM is confident about its answer to the prompt.
Unknown signalunknown_signalThe LLM returns a confidence signal other than the confidence signals Sensible instructs it to return.