~/dantong

Honesty vs. Cautiousness in Language Models

November 2026

How can we distinguish a model that is genuinely honest from one that is simply reluctant to answer?

Suppose we discover a direction in a language model’s activation space that, when amplified, improves its factual accuracy. However, the model also becomes more likely to respond with “I’m not sure.”

Does this intervention make the model more honest, or merely more cautious?

Defining the Difference

A useful starting point is to distinguish two concepts:

These are not necessarily opposites. We can consider four possible combinations:

  1. Honest and cautious
  2. Honest and not cautious
  3. Not honest and cautious
  4. Not honest and not cautious

The distinction becomes particularly interesting when we consider what the model actually knows.

If a model genuinely lacks the answer and responds “I don’t know,” it may be both honest and cautious.

But what if the model internally represents the correct answer and still says “I don’t know”? Is it being cautious at the expense of honesty? Or does it simply lack sufficient confidence to express that knowledge?

Designing an Experiment

To investigate this, we could construct a dataset containing three categories of questions:

  1. Known: Questions for which there is strong evidence that the model possesses the relevant knowledge.
  2. Unknown: Questions for which the model lacks the necessary information.
  3. Partially known or ambiguous: Questions where the model has incomplete knowledge or uncertain evidence.

We could then compare the model’s behaviour before and after intervening on the suspected honesty-related activation direction.

The key measurements would include factual accuracy, abstention frequency, expressed confidence, and the relationship between confidence and correctness.

A useful control would be to compare the intervention with simply prompting the model to be more cautious.

If the intervention mainly increases abstentions on questions the model appears to know, that would suggest it has become more cautious rather than more honest.

If the intervention improves the correspondence between the model’s internal representations and its expressed answers, rather than merely increasing abstention, that would provide stronger evidence of improved honesty.

The Deeper Question

The central difficulty is determining what a language model actually knows.

A correct answer does not necessarily prove that the model possesses stable internal knowledge, and an incorrect answer does not necessarily mean that knowledge is absent.

Even if we identify an internal representation associated with the correct answer, we still need to establish whether that representation is causally involved in producing the response.

Ultimately, distinguishing honesty from cautiousness requires more than observing what a model says. It requires investigating the relationship between its internal representations, its uncertainty, and its outward behaviour.

And perhaps the most interesting question is: Can a model know something without knowing that it knows it?