An LLM predicts plausible continuations, not verified truth
A language model is answering a different question from the one you think you asked.
You ask what is true. It produces what would plausibly come next, given everything it has seen. Most of the time those coincide, which is what makes it useful and also what makes it misleading, because nothing in the output distinguishes the times they do not. There is no internal flag for uncertainty, no moment where it knows it is guessing. Fluent and wrong looks exactly like fluent and right, and that is a property of the machine rather than a bug in it.
More on AI and LLMs
- Context is temporary working material, not permanent knowledgeThe board gets wiped
- Retrieval adds documents, not guaranteed correctnessThe filter slot is empty
- Temperature changes variation, not factualityThe dial only sets the spread
- System prompts are instructions, not a security boundaryA sign, with no fence
- LLM output is untrusted input downstreamIt comes in round the back
- Fine-tuning changes behaviour, not every limitationNew type, same roller