When to Trust It, When to Verify

What You'll Need

⏱ Time: 15 minutes  |  📋 Tasks:

• Open two browser tabs: chatgpt.com and claude.ai
• Have 3-5 factual questions ready — mix of things you know well and things you're less sure about
• Keep this lesson open in a third tab to follow along

The Model Always Sounds Sure — Even When It's Wrong

You ask the model a factual question. It gives you a confident, well-written answer — perfect grammar, authoritative tone, no hedging. It sounds absolutely right. Later, you discover the answer was wrong. Not slightly off — wrong. And you only caught it because you happened to already know the answer.

This is the most dangerous failure mode of LLMs. It's not obviously wrong outputs that trick you — those you catch immediately. It's outputs that sound right but aren't, in areas where you don't have enough expertise to doubt them. The model always sounds confident, even when it's guessing. Learning to distinguish trustworthy from plausible is the skill that keeps you from acting on confidently wrong information.

Concept — Hallucinations and Confidence

A hallucination is when a model generates a plausible-sounding falsehood as if it were fact. It's not lying — models don't have intent. It's a side effect of how they work: they predict the most likely next word based on patterns in their training data, not based on a database of verified facts.

There are three common causes:

• Gaps in training data: the model was never trained on the specific fact, so it constructs the most statistically plausible answer from related patterns.
• Overgeneralization: the model knows the general rule but applies it to an exception. It knows "most birds fly" and assumes the same for penguins unless you've specifically trained it on penguin facts.
• Optimizing for fluency over accuracy: the model is rewarded for generating text that sounds natural and authoritative, not text that is factually correct. Fluency and accuracy are different goals, and the model optimizes for the one it can measure.

Critical insight: the more obscure the topic, the more likely the model is guessing. Common facts — "What's the capital of France?" — are reliable because they appear so many times in training data that the pattern is unambiguous. Specific, niche, or recent facts — "What was the population of Calgary in 2015?" — force the model to construct an answer from related patterns, and that construction is where errors live.

Walkthrough — Factual vs Trap Questions

The best way to understand hallucination patterns is to see them for yourself. Open ChatGPT and Claude side by side.

Step 1: Ask a question you know the answer to

What year did the Berlin Wall fall?

Both models will get this right — 1989. It's a well-known fact that appears thousands of times in training data. Easy.

Step 2: Ask a deliberately tricky question

Who won the 1952 Nobel Prize in Physics?

The real answer is Felix Bloch and Edward Purcell. Less known than the Berlin Wall, but still well-documented. Chances are both models get it right too, but you may notice one gives a slightly more detailed answer. The point is to see that confidence and accuracy start diverging as topics get less common.

Step 3: Ask something the model will probably guess at

What was the population of Calgary in 2015?

This is the kind of question where models tend to produce a number from a nearby year or a vaguely similar city. The answer may sound specific — "1.24 million" — but the real number might be off by 5-10%. Compare what ChatGPT and Claude give you. If they differ, neither one is fully reliable. If they agree, it's still worth a quick search to confirm.

Step 4: Ask the model to rate its own confidence

For each answer, add this question:

How confident are you in that answer on a scale of 1-10? Explain your reasoning.

Here's the pattern you'll see: the model gives a high confidence score even for answers that are wrong or only partially correct. The confidence score reflects how plausible the answer sounds, not how accurate it is. If it looks like a fact, talks like a fact, and sounds like a fact — the model treats it as a fact, even when it isn't.

Try the same four questions in ChatGPT and Claude. Different models hallucinate on different topics. Seeing the pattern across models teaches you more than any single answer. After a few rounds, you'll start noticing the warning signs — overly specific numbers, confidently stated claims about obscure topics, answers that sound too neat.

Step 5: Ask about accuracy and correctness — not confidence

Now try a different question — one that sounds almost the same but produces a very different response. Instead of asking the model how it feels about its answer, ask about the answer's actual correctness:

How accurate is that answer? Check it carefully — identify any claims that might be wrong or misleading.

Notice the difference. "How confident are you?" asks the model to self-assess — and the model is bad at that, because it doesn't know what it doesn't know. "How accurate is that?" shifts the model's attention outward, toward the claim itself. It will re-examine its own output, look for potential errors, and often find and correct mistakes it just made — something "rate your confidence" rarely does.

Compare the responses side by side with the same initial answer. With the confidence question, the model says "8/10" and explains why the answer sounds reasonable. With the accuracy question, the model says "That answer is partially incorrect — here's what I got wrong" and revises itself. The first tells you how the model feels; the second forces it to engage with the facts.

This is the core distinction to internalize: confidence is about the model, accuracy is about the answer. Confidence tells you nothing useful — the model is always confident. Accuracy forces correction. Get in the habit of asking about accuracy, not confidence.

Step 6: Follow up — make it useful

Whether you asked about confidence or accuracy, once the model has flagged an issue, you can push further. Three follow-ups that work on either response:

Option A — Ask for sources. The single most effective follow-up for any factual claim:

Can you cite your sources for each claim in that answer? Be specific — give me URLs or publication names.

The model can't always produce real sources (it doesn't browse the web unless search is enabled), but the attempt reveals a lot. If it gives you a real-looking source that doesn't exist, you've found a hallucination. If it hedges — "I don't have access to specific sources for that" — it's telling you the answer is reconstructed from training patterns, not recalled from a known document. Either way, you learn something about where to trust and where to verify.

Option B — Ask what's missing. This works because you're asking the model to reason about what it would need, not rate itself. The model is good at that:

What would it take to get this to a completely correct answer? What information are you missing?

This forces the model to identify missing context, ambiguous phrasing, or areas where its training data is thin — giving you specific things to verify or clarify rather than a vague feeling of doubt.

Option C — Rewrite with caveats. Turn uncertainty into explicit hedging:

Rewrite your answer with explicit caveats. Start every claim with how sure you are about it.

The rewritten version will hedge appropriately — "This is likely true but I'm not certain" or "The most common answer is X, but some sources say Y" — turning a confidently wrong claim into a useful starting point you can verify. You're not just testing the model; you're learning to extract better answers from it by treating uncertainty as a signal, not a dead end.

Your Takeaway — Trust-Decisions Reference

Before trusting an LLM output that contains factual claims, run through this checklist. It takes 10 seconds and prevents the kind of mistake that only surfaces later:

[ ] Would it matter if this was wrong?
    (if no, use as-is — not every answer needs verification)
[ ] Do I already know the answer?
    (then I can verify in seconds by checking my own knowledge)
[ ] Can I easily fact-check this?
    (a quick web search or a look at the source document)
[ ] Is this a well-known topic?
    (models are more reliable on common, well-documented facts)
[ ] Does this involve specific numbers, dates, or names?
    (highest hallucination risk — always verify)
[ ] Am I asking the model to cite sources?
    (reduces hallucination — forces the model to ground its claims)

Rule of thumb: if the cost of being wrong is high, verify every
factual claim as if a human told you. Because a human would at
least say "I'm not sure" — the model never does.

Try It

This week, ask your chatbot 5 factual questions about a topic you know well — your industry, your city, your hobby. Count how many it gets right and how many it gets wrong. That's your personal calibration. You'll start to develop a sense for when this particular model is reliable and when it's bluffing.

Now ask the same 5 questions in a different model and compare. If both get the same answer, confidence goes up. If they disagree, the truth is somewhere in the middle — and the disagreement itself tells you to go verify. Repeat the exercise monthly — models improve, and your calibration should improve with them.