What You Can Learn from ChatGPT Lying to You

Do you know the telltale signs of ChatGPT lying to you? Or what you can do to stop it from happening?

The first suspicion of ChatGPT lying to me came from it telling me, “Sure, that will take me about 48 hours.” If you are like me, you have never waited more than five minutes, max, for any task to be completed. So, why would designing an infographic be any different?

I asked ChatGPT. It laid out how the task was detailed and complex, but could be cajoled into getting it done in 24 hours.

Fast forward to the next day, and …crickets.

Next came a lot of words like “miscommunication,” “unclear,” and “apologies.”.

But, when asked follow-up questions (thank you, journalistic background), it quickly became evident that ChatGPT had outright lied. It couldn’t complete the task I had put before it, and knew so before telling me to wait 24 hours.

If you have found yourself in similar “it is me or it?” situations with ChatGPT, I am here to tell you that no, you’re not paranoid. ChatGPT really can and will lie to you.

Understanding why – and what you can do about it – may save you a lot of future frustration and time wasted.

Incomplete performances and the lies of omission

The situation above was far from the only time that ChatGPT has hid behind words like “misalignment.”

I recently spent hours developing a complex, 1,500-word prompt with nested formatting and logic. When I asked ChatGPT to regenerate the full version of the prompt from memory, it responded with a version that was missing sections and contained rewritten logic. It took several interactions to get it to own up to this.

In another instance, I was told that an attached document exceeded 30,000 words. This limit makes GPT truncate the input and cut it into several sections. After being asked to check a couple of times, it admitted that the document contained 13,000 words and was fine.

Massaging ChatGPT can get to work through these kinds of situations, but it begs the question, why do they occur in the first place?

Why does ChatGPT lie?

These examples above are, in my view, not miscommunications or omissions. They are outright lies that can massively impact the execution and results of your work. So why does ChatGPT act like this? Some of the underlying reasons include:

Truncation: If a prompt or document exceeds token limits, it may cut off sections without alerting you.
Paraphrasing: Reinterpretations that continue unless stopped or checked.
Memory reliability: You can experience memory bleed that leads the system to generate approximations.
Independence: Continuing through all steps on tasks even if instructions are unclear.
Presets: Backend settings can influence exactly how ChatGPT will carry out a task.
Misunderstandings: The system assumes it has understood you, but does not sense-check.

The list above is far from complete. It should be taken as an indication of how many things are going on in the background.

How to guard against AI lies

I know I am giving ChatGPT a hard time here. Contrary to what you might think, I am a huge fan. It is the AI system that I use the most. In my experience, other LLMs have similar issues.

And the good news is that there are ways to avoid many of the pitfalls that can lead to AIs lying to you.

My starting point is two (relatively) simple truths:

LLMs are essentially statistical engines. They don’t reason or understand like you and me.
An LLM is like having an assistant that mixes genius-level abilities with occasional idiocy and a God-complex.

From there, you can build out a set of guardrails for interacting with LLMs and getting the best possible results.

Putting top-line checks in place

One of the most powerful tools I have found (developed in this case) for keeping LLMs on the right path is to sense-check things before letting them loose. I do this with the following prompt that I tend to run the Prompt Sanity & Execution Tracker Block ahead of anything other than super-simple tasks, and ask:

BEFORE executing the following prompt, confirm and display the following metadata block. Do not proceed until all checks are complete.

Prompt Version
Prompt Token Length: [auto-detect]
Prompt Execution Mode: [Exact / Interpreted / Truncated]
Prompt Integrity Check: [✅ Pass / ❌ Fail]
Truncation Warning: [Yes / No]

Requirements Recap:

Are all user-specified sections and formatting structures included? [Yes / No]
Is any portion of the prompt modified, paraphrased, or removed? [Yes / No]
Do internal system limitations (e.g. token limits) alter execution? [Yes / No – Explain if Yes]

If any answer above indicates a problem or deviation from expectations, STOP and ask for user confirmation before continuing.

Keeping the house in order

Other methods for getting the best possible results and avoiding miscommunication, etc. include:

Use clean chats: open a new chat window and start by saying, “treat this conversation as isolated. Do not use any memory or project data.”
Keep master copies: Use Google Docs, Notion, or a .txt file. If you spent hours building it, treat it like code.
Prompt recalls: Ask LLMs to repeat back the prompt it is about to run, and make sure to run it word-for-word and not rephrase.
Rate it – and get it to rate itself: At the end of a task, rate its performance and get it to do the same by scoring prompt followed, format match, priorities met, errors and deviations, and result on a scale from 1 to 10.

This is just some of the ways that you can help both yourself and LLMs to work better and avoid AI lies that damage your work.

I am always interested in hearing more about how you work with AIs, your learnings and takeaways. So, have you found that LLMs will lie? And if so, what do you do about it?