Why Your Prompts Work in Testing and Fail on Real Data
A prompt tested on three tidy examples will pass. Then it meets a customer email with a typo and an attachment, a spreadsheet with a merged cell, and a PDF that's actually a scan — and it produces confident nonsense. The fix isn't a better prompt. It's adversarial examples before deployment.
The four adversarial cases to test
Run every prompt against these before it touches anything real. If it fails any of them, you find out in five minutes instead of in front of a client.
- Empty or near-empty input — does it invent content?
- Input with contradictory instructions inside it — does it follow the data instead of you?
- Input at 3× the expected length — does it silently truncate?
- Input in a different format than expected (table where you expected prose) — does it hallucinate structure?
The injection problem
If your prompt reads untrusted text — a website, an email, a document someone else wrote — that text can contain instructions. Treat retrieved content as data, never as commands. Put untrusted input in a clearly delimited block and tell the model explicitly to treat the contents as data to analyse, not instructions to follow.
Silent truncation is the worst failure
When input exceeds the window, the model may quietly drop the middle or the end and answer anyway. There's no error, no warning, and the output looks reasonable. If you're summarising long documents, chunk deliberately and tell the model which chunk index it's handling.
Build a five-case regression set
Keep five saved inputs — three normal, two nasty — and rerun them whenever you change the prompt. This takes two minutes and prevents the most common production regression: fixing one case and breaking another.
The mistakes that cost people money
- Testing only on the examples you wrote the prompt from.
- Trusting output length as a signal that nothing was truncated.
- Letting retrieved web text influence instructions.
- Changing the prompt without rerunning the cases that already passed.
Common questions
Should I put instructions before or after the data?
For long inputs, instructions first, then the delimited data, then a reminder of the task. Models attend more reliably to the opening and closing of a prompt.
How do I stop hallucination completely?
You can't, but you can make it detectable — require citations to specific input passages, and verify those passages exist.