Understanding how LLMs think

Real-World Cases of AI Mistakes and What Happened Under the Hood

When Medical Advice Becomes Medical Harm

A patient consulted ChatGPT for detailed health and dietary advice, leading to serious health complications reported in the Annals of Internal Medicine in August 2025 [1]. The patient developed multiple micronutrient deficiencies after following an extremely restrictive diet recommended by the AI, and most critically, suffered bromide poisoning from replacing table salt with sodium bromide.

Technical Explanation:

LLMs lack the personal context and medical training data necessary for making informed individual health decisions. When asked about replacing chlorine in sodium chloride without explaining the reasons and motivations behind this intent, ChatGPT suggests sodium bromide based on the statistically most likely setting: cleaning. These systems have been developed for generic use cases; they are not trained to act as a medical professional. The system pattern-matched from chemistry information without understanding the need to imply medical complications for human consumption.

The Human Mistake:

The patient treated the AI as a qualified medical professional rather than a general information tool, assuming that the AI's scientific-sounding response was equivalent to professional medical advice. This mirrors a broader pattern we've seen with digital literacy: we had to learn not to trust every online source without verification. Just like we fact-check Wikipedia entries or question dubious news articles shared by friends on social media, we now need to apply the same critical thinking to AI-generated content. We need to recognise that authoritative-sounding responses don't automatically equal reliable, and certainly not personalised advice.

The Technical Solution:

Apart from the obvious solution of never replacing professional consultation for health-related decisions, let's examine this from a technical perspective. Without context,  LLMs fundamentally struggle to differentiate between general information requests and queries that require personalized professional judgment, in this case personal medical advice. 

This limitation extends beyond healthcare to any domain requiring professional expertise: legal advice, financial planning, engineering safety assessments, or educational guidance. The core technical problem lies in contextual awareness and risk assessment.

Let's analyse potential technical approaches to mitigate these limitations.

1. Smarter Prompting 

From a user prompting perspective, this means users should frame health (and other mission-critical) queries more carefully and provide more context. Instead of just asking

"What can I replace chloride with?", which sounds like a chemistry question, a better approach would be: 

"I'm trying to understand sodium alternatives for dietary purposes. Can you explain common salt substitutes and their properties?".

This approach clarifies the context, which helps the LLM understand the user's specific situation and critically evaluate its assumptions. The result is a more accurate, elaborate and nuanced response.

2. Creating AI systems that put user safety and trust at their core

For businesses deploying LLMs in health-adjacent or other critical domains, implementing domain-specific safety layers becomes essential. This includes providing additional user and domain context to the AI system. In this case, this could mean state-of-the-art material in nutritional sciences and the user’s health history. AI systems can also be designed to proactively gather additional information about the user's workflow or use case. This context allows the system to tailor its responses and safety measures accordingly. 

Properly designed systems should also include guardrails that automatically recognise sensitive queries and provide appropriate disclaimers while directing users toward professional consultation when needed.

When AI Confidence Meets Legal Reality

In August 2025, a senior lawyer in Australia apologized for filing legal submissions in a murder case that included fake quotes and nonexistent case judgments generated by artificial intelligence [2]. The AI-generated errors caused a 24-hour delay in the proceedings, forcing the judge to postpone his planned ruling while the court verified the fabricated citations and awaited corrected submissions from the defense team. 

Unfortunately, this represents a troubling pattern rather than an isolated incident. Lawyers worldwide increasingly adopt LLM-based systems to improve their workflows, but they have repeatedly failed to recognise hallucinated citations, fabricated precedents, and incomplete legal research in LLM-generated content. The consequences extend beyond professional embarrassment: these errors compromise the integrity of legal proceedings, and in the long run, they can undermine trust in legal institutions. 

Technical explanation:

During their post-training LLMs are trained to provide confident, helpful-sounding answers that users will find satisfying, and they strive to do so even when they lack the specific information requested. This behaviour is actually pretty similar to how humans work. When faced with knowledge gaps, these systems don't admit uncertainty; instead, they attempt to fill it by generating responses based on what seems statistically probable, given their training data.

When asked about legal precedents, the model recognised linguistic patterns associated with court citations and generated plausible-sounding case names and quotes that fit those patterns, thus creating convincing legal fiction without any connection to the actual reality of the given case.

The human mistake:

The lawyer fell victim to two dangerous illusions: that the work was already complete, and that it had been done correctly. Rather than treating the AI output as a starting point requiring verification, the lawyer trusted the system's confident presentation and submitted the fabricated information directly to the court.

The Technical Solution:

A general LLM’s output, although most useful for initial research, ideation and drafting, for more serious purposes, it should be handled like work from a junior assistant, requiring thorough proofreading and fact-checking.

1. Smarter Prompting

From a user prompting perspective, users can reduce risks in legal research and other critical domains by adopting clever prompting techniques, such as explicit citation and verification requirements and clear boundaries. Instead of asking:

"What are the precedents for similar murder cases?", which invites fabrication, a better approach would be: 

"Help me identify potential legal precedents for similar murder cases, but provide only real case names with court and year information that I can independently verify in legal databases." 

This forces the AI to structure responses in a verifiable format and reminds both the user and the system that independent confirmation is required.

2. Responsible and Trustworthy AI Integration in Business

From the business perspective, organisations need to implement systemic safeguards beyond relying on user vigilance alone. This includes deploying specialised guardrails that can detect potential hallucinations and factual inconsistencies in AI-generated content. Additionally, businesses can enhance accuracy by providing ground truth through RAG (Retrieval-Augmented Generation) systems that connect the AI to verified, domain-specific databases and documentation. For legal applications specifically, this might involve integrating the AI with authenticated legal databases, case law repositories, and current statutory resources to ensure responses draw from verified sources rather than statistical predictions based on training data.

However, even with these technical improvements, human oversight remains essential - these tools enhance accuracy but cannot eliminate the fundamental need for expert verification in critical applications.

Conclusion: Building Trustworthy AI Use

The path forward with Large Language Models requires neither blind faith nor complete avoidance, but rather informed integration. Most shortcomings of the technology can be mitigated with careful system designs and user education.

While errors in creative writing or brainstorming may spark new ideas, the same inaccuracies in medical diagnosis or legal research can have devastating consequences.

For organizations looking to harness AI benefits while avoiding costly pitfalls, success relies on combining multiple techniques across different layers of implementation.

Key focus areas include:

1. Prompt Engineering

• User education for better questioning techniques

• Clear system prompts describing, for example, detailed instructions about the expected structure and style of the output

2. Context Engineering

• Information about the specific use case

• Relevant user context and background

3. Adding Ground Truth

• Retrieval-Augmented Generation (RAG) implementation

• Supporting documents and reference materials

4. Guardrails

• Safety validation checks for sensitive topics
• Hallucination detection to prevent false and fabricated information

As AI capabilities continue to advance, organizations that invest in building robust context engineering practices and considerate guardrail systems will be best positioned to capture AI's benefits while minimizing its risks. In this process, the apparent "magic" of AI becomes far less mysterious and far more trustworthy when we understand that these systems are sophisticated pattern-matching tools that perform best when given clear context, appropriate constraints, and realistic expectations about their capabilities.

The cases examined—ranging from medical misadventures to legal fabrications—illustrate a consistent pattern: problems arise not from the technology itself, but from irresponsible use and misaligned expectations about what these general systems can and cannot reliably do. Success requires not just the technology, but deliberate design choices and robust oversight mechanisms. 

Sources:

1. A Case of Bromism Influenced by Use of Artificial Intelligence https://www.acpjournals.org/doi/10.7326/aimcc.2024.1260?ref=404media.co

2. Australian lawyer apologizes for AI-generated errors in murder case https://apnews.com/article/australia-murder-artifical-intelligence-34271dc1481e079c3583b55953a67c38 

FAQ

Q: Are LLMs actually dangerous, or are these just isolated incidents?

A: LLMs themselves aren't inherently dangerous, but misusing them in high-stakes situations without proper oversight can lead to serious consequences. The key is understanding when and how to use them appropriately.

Q: How can I tell when an LLM is hallucinating or making things up?

A: LLMs often present false information with the same confidence as accurate information, making detection difficult. Always verify critical information through authoritative sources, especially for legal, medical, or financial matters.

Q: What's the difference between using AI for creative work versus professional decisions?

A: Creative applications are more forgiving of occasional inaccuracies, while professional domains like law and medicine require strict verification since errors can have serious real-world consequences.

Q: Can better prompting really prevent these problems?

A: Improved prompting helps by providing context and requesting verifiable information, but it's not foolproof. Technical safeguards and human oversight remain essential for critical applications.

Q: Should businesses avoid AI altogether to prevent these risks?

A: Complete avoidance means missing significant productivity benefits. Instead, businesses should implement appropriate guardrails, treat AI as an assistant rather than an expert, and maintain human oversight for important decisions.

Q: What is prompt engineering, and why does it matter?

A: Prompt engineering is the practice of crafting clear, detailed instructions and examples to get AI models to produce better outputs - essentially learning how to "talk" to AI effectively rather than just asking vague questions. It matters because the quality of your results depends heavily on how well you communicate what you want, turning AI from a frustrating guessing game into a reliable tool for getting specific, useful responses.

Q: What is context engineering, and how does it differ from prompt engineering?

A: Context engineering takes prompt engineering further by systematically providing business-specific information, use case details, and relevant background data within prompts to dramatically improve accuracy for professional applications. While prompt engineering focuses on clear communication techniques, context engineering involves systems that automatically inject domain knowledge (often via RAG - Retrieval Augmented Generation), user context, company data, and situational details into prompts, helping AI understand your specific constraints and objectives for real business decisions rather than just generic responses.

Q: Will future AI models solve these hallucination problems?

A: While improvements are likely, the fundamental pattern-matching nature of LLMs means some level of hallucination risk will persist. Focus on building robust verification processes rather than waiting for perfect AI models.

Q: How do I know if my company is using AI responsibly?

A: Look for clear policies about AI use, verification requirements for AI-generated content, domain-specific safety measures, and maintain human oversight in critical decision-making processes.