Introduction
Large Language Models (LLMs) like ChatGPT and Claude are becoming common in modern data workflows. From writing SQL queries to summarizing dashboards, they offer speed and support across both technical and non-technical teams. But as organizations begin to rely more heavily on these tools, the risks start to surface.
The dangers of AI are not in what it cannot do, but in what it does too confidently. LLMs are built to sound convincing, even when the information they generate is inaccurate or incomplete. In the context of data analytics, this can lead to hallucinated metrics, missed context, and decisions based on misleading outputs. Without human oversight, these issues can erode trust, waste time, and create costly setbacks.
This article explores where LLMs tend to go wrong in analytics workflows and why human involvement remains essential. Drawing from current industry perspectives and real-world examples, we will look at how to use LLMs effectively without sacrificing accuracy, accountability, or completeness.
LLMs Are Impressive, But Not Always Right
LLMs like ChatGPT and Claude are excellent at sounding smart. That’s the problem.
They’re built to generate natural-sounding language, not truth. So, while they may give you a SQL query or dashboard summary in seconds, that doesn’t mean it’s accurate. In fact, many LLMs can hallucinate metrics, invent dimensions, or produce outputs that seem plausible but are completely wrong.
You wouldn’t hand over your revenue targets to someone who “sounds confident,” right? So why are so many teams doing that with AI?
Here's the challenge. Even experienced data professionals can be fooled. The more fluent the output, the easier it is to miss the flaws underneath.
This isn’t just theory, Castor warns that “most AI-generated queries will work, but they won’t always be right.” That tiny gap between function and accuracy is where risk lives.
The Real Dangers of AI in Analytics
If you’re leading a data-driven team or making decisions based on LLM-generated outputs, these are the real risks you need to watch out for.
1. Hallucinated Insights
LLMs can fabricate filters, columns, and logic that don’t exist. In the moment, you might not notice but if those false insights inform a slide for board meeting or product decision, the damage is done.
Here is an example where an image was provided to ChatGPT and asked to point out the three differences between the two pictures as illustrated below:
Here is the output from ChatGPT:
Here are the 3 differences between the two images:
- Eyebrow: The character on the left has an eyebrow, while the one on the right does not.
- Shirt Stripe: The red shirt under the jacket has two white stripes on the left image, but only one stripe on the right image.
- Shoe Sole: The left image shows a visible gray sole on the right shoe, which is missing in the right image.
Let me know if you want these highlighted visually!
When asked to highlight the differences, ChatGPT produced the following image:
As you can see, ChatGPT skewed the information and exaggerated the differences in the image. Only one (the shirt stripe) out of the three was correct, while it missed the differences with the sock, the hair, and it even changed the author in the copyright!
2. No Understanding of Business Context
AI doesn’t know your KPIs, fiscal calendars, or market pressures. Business context still requires a human lens to interpret properly. Without that, you risk misreading what the data is trying to say.
It’s Hard to Verify
AI doesn’t give sources or confidence scores. You often can’t tell where a number came from. Secoda notes that teams still need to double-check model outputs before trusting them in critical workflows.
Non-Technical Teams May Misuse It
One of the great things about LLMs is how accessible they are. But that also means anyone can generate analytics even if they don’t understand the data underneath. This creates a gap between surface-level insight and actual understanding.
Too Much Automation Slows You Down
Over-relying on LLMs can create more cleanup work than if a skilled analyst had just done it from the start. Over-relying on automation often creates cleanup work that slows teams down. As we noted in Modern Data Stack Acceleration, true speed comes from workflows designed with governance and best practices.
Why Human Oversight Still Matters
If you’ve ever skimmed an LLM-generated responce and thought, “That’s not quite right,” you already know the value of human oversight.
AI is fast, but it doesn’t understand what matters to your stakeholders. It can’t distinguish between a seasonal dip and a business-critical loss. And it won’t ask follow-up questions when something looks off.
Think of LLMs like smart interns. They can help you move faster, but they still need supervision. Your team’s expertise, your mental model of how data maps to outcomes, is irreplaceable. Tools like Datacoves embed governance throughout the data journey to ensure humans stay in the loop. Or as Scott Schlesinger says, “AI is the accelerator, not the driver.” It’s a reminder that human hands still need to stay on the wheel to ensure we’re heading in the right direction.
Use cases that add immediate value without introducing much risk
Datacoves helps teams enforce best practices around documentation, version control, and human-in-the-loop development. By giving analysts structure and control, Datacoves makes it easier to integrate AI tools without losing trust or accountability. Here are some examples where Datacoves’ integrated GenAI can boost productivity while keeping you in control.
- Debugging errors: GenAI helps humans pinpoint code errors and suggest fixes and human expertise ensures changes are correct and safe.
- Model testing: GenAI along with MCP servers can be used to profile data and recommend tests to an experienced analytics engineer and assure tests make sense for the business.
- Automating documentation: GenAI can be used to add and populate missing dbt yml files by adding column details and documentation. This automation saves time, but human validation remains critical to ensure accuracy and context.
- Onboarding docs: AI can draft detailed onboarding guides with project structure, tools, and best practices, saving time for team leads, while allowing them to validate output for completeness and accuracy.
- Other Documentation: GenAI can accelerate generation of documentation for webinars or repos, but manual review is key to catch inaccuracies.
How to Use LLMs Without Losing Control
Want to keep the speed of LLMs and the trust of your team? Here’s how to make that balance work.
- Always review AI-generated queries and summaries before sharing
- Use LLMs to support analysts, not replace them
- Make sure the underlying data is reliable and well-documented
- Train business users to validate outputs before acting
- Create workflows where human review is built in
Conclusion
AI can enhance analytics but should not be used blindly. LLMs bring speed, scale, and support, but without human oversight they can also introduce costly errors that undermine trust in decision-making.
Human judgment remains essential. It provides the reliability, context, and accountability that AI alone cannot deliver. Confidence doesn’t equal correctness, and sounding right isn’t the same as being right.
The best results come from collaboration between humans and AI. Use LLMs as powerful partners, not replacements. How is your team approaching this balance? Explore how platforms like Datacoves can help you create workflows that keep humans in the loop.
