- In an April 2026 paired benchmark, adding a 4 KB hand-written document describing a dataset’s measures and conventions raised accuracy by 17 to 23 points across Claude Opus 4.7, Claude Sonnet 4.6, and GPT-5.4.
- That document accounted for nearly all of the significant variance in the results. Model choice within a tier did not.
- Even with the context document in place, accuracy topped out near 68%. Context moves an agent off a coin flip without making it dependable.
- Business definitions decay when the people who set them leave. The column name survives; the shared understanding doesn’t.
- An agent never signals the gap. It returns a confident answer, the same way a listener skips past a reference they don’t recognize.
Your AI returns confident wrong answers because nobody wrote down what your terms mean, and it has no way to tell you that’s what happened. A better model won’t close that gap. In a paired benchmark published in April 2026, one page of human-written context moved three frontier models up by 17 to 23 points, and which model you picked made no statistical difference at all.
I worked this out from a song.
The line I’d been hearing wrong for years
“Eso Se Baila Así” is the sixth track on The Hustler, Willie Colón’s 1968 album with Héctor Lavoe on vocals. Somewhere in the montuno, voices in the band call out three names, Gabi, Fofo, Miliki, and Lavoe answers them.
I’d heard the song for years, and what I heard was “Migaly, Fofo, Miliki. Miliki, Fofo, Migaly.” I’d never read the lyrics. It sounded like gibberish thrown in to make the line rhyme, and nothing about it asked me to stop and check.
Then, recently, I was watching something else. I think it was a movie, and I still can’t remember which one. Someone said the clowns’ names, and the connection landed all at once. I looked up the lyrics and confirmed it: I’d been hearing it wrong the whole time. I never heard the line right because I had no context to hear it right.
They’re clowns. Gaby, Fofó and Miliki were three Spanish brothers, the Aragóns, and between 1965 and 1971 they ran a daily program in Puerto Rico called El Show de las 5. It was one of the most watched shows in the island’s television history, and their catchphrases, “¿Cómo están ustedes?” and “se me lengua la traba,” worked their way into everyday Puerto Rican speech.
Get the free ebook: dbt for the Enterprise
Four ways to run dbt at enterprise scale: the dbt platform, dbt on Snowflake, a DIY stack, and managed dbt in your own cloud. Compare them on cost, security, and what each one leaves you to build.
Get the PDF
The reference had expired
The album came out in 1968, right in the middle of that run. Colón was still a teenager, and the band was recording in New York for an audience that had watched those clowns that same afternoon.
Nothing about the line was cryptic at the time. The band was being current, and in 1968 it landed instantly, the way naming a contestant from last night’s reality show lands now. It needed no explanation because the band and the audience shared the same week.
Then the audience aged out. The recording never changed. Not one sample, not one syllable. What changed is that everyone who needed no footnote stopped being the people listening.
Now I’m the one whose references die
I have kids. I make a reference to something from my own childhood and watch it travel about three feet and drop.
What gets me is what they do next. They don’t ask. Their faces don’t change. The sentence parses, nothing in it registers as missing, and the conversation keeps going. From the outside, that exchange looks exactly like one where I was understood.
Silence reads as comprehension. That’s what makes this failure so hard to catch, in a conversation and in a dashboard.
At Amgen we defined “dormant vendor,” and then we both left
Dormant vendors got handled differently in reporting and in procurement, so the definition had teeth. Two of us settled it: me and the person who ran vendor management. We agreed on the rule, applied it, and the business ran on it for years.
She left the company. Later, I left too.
The data is still there. The process is still there. The term is still in use. The agreement that gave it meaning walked out of the building in two pieces, a few months apart, and nothing in any system noticed. Anyone asking “how many dormant vendors do we have” today gets a number. Whether it’s the number we meant is not a question the warehouse can answer.
Your organization has dozens of these. A status code whose third value means something specific to the team that created it in 2016. A flag that was a stopgap during a migration and became permanent. A date field that gets populated only when a different system fails. A metric whose definition lives in the head of someone two reorgs away. None of it is documented, because at the time it was obvious to everyone who needed it.
Your agent is the newest person in the room, permanently
A new analyst picks this up slowly and badly, by sitting near people, overhearing corrections, and asking questions that make them look uninformed. It takes months. It works because they’re embedded in the room where the context lives.
An agent gets none of that. It never attended the meeting where you settled the definition, never overheard the correction in the hallway, and never built up the sense of which fields to distrust. Every model upgrade resets whatever it had. And unlike the new analyst, it’s expected to produce an answer immediately, every time, without the social permission to say it doesn’t know.
So it does what my kids do with my jokes. It parses the sentence, finds nothing marked as missing, and keeps going.
A page of writing beat a better model
Two researchers put a number on this. In “Semantic Layers for Reliable LLM-Powered Data Analytics”, published 28 April 2026, Michael Rumiantsau and Ivan Fokeev ran 100 natural-language questions against a retail dataset in ClickHouse, using Claude Opus 4.7, Claude Sonnet 4.6, and GPT-5.4.
Each model answered the same questions twice. Once with only the warehouse schema. Once with the schema plus a 4 KB hand-authored markdown file describing the dataset’s measures, its conventions, and its disambiguation rules.
The paper’s conclusion is the line I’d put on a wall: “the semantic-layer document accounts for essentially all of the significant variance; model choice within tier does not,” at p < 0.01.
Three of the strongest models available clustered together without the context and clustered together with it. The gap between them was noise. The gap between having the document and not having it was the entire result. Four kilobytes of plain writing by someone who understood the data outperformed every difference between frontier models.
Two caveats I’d rather state than have you find.
The first is the ceiling. Even with the document, accuracy landed near 68%. That’s a meaningful lift from a coin flip, and it is nowhere near good enough to point at a board deck and walk away. Context is necessary here without being sufficient.
The second is about benchmarks generally. A January 2026 paper found annotation error rates of 52.8% in BIRD Mini-Dev and 62.8% in Spider 2.0-Snow, with corrected labels moving agent rankings by as much as nine positions. Text-to-SQL leaderboards move fast and measure less cleanly than their headline numbers suggest. The reason the April study holds up is its design: the same models, the same questions, one variable changed.
Why adding more of everything doesn’t close the gap
The instinct when an agent gets something wrong is to feed it more. More rows, a longer context window, more documents in the retrieval index, a newer model. Each of those fixes a real problem. None of them fixes this one.
Playing “Eso Se Baila Así” a thousand more times would have taught me nothing about those three names. One sentence from someone who grew up watching that show taught me everything. The information was never in the recording, so no amount of the recording was going to produce it.
Writing it down starts it, agreeing on it makes it hold
The last row of that table is where most documentation efforts quietly fail. Someone writes a definition, it goes in a wiki, and a year later there are four wikis with four definitions and no way to tell which one the business stands behind. We’ve written before about why LLMs still need people checking their work, and this is the structural version of that argument.
What my Amgen story needed was not a better data dictionary. The definition existed and was correct. It needed to be on the record, attributed to the people who owned it, in a place that outlived both of us, and changed only through a review that those people’s successors would see.
That’s the problem Atlas was built for. When a question has two defensible readings, Atlas states the judgment call and asks instead of guessing, routes it to whoever owns the answer, and turns the outcome into a Definition of Record: plain English, versioned, attributed, changed through review. Every AI tool reads it over MCP. I wrote about why we built it in Introducing Atlas.
Governance earns its place here for a reason you can measure. GROUND, published in July 2026, tested governed semantic definitions against a 100-question enterprise reporting benchmark and found that a semantic-only setup with accurate metric definitions but no access controls still leaked data. Correct definitions and governed definitions are not the same artifact.
Where to start this week
Pick one term your executives use in meetings without ever defining it. Dormant vendor, active customer, qualified lead, whatever yours is.
Ask three people what it means, separately, in writing. If you get three answers, you’ve found a definition your AI is currently guessing at, and you’ve found it before it reached a board deck. Write down the one the business stands behind, put a name on it, and make your tools read it from one place. If you want help with the platform underneath that, book a free architecture review.
Then ask who else knows what that term means, and what happens the week they resign.
