A recent study confirms what I've suspected: giving your AI coding agent too many instructions in repository-level context files actually reduces task success.
We often assume that giving our AI coding agents more instructions will yield better results. We write extensive .cursorrules, claude.md, or AGENTS.md files to ensure our agents understand every nuance of our architecture. However, a recent study published on arXiv titled "Evaluating AGENTS.md: Are Repository-Level Context Files Helpful for Coding Agents?" reveals that this practice is often counterproductive.
The researchers evaluated how coding agents perform on real-world tasks when provided with LLM-generated or developer-provided context files, compared to having no repository context at all. The results are striking. Across multiple agents and models, providing these context files actually tended to reduce task success rates. Worse, it increased inference costs by over 20%.
While this seems counterintuitive, it perfectly aligns with the well-documented "Lost in the Middle" phenomenon (Liu et al., 2023), which proved that as we stuff more context into an LLM's prompt window, it becomes increasingly likely to ignore information buried in the middle of that context in favor of the extremes.
But the most fascinating insights come from the paper’s trace analysis. The researchers looked closely at what the agents were doing when given a context file. They found that providing AGENTS.md and similar files significantly increases the number of tests run by coding agents and drastically expands their codebase exploration. The agents were trying to be "good interns"—they followed the instructions, but doing so required more "thinking" tokens and led them down unnecessary rabbit holes.
The study confirms a concept I explored recently regarding context poisoning and the folly of MCP. When we give an AI too much information, we introduce noise. As we've seen with the rush to connect every possible data source via MCP, a crowded context window often degrades the model's ability to reason. The agents respect the instructions they are given, but unnecessary requirements (like comprehensive code quality checks or unrelated test suites) simply make the core tasks harder to accomplish. As the paper’s per-repository success analysis showed, there was no single repository where the presence of context files actually had a significant positive impact.
When you hire a junior engineer, you do not hand them a 100-page manual on your preferred logging syntax before asking them to fix a typo on the marketing page. You give them the specific, actionable instructions they need for the task at hand. This echoes why it is so crucial to separate understanding from execution and strategically clear your context in plan mode.
We need to treat our AI agents with the same focused constraint. The primary takeaway from the study is clear: human-written context files should describe only minimal requirements.
If we overload our foundational instructions with every possible edge case and architectural preference, we dilute the signal. The agent spends its limited cognitive capacity trying to satisfy constraints that may have nothing to do with solving the current problem.
To build more effective AI workflows, we need to rethink what goes into our repository-level context files. We must transition from exhaustive rulebooks to targeted guardrails.
Here is what a problematic, overly verbose context file looks like:
# BAD: The Exhaustive Rulebook
- Always use our custom LoggerInterface for every error, never use console.log or standard print statements.
- Every new function must include at least three standalone unit tests using Jest.
- Never use relative imports, always use absolute paths configured in tsconfig.
- Always check the database schema before making any API changes.
- Commits must follow the exact Conventional Commits format with Jira ticket numbers.
- Ensure all new UI components are responsive across mobile, tablet, and desktop breakpoints.
This is too much noise. An agent trying to fix a simple text alignment issue will suddenly start looking for database schemas and writing unnecessary unit tests because the file instructed it to do so.
Instead, here is what a minimal, effective context file looks like:
# GOOD: The Targeted Guardrails
- We use Tailwind CSS for all styling.
- The project is a Next.js App Router application.
- Check the local `mistakes.md` file before starting a new feature to review previous known issues.
This approach gives the agent the architectural foundation it needs without imposing a heavy cognitive burden. The agent knows the styling framework and the routing paradigm, and it has a pathway to handle repeated mistakes without being bogged down by irrelevant constraints.
I enjoy building tools that leverage AI to solve complex problems, and the key to success is almost always focused intent. If you want to explore this idea further or build something interesting together, reach out. The next time you set up a repository for an AI agent, resist the urge to document every rule. Keep your context files minimal, and let the agent focus on the code that actually matters.