Benchmarking AI Models for Large Repository Analysis

Context windows are no longer the primary bottleneck for automated code analysis. The industry shifted from fitting code into memory to reasoning across disconnected modules.

We benchmarked GPT-4o, Claude 3.5 Sonnet, and Gemini 1.5 Pro against the git11 internal evaluation suite. This suite consists of 500 tasks across 10 private TypeScript and Go repositories. Every repository exceeds 200,000 lines of code.

Raw context size matters less than retrieval-augmented generation (RAG) efficiency and logic tracing. Models often hallucinate imports or fail to follow interface implementations across deep directory structures. Our data shows a 14% delta between the top and bottom performers.

Logic Tracing Scores

Logic tracing measures a model's ability to follow a function call through multiple layers of abstraction. We tested this by asking models to identify the exact SQL query triggered by a specific API endpoint. The path typically involves a controller, a service layer, a repository interface, and a database driver.

• Claude 3.5 Sonnet: 92% accuracy.
• GPT-4o: 84% accuracy.
• Gemini 1.5 Pro: 78% accuracy.

Claude 3.5 Sonnet correctly identified the JOIN syntax in infra/db/postgres/query_builder.go after being prompted from api/v1/users.go. It did not get lost in the middle-ware wrappers. GPT-4o failed when the interface implementation resided in a separate git submodule.

Gemini 1.5 Pro has the largest context window but suffers from the lost-in-the-middle phenomenon. It remembers the beginning and end of the codebase but misses nuances in the middle layers. We found its attention mechanism skips over complex generic types in Go.

Dependency Mapping Performance

Mapping a dependency graph requires understanding how changes in one file propagate through the system. We tasked the models with predicting the blast radius of changing a core AuthContext type. This type is used in 42 separate files across the frontend and backend.

Claude tracked 40 of 42 dependencies accurately. It flagged a subtle edge case in an obscure webhook_handler.ts file that uses the type via a dynamic import. GPT-4o missed four files, focusing only on direct imports. Gemini mapped 31 files but failed to categorize the secondary effects on the CI/CD pipeline.

Accuracy drops as the distance between the source file and the dependency increases. We define distance by the number of directory hops. Claude maintains high precision up to seven hops. GPT-4o degrades significantly after four hops.

Handling Technical Debt

Most benchmarks use clean, green-field code. We used real repositories filled with technical debt, circular dependencies, and deprecated functions. Understanding bad code is more difficult than writing new code.

1. Claude identifies deprecated patterns and suggests modern replacements based on existing project conventions. 2.

GPT-4o tends to rewrite code from scratch rather than fixing the existing logic. 3. Gemini 1.5 Pro often suggests external libraries that are not currently in the package.json file.

We evaluated models on their ability to respect eslint and golangci-lint rules without being told. Claude checked the .eslintrc.json file automatically. The other models ignored it until we explicitly pointed to the file path. This autonomy defines the next generation of analysis tools.

Latency and Token Costs

Performance is irrelevant if the developer experience is too slow. We measured the time to first byte (TTFB) for a standard "Explain this repository" query. We used a standardized input of 50,000 tokens.

• GPT-4o: 1.2 seconds.
• Claude 3.5 Sonnet: 2.1 seconds.
• Gemini 1.5 Pro: 4.5 seconds.

GPT-4o is the fastest for small, iterative tasks. If you need a quick explanation of a 50-line function, use GPT-4o. If you need to refactor a feature that spans ten files, wait the extra second for Claude. The cognitive load of fixing a model's mistake far outweighs the latency of the initial response.

Gemini is prohibitively slow for interactive use. It serves better for batch processing tasks, such as generating documentation for an entire repository overnight. Its cost per million tokens is lower, making it the choice for high-volume, low-criticality tasks.

Context Window Realities

Gemini 1.5 Pro offers a 2-million-token window. This allows you to dump an entire monolith into a single prompt. However, more noise leads to lower signal. The model struggles to prioritize the README.md over a random temp.log file if both are in the context.

We developed a strategy called selective pruning for git11. We do not send the whole codebase. We send relevant files based on a vector search, then let the model request more files. This iterative approach works best with Claude’s reasoning engine.

1. Vector search identifies the top 20 relevant files. 2. Model analyzes files and identifies missing dependencies.

1. System fetches missing files on-demand. 4. Final answer is generated with high precision.

This method reduces token usage by 60% and increases accuracy by 25%. Blindly using a massive context window is a lazy engineering choice that yields sub-optimal results.

Identifying Logic Flaws

We inserted 50 intentional logic bugs into a fork of a production repository. These were not syntax errors. These were concurrency issues, race conditions, and incorrect permission checks. We measured how many bugs each model could find voluntarily.

Claude found 38 bugs. It accurately predicted a race condition in a sync.Map implementation in Go. It explained why the current locking mechanism would fail under high load. GPT-4o found 22 bugs, mostly focusing on basic null pointer exceptions. Gemini found 15 bugs.

Logic flaws require a deep semantic understanding of the codebase. A model must understand not just what the code does, but what it is expected to do. Claude 3.5 Sonnet is the only model that consistently infers intent from variable names and surrounding comments.

Refactoring Complex Logic

Refactoring requires a model to maintain the original behavior while improving structure. We gave the models a 400-line "God Object" in Java and requested a decomposition into smaller classes. The goal was to maintain binary compatibility with existing calls.

• Claude 3.5 Sonnet: Created 5 classes, maintained API compatibility, and wrote unit tests for the new structure.
• GPT-4o: Created 3 classes but broke two public methods.
• Gemini 1.5 Pro: Suggested a rewrite that required changing the database schema.

Claude's ability to generate unit tests that actually pass is its greatest asset. When it refactors code, it validates its own assumptions. It uses the testing package correctly and mocks external services without prompting.

The Git11 Stack

After six months of testing, we moved the git11 backend to a multi-model architecture. We use GPT-4o for simple syntax highlighting and quick summaries. We use Claude 3.5 Sonnet for all complex reasoning, dependency mapping, and refactoring tasks. Gemini is reserved for long-running background tasks like building full-repo indexes.

Choosing one model for every task is a mistake. The variance in performance across different tasks is too large to ignore. You must match the model's strengths to the specific requirements of the code analysis step.

Use Claude 3.5 Sonnet as your primary engine for code understanding.