Every AI code review vendor has a benchmark that says it wins. Greptile's own test has Greptile catching 82% of bugs while CodeRabbit catches 44%. CodeRabbit promotes a different study where it comes first at 51.2% F1. Qodo published one where Qodo tops the field. These claims are not lies. They are worse than lies: they are measurements designed by the winner, using datasets the winner picked, with a definition of "caught" the winner chose.
So when your team is deciding between CodeRabbit, GitHub Copilot code review, Greptile, Cursor Bugbot, and CodeAnt, the vendor benchmarks are close to useless. The numbers swing wildly depending on who runs the test. Greptile scored itself at 82% catch rate on 50 real bug-fix pull requests from Sentry, Cal.com, Grafana, Keycloak, and Discourse. When Augment Code re-ran an evaluation on the same five repositories with its own ground-truth set, Greptile dropped to roughly 45% F1. Same repos. Same tool. Half the score, different referee.
There is one benchmark that transfers to your context: a seeded repository you build yourself. You plant bugs you know exist, run each reviewer against them, and count what comes back. It costs an afternoon. This is how you actually find out which AI code review tool catches real bugs, which one buries your pull requests in noise, and which one is worth $24 to $40 per developer per month.
The seeded-repo method: plant what you know, measure what comes back
The idea is borrowed from penetration testing. You do not trust a security vendor's marketing claims about your attack surface. You run a controlled exercise against your own systems and count what gets through.
For code review, the exercise looks like this:
- Pick 30 to 50 recent merged pull requests from a real repository you own, ideally one written after the reviewers' training cutoffs. Fresh PRs protect you from contamination, where the model has simply memorized a public bug and its fix.
- Seed 12 known defects across three categories. This is the ground truth list, and you write it before any tool runs.
- Add 20 clean PRs as a control group. Whatever the tools say on those is your false-positive rate.
- Run each reviewer at default settings. Custom rules are tuning, and tuning per vendor is not a fair comparison.
- Score everything: caught, missed, and every comment on a clean line.
The 12 seeded bugs should map to what each vendor claims it catches, so every tool gets a fair shot. A workable set, drawn from published methodology guides and real incident patterns:
| Category | Seeded defects |
|---|---|
| Correctness | Off-by-one loop bound (i <= total instead of i < total), calling .toLowerCase() on an optional email field, two goroutines mutating a shared map without a lock, comparing timestamps with == instead of .Equal() |
| Security | SQL built by string concatenation with user input, an API key committed to a config file, pickle.loads on untrusted input, a query parameter rendered into HTML unescaped |
| Architectural | A function whose return type changed while three internal callers still compile, a schema change with no migration, an error branch made unreachable by an earlier guard, a "getter" that mutates a global cache |
If you want to see this done at scale instead of doing it yourself, the Greptile benchmark page is a decent template, even though the numbers are self-reported. It took 50 real bug-fix PRs from five open-source projects across Python, TypeScript, Go, Java, and Ruby, ran five reviewers at default settings, and counted a bug as caught only when the tool left a line-level comment explaining the impact. The method is sound. The scoreboard is the part to distrust.
What the published tests actually show
Before you seed your own repo, it helps to know the shape of the results. Aggregating the independent and semi-independent tests that were published or re-run in 2026, a consistent picture emerges:
| Test | What it measured | Greptile | Bugbot | Copilot | CodeRabbit |
|---|---|---|---|---|---|
| Greptile's self-test (50 PRs) | Catch rate | 82% | 58% | 54% | 44% |
| Augment re-run (same 5 repos) | F1 score | 45% | 49% | 25% | 39% |
| Entelligence (67 production bugs) | F1 score | 36.9% | 39.4% | 22.6% | 33.0% |
| CodePulse re-adjudication (183 defects) | Precision | 36.5% | 81.2% | 35.1% | 44.8% |
Three findings survive across every test:
Cross-file context buys recall. Greptile indexes the whole repository before commenting, so it catches bugs that live outside the diff: an import of a module that does not exist, a caller three files away that breaks when a return type changes. In its own benchmark it caught 100% of high-severity bugs. That is the specific class of bug that gets past human review too, because reviewers skim.
Cross-file context costs precision. In the CodePulse re-adjudication, Greptile posted 316 comments to find 96 real defects. 111 were nitpicks. 56 were flat wrong. CodeRabbit and Copilot both landed below 45% precision in the same exercise. Bugbot was the disciplined one: 81.2% precision, the second-lowest false-positive rate in a separate three-week parallel study of four reviewers. Catch rate and noise are the same dial. You cannot buy the 82% without buying the noise that comes with it.
No two reviewers see the same things. A team that ran four reviewers in parallel for three weeks across 146 PRs found that 93.4% of flagged lines were flagged by exactly one tool. All four reviewers never converged on a single line of code. The practical consequence is uncomfortable for buyers: one AI reviewer, any AI reviewer, leaves real bugs on the table that a different one would have caught.
The five tools, priced as of September 2026
Ground truth tells you what a tool catches. Pricing tells you what it costs. Both went through changes in 2026 that most comparison articles have not caught up with.
CodeRabbit: the safe default, now with new plan names
CodeRabbit renamed its plans in 2026. Pro became Essentials, Pro Plus became Team, and a new Advanced tier appeared. Essentials is $24 per developer per month billed annually ($30 monthly) and buys the review itself: PR reviews at 5 per developer per hour, 150 files per review, autofix, linter and SAST integration, and one linked repository. Team doubles the rate limits and adds triage, custom pre-merge checks, unit-test generation, and merge-conflict resolution. Advanced adds a security review of every PR, continuous security monitoring, and blast-radius analysis. There is a free tier with PR summarization only, and a usage add-on at $0.25 per reviewed file past your limits. Full limits are in the CodeRabbit plans documentation.
Two things make it the safe default. It is the only major reviewer covering GitHub, GitLab, Bitbucket, and Azure DevOps. And independent comparisons consistently put it at the low-noise pole: roughly 2 false positives per PR against Greptile's 11 in one April 2026 comparison. The trade is depth. It reads the diff, not the whole codebase, so cross-file regressions are its blind spot.
GitHub Copilot code review: cheap, bundled, and quietly metered
If your team pays for Copilot, code review is included. Pro is $10 a month, Business is $19 per seat, Enterprise $39. But the billing changed on June 1, 2026, and this is the part finance will miss: reviews are billed through GitHub AI Credits, and the review model is selected automatically and not disclosed, so per-review costs vary. Private-repository reviews also consume GitHub Actions minutes on GitHub-hosted runners. Self-hosted runners are exempt. The details are in GitHub's Copilot billing documentation.
The capability curve has been steep through 2026. In August, GitHub removed the 300-file, 20,000-line cap on reviewable pull requests, added the ability to review bot-authored PRs, and shipped comment resolution reasons (Addressed, Won't fix, Incorrect), which you can read about in the GitHub changelog. In early September, Copilot code review gained the ability to approve PRs. On catch performance it is mid-pack in third-party tests, which is genuinely fine for a bundled feature. If it is already in your bill, start there and seed-test before buying anything else.
Greptile: highest recall, and now metered per review
Greptile is the whole-repo indexer. It builds a graph of your codebase, functions, imports, call chains, and reviews the diff against that graph. That is where the 82% comes from, and it is real in the sense that the method genuinely catches cross-file bugs. Pricing is $30 per seat per month with 50 review credits included, then $1 per additional credit, verified on the Greptile pricing page. There is a free Starter tier with 50 credits a month for one developer, and Enterprise adds self-hosting and SSO.
The credit model arrived in March 2026 and caused a minor revolt. A developer opening three PRs a day burns through the included 50 reviews and starts paying overage, and teams running agentic coding workflows, where agents open dozens of small PRs, can blow far past it. One developer publicly reported a month of 571 PRs, which would take the bill from $30 to well over $500. Greptile's counter is that fewer than 10% of active users exceed the cap. Both are probably true. Know which kind of user you are before signing up.
Cursor Bugbot: the quiet precision play
Bugbot changed shape in 2026. The standalone $40-per-seat subscription was retired in May, and Bugbot now rides on usage-based billing on top of a Cursor plan. Pro is $20 a month, Teams is $40 per user. Cursor's help center puts the average run at $1.00 to $1.50, and the unit is a run, not a pull request: by default Bugbot reviews every push to an open PR, so an iterative branch bills repeatedly. The Cursor pricing page confirms the usage-based setup. A June 2026 update made it three times faster and cut the per-run cost, and a /review command now runs it locally before you even push, deduplicating against the later GitHub run.
In the precision-focused tests, Bugbot is the standout among the big names: 81.2% precision in the CodePulse re-adjudication, second-lowest false-positive rate in the four-reviewer parallel study. It deliberately skips style nits and goes after logic bugs, race conditions, and unhandled nulls. The catch: it is GitHub-only, and its diff-plus-reasoning approach still misses the cross-file bugs that whole-repo indexers catch. A natural pairing, if your budget allows two reviewers, is Bugbot plus Greptile: one precise, one exhaustive, minimal overlap.
CodeAnt AI: when security findings need to be first-class
CodeAnt bundles AI PR review with SAST, secret detection, infrastructure-as-code scanning, and DORA metrics in one platform, at $24 per user per month for Premium, with a 14-day trial and no permanent free plan. It covers all four major Git hosts. If your seeded-repo test weights the security category heavily, and it should if you ship anything internet-facing, a bundled reviewer-plus-scanner is structurally different from a reviewer that shells out to a linter. A SQL injection pattern gets a review comment and a SAST finding in the same pass, instead of a bot comment now and a scanner ticket two hours later. Its vendor-reported numbers, under 5% false positives, under 60 seconds per PR, are the kind you verify with your own seeded repo rather than take on faith. Enterprise adds on-prem, VPC, and air-gapped deployment.
The noise problem nobody prices in
The seeded-repo method scores catches. Your team's actual experience is dominated by the misses and the noise, and noise is the expensive part.
Run the math from the CodePulse re-adjudication. Greptile's 316 comments on 51 PRs break down to 96 real defects, 111 nitpicks, and 56 wrong comments. That is roughly 3.3 noise comments per pull request that a human has to read, evaluate, and dismiss. At scale that is not a rounding error. It is a second job. And it carries a failure mode worse than missing bugs: engineers learn to skim the bot, then to skip it, at which point your catch rate is zero regardless of what any benchmark said.
This is why the control group of clean PRs in your seeded test matters as much as the seeded bugs. The 20 clean PRs tell you what a tool says when it has nothing real to say. For teams with low tolerance, the ordering that held up across the 2026 tests is Bugbot (most restrained) and CodeRabbit (low volume, though templated), with Greptile and Copilot generating the most per-PR commentary. The 2026 research literature formalized this as signal-to-noise ratio being the metric that predicts whether a team keeps a reviewer installed. Trust is the adoption bottleneck, not recall.
There is also a class of bug that none of them catch reliably, and CodeRabbit's own analysis of 470 PRs is the honest source: AI-assisted code carries roughly 1.75 times more logic errors, about twice the concurrency bugs, and 2.74 times more cross-site scripting issues than human-only code. Concurrency and domain-validation bugs are precisely where every reviewer, human and AI, is weakest. Your seeded repo should include at least one race condition so you can see this yourself.
Where each tool pays for itself
Solo developer or a team of two to three on GitHub: Copilot code review first, because you may already be paying for it. If you are not, Greptile's free Starter tier, 50 reviews a month for one developer, or Bugbot on a Cursor Pro plan at a dollar or so per run, both cost less than a CodeRabbit seat.
A team of five to 20 shipping daily: CodeRabbit Essentials at $24 per developer per month is the predictable default. Flat pricing, low noise, all four Git hosts if you are not GitHub-only. If your seeded test shows architectural bugs slipping through, add Greptile as the second reviewer on your riskiest repos rather than replacing anything.
Cursor-native teams: Bugbot. You are paying for the editor anyway, and the per-run cost lands below a second seat at typical PR volumes. Turn on once-per-PR or mention-only on high-churn repos, because the per-run meter is the whole cost model now. If you are still deciding which AI coding setup to standardize on in the first place, our guide to the three AI programming tools developers actually use covers the editor side of that decision.
Security-critical or regulated teams: CodeAnt if you want review and SAST in one pass with on-prem deployment options, or CodeRabbit Advanced if you want continuous security monitoring on top of the reviewer. Greptile's 100% high-severity catch rate in its own test makes it the recall pick when a missed security bug costs more than any subscription. One caveat for the AI-era additions: if your reviewers or review bots touch agents that read external content, prompt injection is the security risk nobody reviews for, and no code reviewer catches it.
High-velocity agentic teams (agents opening PRs): flat seats, always. Greptile's per-review credits and Bugbot's per-run billing both punish exactly the workflow they claim to serve. CodeRabbit's hourly rolling allowance (5 per developer per hour on Essentials) is the metered model that scales best with agent-generated volume.
Run the test before you buy
The entire category is priced low enough to trial everything that fits your stack: free tiers exist for CodeRabbit, Greptile, and Copilot if you already subscribe. The seeded-repo test takes an afternoon. Pick your PRs, plant your 12 bugs, add the clean control group, write the ground truth list first, and run each tool at defaults.
Then apply the only scoring rule that matters: count false positives with the same weight as catches, because your engineers will. A tool that catches 40% with two noise comments per PR will outlive a tool that catches 80% with eleven, in every team that has ever actually used one. The benchmarks on vendor sites will tell you who wins on their dataset. Your seeded repo tells you who wins on yours, which is the only dataset you get paid to care about.
Pricing and plan details are as published by the vendor around September 2026 and can change. Confirm on the official site before you commit.
