Botmonster TechAlibaba's code reviewer finds fewer bugs on purpose
Alibaba Open Code Review is a command-line reviewer that finds fewer bugs than a general agent on purpose. Its own benchmark chart, published only as an image, puts the best run at 20% recall against 1,505 known defects. You get far fewer false alarms in exchange for far more misses.
Key Takeaways
- The tool splits the job: code picks the files, the model finds the bugs.
- Its best benchmark run catches one real defect in five, and the project says so.
- The numbers only exist inside a chart image, so you can’t check them.
- An outside tester got the opposite result: strong recall, weak precision.
- Delegation mode runs the review on your own coding agent with no API key.
What Alibaba Open Code Review does differently
Most AI review tools are a prompt wrapped around a diff. Open Code Review splits the job in two and gives each half to whichever component handles it better.
The fixed half is ordinary Go code. It picks exactly which files go in and which get filtered out. It then bundles related files into one review unit and matches rules to each file through a template engine. Separate positioning and reflection passes clean up the model’s output afterwards.
The agent half does what only a model can do. It reads a file in full, searches the codebase, pulls in other changed files for context, and decides which lines look broken.
File bundling is more important than it sounds. A set of parallel translation files goes into a single review unit that runs as its own sub-agent with isolated context. That is what keeps output stable on a huge changeset, and it makes concurrent review natural.
The project aims all of this at three failures it says general agents have. On large changesets they cut corners and skip files without saying so. Reported issues land at line numbers that have drifted off target. Quality swings whenever a prompt changes slightly.
Comments here land on specific lines rather than in a summary paragraph, which is the difference between a review you can act on and one you skim.
The ocr review command works over a diff, while ocr scan reads whole files when you are auditing a directory with no meaningful diff.
The rule set started out in Chinese
The built-in rules ship as one markdown file per language, and the collection now runs to about 30 files. Go, Python, Rust, Java, C, C++, Kotlin, PHP, TypeScript and Terraform all get their own, alongside config formats like pom.xml and package.json. So the worry that this only works on Java-heavy code no longer holds.
At launch, though, every one of those files was written in Chinese. Readers on Hacker News had to machine-translate them to see what the tool was checking for. pramodbiligiri published a translated copy of the rules so others could skip that step. The project has since merged English versions into the repo.
What the benchmark chart actually says
The published benchmark is real work. It covers 50 popular open-source repositories, 200 real pull requests, and 10 programming languages. Behind it sit 1,505 annotated ground-truth issues, cross-validated by more than 80 senior engineers. The summary line claims higher precision and F1 than a general agent on the same model, at roughly a ninth of the tokens. Lower recall is named as the deliberate cost.
Those figures live in one place, imgs/benchmark-en.png, and here is what the chart shows for the two best-scoring setups, both running Claude 4.6 Opus:
| Reviewer | F1 | Precision | Recall | Avg tokens | Avg time |
|---|---|---|---|---|---|
| Open Code Review | 25.10% | 33.90% (301/889) | 20.00% (301/1505) | 385K | 1m23s |
| Claude Code | 11.57% | 7.23% (435/5980) | 28.90% (435/1505) | 5,664K | 13m6s |
Token use here is nearly 15 to 1. Across the seven model pairings in the chart the gap runs from about 5x to 14x, so a ninth is a fair average.
Precision of 33.9% means two out of every three flagged issues are still not real defects. Recall of 20% means four out of every five known defects go unmentioned. The tool wins this comparison by a wide margin while being wrong most of the time.
To surface 134 extra real issues, the general agent made you read 5,545 false alarms instead of 588. One Hacker News commenter put the flip side of that well.
If I flag every line in your PR as a potential security bug then I have 100% recall.
The run used Open Code Review v1.3.1 against Claude Code v2.1.169. The tool now ships v1.8.1, so the numbers describe an older build. More awkwardly, an image is not a checkable result. The repo ships no table, no raw output, and no harness anyone can rerun.
Credit where it is due, though. Naming the repositories, the pull request count, and the annotator pool is more disclosure than most vendors in this space offer. Nobody forced them to publish the recall weakness at all.
An independent run flipped the result
Someone did try to reproduce it. The launch Hacker News thread drew 284 points and 73 comments. In it, eranation ran Open Code Review against the independent Martian code review benchmark on 10 of its 50 pull requests.
The result came out backwards. That run measured about 74% recall against about 12% precision, for an F1 near 20%, so a tool built to trade recall for precision behaved like a noisy high-recall reviewer.
The project lead replied in the same thread.
We noticed that in the version tested, there was an anomaly in a critical tool call that significantly impacted the overall performance, particularly contributing to the high false positive rate you observed. We were able to reproduce the issue on the benchmark and have since fixed it.
So the honest status is a standoff. The vendor chart and the one outside attempt disagree, a known bug explains part of the gap, and nobody has published a clean rerun since. For scale, another commenter ran their own review app across all 50 pull requests in that benchmark. It scored 43.0% precision against 35.8% recall, at a cost of roughly $7. Every tool in this category is missing most of the defects.
Delegation mode is the cheapest way to try it
In default mode the tool calls its own configured LLM, which means setting up a provider with an API key and paying for the tokens. In delegation mode your coding agent runs the review itself and the tool only handles file selection and rule resolution. No provider setup and no separate key are needed.
Run ocr delegate preview to see what would be reviewed, and ocr delegate rule src/main.go to get the resolved rules for specific files.
This is the mode worth trying first if you already pay for one of the big CLI agents . The deterministic scaffolding is exactly the part a hand-rolled review skill does badly, and delegation mode hands it to you for nothing.
Integrations ship per platform. You get slash commands for Claude Code, callable skills for Codex , portable skills for Cursor , and native tools for OpenCode . Anything else that reads the skill format can load a portable agent skill instead. Those four handle rules and context quite differently. CI paths are documented for GitHub Actions, GitLab CI, GitFlic CI, and Gerrit.
An MCP server hands the review agent external tools, an OpenTelemetry hook exports traces, and a session viewer replays past reviews in a browser. That last one earns its keep when you are trying to work out why something got flagged.
How to review a branch before you open the pull request
Install the CLI, point it at a model, and get line-level comments on a branch diff in a few minutes. Every flag below comes from the CLI reference .
Check your Git version
Run git --version and confirm 2.41 or newer. The tool leans on Git for diff generation, code search, and repository operations, so older versions won’t do.
Install the CLI
Run npm install -g @alibaba-group/open-code-review. The ocr command becomes available globally. Binary releases and a from-source build are documented as alternatives.
Pick a provider
Run ocr config provider and choose a built-in provider or add a custom endpoint. The prompts walk you through the API key, then test connectivity for you.
Pick a model
Run ocr config model to choose a model on the active provider. Anything OpenAI-compatible or Anthropic-compatible works.
Review your working tree
Run ocr review inside the project. It covers staged, unstaged, and untracked changes, and reports comments against specific lines.
Review a branch instead
Run ocr review --from main --to your-branch for the diff you are about to propose. Add --commit <sha> when you want a single commit.
Scan a directory with no diff
Run ocr scan --path internal/agent when you are auditing unfamiliar code rather than reviewing a change. This reads whole files instead of a diff.
Resume a long review
Run ocr session list, then re-run the same review with --resume <session-id>. An interrupted pass over a large changeset then picks up where it stopped.
Where an AI reviewer belongs in a real workflow
The trade-off dictates the position. Put this on the author’s own machine as a pre-push or pre-PR pass. The goal there is catching the obvious before a human is asked to look, and high precision is the right property for that job. A reviewer that cries wolf gets muted within a week.
The second-best position is an advisory comment on the pull request that a human reads beside the diff. Advisory means nobody gets to say “the bot approved it.”
A required merge gate with no human behind it is the wrong position. A tool that trades recall away by design will let real defects through, and the project says so in its own summary.
Tune the rules before you judge the noise. Rules match to files by path and characteristics, so a repo full of generated code, vendored dependencies, or migration files should exclude them first. Otherwise the score you get back describes your own configuration.
One commenter in the same thread framed the underlying choice sharply.
Finding problems is optimizing for the customer. Avoiding false positives is optimizing for the developer. Which is right depends on your org’s culture.
The useful comparison runs tool-then-human against human alone, and everything comes down to whether the human’s attention lands on better lines. Measure that in a pilot by counting the flagged issues you actually fixed against the ones you dismissed, plus the defects that shipped without the reviewer mentioning them.