Google's Gemini 4 Carbon halves hallucinations but lags behind Anthropic's Opus in coding tasks
Independent testing shows Gemini 4 Carbon reduces false-claim rates from 36% to 18%, yet its coding performance, speed, and cost remain inferior to competing models, with only a narrow security advantage.
Independent reviewer Theo Browne reported that Google’s Gemini 4 Carbon model cuts the false-claim rate in code analysis from 36% for the prior Gemini model (Argan) to 18%, matching internal DeepMind measurements. Despite the improvement, competing models such as Anthropic’s Opus typically fabricate only 0–1% of claims, leaving Carbon still far behind in reliability.
In a Rust architecture review, Carbon generated 79 claims, 23 of which were false, while Argan produced 36 claims with 24 false. Across a broader test set, Carbon failed 23 of 86 claims versus Argan’s 47 of 92. Browne noted that many hallucinations stem from invented column names, line numbers, and architectural details that are not present in the codebase.
Browne also criticized the “fast model” label, pointing out that Carbon’s average task times were 22–27 minutes, slower than Opus, which ranged from about three minutes to 25 minutes depending on reasoning effort. He described the user experience as “genuinely obnoxious” and “miserable to use.”
Pricing is another concern. Carbon is offered at $2 per million input tokens and $10 per million output tokens until November 30, 2026, after which prices double. Browne argues that token price is less relevant than cost per task, especially given the model’s high token consumption and unpredictable caching behavior, which can increase request volume by up to 50%.
The review highlighted a single security scenario where Carbon outperformed rivals. When addressing a vulnerability report, Carbon correctly identified the issue and proposed a practical fix, earning high scores from other AI models. Browne called the result “insane” but cautioned that his early-access status might have influenced the outcome.
Browne praised DeepMind’s responsiveness, noting rapid fixes to integration bugs and policy clarifications. However, he concluded that Carbon is not suitable as a default coding assistant, stating, "> Would I ever pick this model as my default for coding? No. Absolutely not. It's just Opus is unbeatable still."
Overall, while Gemini 4 Carbon halves hallucination rates and shows promise in security-focused tasks, its slower performance, higher effective cost, and lingering inaccuracies keep it behind leading competitors for general coding work.