Coinbase says Qwen3.5-9B beats frontier models in fraud detection

Key takeaways

  • Coinbase says it fine-tuned Qwen3.5-9B, an open-weight model, on proprietary transaction data and found better detection scores and lower serving latency than a leading closed-source model in fraud detection.
  • The exchange's Owning Intelligence series frames the result as evidence for specialized models, not a claim that smaller models are always superior.
  • An earlier Onramp AI risk agent produced 30% fewer fraudulent transactions and 22% less fraud value in an online experiment.
  • Benchmark tests found newer Opus, Sonnet, and GPT versions weaker on several measures than predecessors, with GPT-5.6 improving precision but lowering recall.
  • The case sits within a broader debate about autonomous AI security agents, where Certik has warned about incorrect outputs and reduced human scrutiny.

What happened
In an October 12 post, Coinbase CEO Brian Armstrong said the company fine-tuned Qwen3.5-9B using internal data for fraud prevention. He described the specialized model as outperforming frontier models at much lower cost. The intended lesson was not that smaller models are universally better, but that results depend on the task they are trained to perform.

The claim comes from a three-part technical series, Coinbase's Owning Intelligence. The series first describes adding an AI risk agent to Coinbase's existing machine-learning model and rules engine for Onramp, which lets users buy crypto through partner applications, including guest checkout without a Coinbase account.

In an online experiment, transactions reviewed by the additional AI agent saw 30% fewer fraudulent transactions and 22% less fraud value. The agent examined recent transaction sequences to identify patterns that might not be apparent from a single payment. It could block a transaction that passed initial checks, but it could not reverse an upstream block.

Coinbase then tested newer versions of closed-source models against a fixed benchmark of 16,140 transactions across 7,293 users, including 813 confirmed fraudulent transactions. It reported that newer Opus, Sonnet, and GPT versions performed worse on several fraud-detection measures than their predecessors. GPT-5.6 improved precision but recorded lower recall, meaning it identified a smaller share of fraudulent transactions. That result pushed Coinbase toward an open-weight model whose parameters could be adapted directly, rather than accessed only through a provider's API.

Market context
The exchange's series argues that companies can get better results from specialized models trained for their own needs than by relying only on increasingly capable general-purpose AI systems. The episode reflects a wider pattern in AI deployment: general-purpose models continue to advance, but many operational use cases are narrow enough that task-specific fine-tuning can deliver better accuracy, lower latency, or lower cost.

Fraud detection fits that profile because the relevant signals are tied to a company's own transaction flows and historical patterns. A model trained on proprietary data may detect those patterns better than a generalist model, especially when serving latency matters in checkout flows.

Open-weight models also change the control layer. When a model is accessed only through an API, the provider controls updates and behavior. When parameters can be adapted directly, a company can tune the model to its own data and deployment constraints. That can improve fit, but it also places more responsibility on the user to manage safety, evaluation, and operational limits.

Certik's warning adds context. The security firm said more autonomous security agents could create risks from incorrect outputs and reduced human scrutiny, and recommended clear limits on autonomous decisions and a named owner for every agent. Coinbase's agent works within such limits, returning a risk score and using application codes that apply the decision policy. It can block a transaction that earlier checks allowed, but it cannot override a block that has already been made.

Why it matters
For exchanges and payment platforms, fraud detection is not a single decision made from one transaction. It depends on sequences and patterns that emerge across many events. The result also matters for AI strategy. If newer closed-source versions perform worse on a fixed benchmark, model upgrades are not automatically beneficial for every task. Teams may need to benchmark each version against their own failure cases before switching. Lower recall is particularly important in fraud, because missing fraudulent transactions can translate directly into losses.

The design of the agent is as important as the model choice. Coinbase's approach keeps the AI inside a defined policy boundary: it can add a block after initial checks, but it cannot undo an upstream block. That reduces the risk of an autonomous agent making irreversible decisions without human oversight.

What to watch
The key test will be whether Coinbase can maintain benchmark gains over time as fraudsters adapt, transaction patterns change, and model versions evolve. The October 12 post describes a specialized fraud-prevention task, not a universal model comparison.

Another item to watch is whether the company publishes more details on how it evaluates precision, recall, latency, and cost across model versions. The Owning Intelligence series suggests a practical path for companies that want to use AI for narrow operational risks, but the result still depends on proprietary data, fixed benchmarks, and controlled decision limits.