OpenZeppelin Finds Flaws in OpenAI's EVMbench Blockchain Security Benchmark

Blockchain cybersecurity firm OpenZeppelin has conducted an independent audit of OpenAI's newly launched EVMbench benchmark, uncovering significant methodological flaws and data contamination issues that undermine the reliability of AI agent testing results.

«https://t.co/yW00RmRBZQ» — OpenZeppelin (@OpenZeppelin), original post

What Is EVMbench and Why Does It Matter

OpenAI launched EVMbench in mid-February in partnership with investment firm Paradigm. The tool is designed to evaluate how well AI agents can detect, fix, and exploit vulnerabilities in smart contracts. It is built on a curated set of 120 vulnerabilities identified during security audits conducted between 2024 and 2025.

OpenZeppelin welcomed the initiative but decided to subject EVMbench to the same rigorous standards it applies when auditing major DeFi protocols, including Aave, Lido, and Uniswap.

Why This Matters

Benchmarks are foundational to the development of AI-powered blockchain security tools. If the testing dataset contains errors or allows models to recall answers from training data rather than discovering them independently, the results become meaningless. In the DeFi sector, where smart contract exploits routinely cause losses in the hundreds of millions, the integrity of such evaluation tools carries enormous weight.

Training Data Contamination

OpenZeppelin's primary concern centers on data contamination. The leading AI models tested against EVMbench have a knowledge cutoff of August 2025. Since the benchmark's vulnerability set spans 2024–2025, these models likely encountered information about those specific bugs during their training phase.

Even with internet access disabled, the models could have been "remembering" known vulnerabilities rather than genuinely identifying them. This raises a fundamental question: can these AI agents actually detect novel threats, or are they merely retrieving memorized information? The benchmark's current design cannot answer that question.

Factual Errors in the Vulnerability Dataset

Beyond the contamination issue, OpenZeppelin's auditors found concrete factual errors within EVMbench's dataset. At least four vulnerabilities classified as "high risk" turned out to be non-functional — the described attack vectors simply do not work in practice.

Despite this, AI agents received credit for correctly identifying these nonexistent issues. OpenZeppelin emphasized that these are not subjective disagreements about severity ratings but clear cases where the described attack scenario is technically impossible.

OpenZeppelin's Verdict: Standards Must Match Ambitions

Despite the identified shortcomings, OpenZeppelin affirmed that artificial intelligence will play a pivotal role in the future of blockchain security. However, the firm cautioned against rushing deployment at the expense of data quality and testing rigor.

The question is not whether AI will change smart contract security — it will. The question is whether the benchmarks and data we use to build these tools will meet the same standards as the contracts they are meant to protect.

This audit follows a broader trend of scrutinizing AI tools in the security space. In November, Microsoft researchers introduced their own testing environment for AI agents and identified vulnerabilities inherent to modern digital assistants. OpenZeppelin's review of EVMbench adds another layer of critical evaluation as the industry works to integrate AI into blockchain security workflows.