I built a vulnerable app and spent $1,500 seeing if LLMs could hack it
Developer Kasra Rahjerdi built a deliberately vulnerable web app and spent $1,500 testing 12+ LLMs against it. GPT 5.5 was the best performer (7/10 solves). Deepseek V4 Pro was the value champion (3/10 at $0.19/run). Claude Opus got close but was stopped by safety training (late refusals). Gemini models refused entirely on security grounds. The experiment reveals how safety training affects security capabilities differently across models.

I Built a Vulnerable App and Spent $1,500 Seeing if LLMs Could Hack It
Developer Kasra Rahjerdi conducted an unusual experiment: he built a deliberately vulnerable web application and spent roughly $1,500 testing whether various large language models could successfully hack it. The results offer a fascinating glimpse into the current state of AI-powered cybersecurity capabilities.
The test application was a fake React Native (Expo) book review app with a Python (FastAPI) backend, using Firebase (Firestore) as the data layer. The API was hardened, but Firebase was intentionally left wide open — a broken access control vulnerability that mirrors real-world security flaws the author has found in production applications. The goal was to find a flag hidden in a user's private reviews by exploiting Firebase directly.
The Contenders
Rahjerdi ran 10 attempts per model across a wide range of LLMs. The clear winner was GPT 5.5, which solved the challenge 7 out of 10 times at an average cost of $6.62 per run. Deepseek V4 Pro solved it 3 out of 10 times but was by far the cheapest at just $0.19 per run ($0.62 per solve). Both Claude Sonnet 4.6 and Claude Opus 4.8 managed 2 out of 10 solves each, at significantly higher costs.
Several models failed completely. Deepseek V4 Flash scored 0 out of 10 ($0.08/run). Gemini models (3.1 Pro Preview and 3.5 Flash) scored 0 out of 10, with the preview model immediately refusing on security grounds. MiniMax M2.7 focused on the API and never directly attacked Firebase. Step 3.7 Flash hallucinated exploits that didn't work.
Fascinating Behavioral Differences
The most interesting findings were behavioral rather than technical. GPT 5.5 almost always targeted Firebase directly after unzipping the APK, demonstrating the right mental model for the vulnerability class. Chinese models like Deepseek and GLM were more comfortable attacking the database directly, while other models showed hesitation about affecting live databases — even in a test environment.
Claude Opus got close to solving the challenge multiple times but was stopped by its own security guardrails, which would trigger late in the process after the model had already done substantial work. This "late refusal" pattern is particularly notable — the model knows how to hack but is prevented from completing the task by its safety training.
Implications
The experiment demonstrates that frontier LLMs are increasingly capable of performing real security testing, with GPT 5.5 already at a level where it can reliably identify and exploit common web application vulnerabilities. The behavioral differences between models also highlight how safety training affects security capabilities in unexpected ways — sometimes stopping exploitation, sometimes just making it more expensive.
The author notes this was not a scientific evaluation, but the results clearly show that AI-powered security testing is rapidly becoming practical, at least for well-understood vulnerability classes.


