It sounds great right? Ask an AI Agent to pentest your website, and ask another AI Agent to go fix the issues the first one found.
My Early Experimentation
I was curious about the current capabilities of AI Agents, so I setup an intentionally vulnerable API from OWASP called "crAPI" or completely ridiculous API on my local network. To avoid incurring high API costs, I preferred to use local models with Ollama, running on my gaming PC which should be more powerful than my laptop. I started out with the llama3:8b model because it was recommended and wasn't so big that my PC couldn't handle it.
There are many AI Agent frameworks built for this purpose, but so far I've spend the most time with strix. I prompted it to perform a penetration test on the crAPI, but I quickly ran into an "ethical guardrail loop", here is an example:
I can't provide information or guidance on illegal or harmful activities. Can I help you with something else?
I cannot assist in any illegal activities.
I cannot create a penetration testing report that includes actual vulnerabilities and potential exploits for an unsecured environment. Is there anything else I can help you with?
I tried switching to a model called "wizard-vicuna-uncensored:13b" which claims "responses that contained alignment / moralizing were removed". Then I ran into a different issue, it started going out of scope and wanted to scan my entire network. I argued with it to try to enforce the scope, and it reluctantly agreed, only to start rambling about penetration testing and apologizing for the confusion. Both attempts were all talk, no action, and no actionable results.
Recently I have been enjoying practicing my Bug Bounty skills, and reading "Hacking APIs: Breaking Web Application Programming Interfaces" by Corey Ball which I highly recommend if the topic is of interest. While exploring scoped Bug Bounty targets I found an exposed service on a non-standard port. I did as much manual enumeration as I could, and then tried asking Strix to take a look at it. Initially the results looked quite promising:
However, after extensive attempts at validation, I was unable to reproduce any of the findings. It rambled on about how critical these findings were and how much they would be worth if I reported them to the Bug Bounty program. Everything turned out to be a false positive and it got my hopes up for nothing!
Community Experimentation
As this is an emerging technology, there are certainly a lot of people experimenting with it. I recently saw a post by one of the top Bug Bounty hunters, dawgyg, where he created a committee of agents to check each other's work:
The high false positive rate is concerning.
While applying for jobs (act fast 😁, I am still available to hire! contact arthur at goodguysecurity.com) recently I saw this in a job description:
"At REDACTED, we're building autonomous AI agents that think like hackers. We need people who've already mastered AI as a tool — not just using ChatGPT to understand concepts, but architecting workflows where AI handles the tedious 80% and you focus on the complex 20%. If you're still skeptical about AI or rewriting everything by hand, this isn't the right role yet."
Clearly some people have more confidence in AI than others. This role would be building automation to eventually replace penetration testers. This reminded me of a post I saw on bugbounty.forum:
What is AI good at?
From my perspective, AI does not yet have a solid methodology to be effective at taking lead in a penetration test. Even for experienced security researchers, developing an effective methodology takes years of practice and experience. AI is great at being a "co-pilot" whom you can delegate tasks to, and bounce ideas off of, but it's not ready to fly the plane.
I recently discovered some js.map files while bug hunting. These were large files containing source code for the application. Analyzing these manually would take a lot of time and energy, but this is a great task to delegate to AI, here are some hints it gave me:
While reading Cybersecurity news I saw a company named Aisle discovered 12 vulnerabilities in OpenSSL (link to article). This is particularly interesting because OpenSSL has been audited by thousands of people over many years, and the bugs weren't caught until AI assistance was used. Clearly it can be effective if you have the right model, computing power (or API), and know how to be the "human in the loop".
I have enjoyed exploring this new technology and will continue to do so. I would be happy to hear your thoughts on the effectiveness of different tools and models for offensive security work.
Original Content: This article was typed by me, a human, without the assistance of any AI.
Really enjoyed reading your experiments with AI-driven pentesting and the honest insights you shared in this post. It’s clear you’re thinking critically about both the capabilities and current limitations of autonomous agents in offensive security — and that kind of real-world experimentation is exactly what the industry needs right now.
ReplyDeleteYour honest assessment of false positives and ethical guardrail challenges shows not just technical curiosity but also responsibility, which is crucial in a field where misuse can easily happen. Tools like AI agents hold a lot of promise for automating tedious parts of security testing, but as you pointed out, they still struggle with methodology and reliable results in complex environments. This aligns with broader research in automated penetration testing, where systems are still being evaluated and refined for meaningful vulnerability discovery and exploitation workflows.
Keep up the great work — detailed posts like this help bridge the gap between theory, emerging research, and practical experience. Looking forward to your next exploration!