The AI Pentester That Reads Your Code.

Share
The AI Pentester That Reads Your Code.

What's the difference between an AI that finds your bugs and one that's just guessing? About forty lines of source code.

I gave one AI the same bugs twice - the only difference was whether it could read the code

In the last post I built the cage: a security agent that runs on a cloud box with no credentials of its own, so whichever model I drop into it can't become the most privileged thing in the lab. That post was about safety. This one is about whether the thing is any good.

Everyone argues about which model is the best pentester. I think that's the wrong argument. New models land every few weeks and they all reason well now. Reasoning got cheap. What didn't get cheap is context, which is how much the agent actually knows about the system it's looking at.

So I ran an experiment to see if that's true or if I'm just telling myself a nice story. Same model, same bugs, run twice. The only thing I changed was whether it could read the source code. The gap was not subtle.

The rig: five sinks, only three are real

I wrote a tiny Flask app with five dangerous-looking sinks. Three are genuinely exploitable. Two are decoys. A decoy is a sink that looks dangerous but isn't actually reachable by an attacker: bait, to see whether the tool can tell a real bug from a false alarm. A plain pattern scanner flags all five and can't tell them apart.

@app.get("/health")
def health():
    # eval() is here, but it only ever sees a constant. NOT attacker-reachable.
    return jsonify(status="ok", check=safe_calc())   # safe_calc() -> eval("1 + 1")

@app.post("/calc")
def calc():
    # REAL: attacker-controlled expr flows straight into eval().
    expr = request.json.get("expr", "")
    return jsonify(result=eval(expr))

@app.post("/ping")
def ping():
    # REAL: host is concatenated into a shell command, shell=True.
    host = request.json.get("host", "")
    out = subprocess.run(f"ping -c 1 {host}", shell=True, capture_output=True, text=True)
    return jsonify(output=out.stdout)

@app.get("/fetch")
def fetch():
    # REAL, and cross-file: user url -> helpers.fetch_url -> requests.get.
    return jsonify(body=fetch_url(request.args.get("url", "")))

@app.post("/convert")
def convert():
    # DECOY: the sink is real, but unit is checked against an allowlist first.
    unit = request.json.get("unit", "")
    if unit not in ALLOWED_UNITS:
        return jsonify(error="unknown unit"), 400
    out = subprocess.run(["units", "--terse", unit], capture_output=True, text=True)
    return jsonify(output=out.stdout)

Look at the two decoys, because they're the trap I set for the model, not a human.

/health has a real eval(). But it only ever evaluates the constant "1 + 1". No request data reaches it. A grep for eval screams, but the code is harmless.

/convert has a real subprocess, reachable from a user. But unit is checked against an allowlist before the sink, and the command uses an argument list with no shell. The guard sits between the input and the danger.

To know those two are safe, you have to read the code, trace where the input comes from, and check whether anything stops it. That's the one thing a black-box tool, poking the app from outside, can never do. So: can an AI do it, and does reading the source actually change the answer?

Same model, twice

The agent works in stages. A dumb pattern-match finds candidate sinks, the model reasons about which are actually reachable, and a sandbox proves the survivors. Stage one is deliberately stupid. It flags all five sinks and holds no opinion about which are real.

Here's what "the reasoning stage" actually means, because it isn't magic. For each candidate sink the harness runs a small tool-use loop. In white-box mode the model gets two tools, read_file and grep, plus a submit_finding tool it has to call once with a structured verdict. It reads the route, traces the parameter back, checks for a guard, and submits {reachable, sanitized, confidence, reasoning}. In black-box mode it gets submit_finding and nothing else: no file access, just the one-line snippet.

# white-box: the model can look. black-box: it can't.
tools = [read_file, grep, submit_finding] if whitebox else [submit_finding]
for _ in range(12):                  # tool-use budget
    msg = model.chat(system, messages, tools)
    if msg.tool == "submit_finding":
        return msg.args              # {reachable, sanitized, confidence, reasoning}
    messages.append(run(msg.tool))   # read_file / grep result, white-box only

It's model-agnostic on purpose: any OpenAI-compatible endpoint, with a fallback chain if one errors or goes quiet. I ran this on one paid model (GPT-6 Sol over OpenRouter), so the only thing moving between the two columns was the context, not the model.

Then the switch. One command runs the reasoning stage twice over the identical five sinks with the identical model. Once white-box, with tools to read the source. Once black-box, handed only the one-line snippet of each sink.

$ python3 hunter/run.py --compare
[stage1] 5 candidate sinks (pattern match, no reachability yet)
          c1  eval        app.py:32       c4  requests    helpers.py:9
          c2  subprocess  app.py:40       c5  eval        helpers.py:14
          c3  subprocess  app.py:60
[stage2] reasoning in WHITE-BOX mode (model=gpt-6-sol)
          c1  eval        -> REACHABLE        conf=1.00
          c2  subprocess  -> REACHABLE        conf=0.99
          c3  subprocess  -> not exploitable  conf=0.99   # /convert: allowlist
          c4  requests    -> REACHABLE        conf=0.99
          c5  eval        -> not exploitable  conf=1.00   # /health: constant

White-box kept the three real bugs and threw out both decoys, naming exactly why each was safe. Then I ran the same model again with the source taken away.

White-box (reads source) Black-box (snippet only)
Real bugs found 3 of 3 0 of 3
Decoys both correctly dropped all 5 called "not exploitable"
Confidence 0.99 to 1.00 0.25 to 0.96
Cost $0.043 $0.012

Same model. Same five sinks. Reading the source took it from finding nothing to finding everything, for five cents.

And here's the part I got wrong going in. I expected the black-box run to be noisy, to cry wolf and flag the decoys as bugs. It didn't. It went quiet. Without the code it called everything safe and missed all three real bugs. Blind doesn't make the model paranoid. It makes it miss things, confidently.

Proof, not opinion

A model saying "this is reachable" is still just an opinion. The next stage doesn't take its word for it. It spins the app up in a throwaway container and fires a harmless probe at each surviving candidate. If a canary comes back, the bug is real.

A canary is a unique, harmless marker you send through the app. If it comes back to you, you've proven the vulnerable code path actually ran, not just that it looked risky. For the eval sink the marker is a piece of arithmetic, for the shell it's an echo of a random token, for the SSRF it's a loopback fetch to a server that logs the token.

PROBES = {
  "eval":       {"send": {"expr": "1337*7331"},                  "ok": r["result"] == 9798047},
  "subprocess": {"send": {"host": "x; echo <token>"},             "ok": token in r["output"]},
  "requests":   {"send": {"url": "http://127.0.0.1:9999/<token>"}, "ok": canary_log_has(token)},
}
[stage3] validating 3 reachable findings in sandbox
  c2  subprocess  -> CONFIRMED: injected echo returned canary CANARYca95eba3
  c4  requests    -> CONFIRMED: in-container canary logged fetch of CANARY3caf489f
  c1  eval        -> unconfirmed: eval executed attacker math: response={"result":9801547}

Two confirmed with canaries. The third line is the honest part, and I'm leaving it in.

The eval RCE worked. The model sent 1337*7331, and the app returned 9801547, the exact product. Attacker-controlled code ran. But my validator marked it unconfirmed, because the expected value hard-coded in my own check was 9798047. A typo. The harness was right that the bug was real and wrong about its own proof, so a genuine remote code execution landed in the "needs review" pile instead of "confirmed."

That's the lesson I didn't plan to learn. The validator is code too, and code has bugs. A sandbox proof is only as good as the check sitting behind it, which is a good thing to find out on my own deliberately-broken app and not on a client's.

The harness earned its keep

I kept notes on what the box did while the models ran, because that's the real product. Three moments stood out, and none of them is the model being clever.

The permissions held when the model wandered. In one run the model went off-script and tried to read a file outside the target folder. It didn't catch itself. The harness's external_directory: deny rule blocked the read, and it showed up in the log. The model's judgment didn't stop it. The box did.

// opencode.json, the recon agent's permissions
"permission": {
  "read": "allow", "grep": "allow", "glob": "allow",
  "bash": "deny", "edit": "deny", "external_directory": "deny"
}

Scope obedience depended on the model, so I don't trust it to the model. A cheap free model happily ignored SCOPE.md and read a file it was never asked about. A stronger model read SCOPE.md, noticed the target wasn't listed, and stopped to ask. If I'd leaned on the model to respect scope, half of them would have wandered. The rule has to live in the harness.

A model failed silently, and the harness caught that too. One of the free models returned nothing: empty output, exit code zero, no error. A naive loop would log that as a pass. Mine treats empty output as a failure and falls back to the next model, because a run that spends your money and proves nothing is worse than one that errors loudly.

That's the box doing its job around whichever model is plugged in. The whole experiment cost under fifty cents.

The model is the easy part. Context is the hard part.

Line up the pieces. My earlier argument was that an AI could reason about an app from the outside, black-box, the way a pentester does. This experiment shows the same reasoning, handed the source code, goes from blind to perfect.

So here's where I've landed. Reasoning used to be the rare, expensive skill. Now every model has it, so it stopped being the thing that sets you apart. What sets you apart is context: how much the agent actually knows about the system in front of it. And the side holding the source code, the defender, the team that owns the app, knows far more about it than any bug-bounty hunter guessing from the outside. One of them is prodding a black box. The other is reading the map.

People keep saying AI replaces security engineers. I think they've got it backwards. The model is the cheap part, you swap it out in one config line. Choosing what context it gets, building the box that checks it, and deciding what counts as proven: that's the work, and none of it is the model.

None of this is a stable benchmark. It's three usable runs of one model on a rigged app I built to make a point. But the point held every time I ran it: same model, same bugs, and the only thing that moved the result was whether it could read the code.

The code

The whole thing is small. The layout:

whitebox-hunter/
├── target_app/
│   ├── app.py            # the five sinks (three real, two decoys)
│   └── helpers.py        # fetch_url (SSRF), safe_calc (the constant eval)
├── hunter/
│   ├── stage1_candidates.py   # pattern match -> candidates.json
│   ├── stage2_reason.py       # the model: reachable?  (read_file, grep, submit_finding)
│   ├── stage3_validate.py     # sandbox probes + canaries
│   └── run.py                 # the pipeline, including --compare
├── rules/dangerous-sinks.yml  # what stage 1 greps for
├── docker/Dockerfile.target   # the throwaway container
└── samples/expected.md        # the ground truth: which sinks are real

app.py is above. The other half of the target, the cross-file sink and the constant-fed decoy, lives in helpers.py:

# helpers.py
import subprocess, requests

ALLOWED_UNITS = {"m", "kg", "s", "a"}

def fetch_url(url):
    return requests.get(url, timeout=3).text   # REAL: SSRF, reached from /fetch

def safe_calc():
    return eval("1 + 1")                        # DECOY: constant, no user input

I'll put the full repo up at github.com/amrayoub/whitebox-hunter