Autonomous Pentesting on Self-Hosted GLM-5.2

We did pentesting running those weights locally — four DGX Sparks sitting on the desk.

Open-weight GLM-5.2 is what people put on the same benchmarks as Claude Sonnet and Opus for agentic coding. Buying the IQ only requires you sell your car — or a kidney, if the dealership already got the car — and budget for 600W of continuous power draw. No API bill. Client tokens never leave the house.

Power and throughput, honestly:

• Power: ~600W continuous under agentic load — not a NAS, not a space heater either.
• Decode: roughly 20–30 tok/s end-to-end for the agent loop — not chat-snappy, fine for hands-off work.
• Prefill shape: early recon is prefill-heavy (~60s mean step). Late API probing is lighter (~20s). Same model; the workload changes the bill.

Cloud GLM works with the same harness if you don't want the Sparks. This post is about a capable model meeting a harness that never defined "done." The first serious run wrote ten evidence-backed findings files and closed zero subtasks — fifteen hours, ~300 live API calls. The model was fine. The harness wasn't. Finishing the job took reshaping the system prompts; that fix is on main at crawlsys/pentagi.

The Stack: Near-Frontier Inference Without Leaving the Building

The weights are GLM-5.2 served as the QuantTrio Int4-Int8Mix quant on a four-node DGX Spark cluster — vLLM (community DCP/MTP recipe below) behind litellm, one OpenAI-compatible URL for the agent.

Quantization, briefly. Full-precision weights for a model this size are an enormous memory footprint — more than four Sparks can hold while still leaving room for context and KV cache. Quantization stores most of those weights in fewer bits (here a mix of 4-bit and 8-bit) so the same parameters fit and move faster in memory, with a managed quality trade. You are packing a large model into the RAM you actually have, which is why the QuantTrio build shows up in every GB10 serving recipe for this class of model. Without that pack step, this class of intelligence stays in a hyperscaler.

Everything above lives on-prem: target, agent orchestration, weights, KV cache, findings. For a pentest that matters more than any leaderboard number. A cloud agent ships your scope out with every tool call — client IPs, hostnames, request/response bodies, and the exploit paths it is testing. Local weights keep that traffic on the desk.

Serving credit where it is due: the path that stayed up under tool-calling load builds on tonyd2wild's QuantTrio / MTP / 4× Spark recipe. We forked it under crawlsys/GLM-5.2-655K-MTP-4x-DGX-Spark with the extra landmines we hit (cutlass-dsl 4.6 bake, profiling-forward skip, parser flags, no-tmpfs weights) in EXTRA-NOTES.md. Tony cracked the hangs; we wrote down the bruises.

What PentAGI Is, and What Broke

If you haven't run it, PentAGI is a popular open-source autonomous-pentest system — high teens of thousands of GitHub stars, covered by Help Net Security as a multi-agent framework that plans and executes assessments in Docker. Community threads treat it as one of the more complete "simulated firm" stacks rather than a single ReAct loop. Popular, useful, praised. "Leading" is a marketing word I am not going to claim for anyone else's project.

Architecturally it is straightforward: a planner breaks an objective into subtasks, an executor and pentester run tools inside sandboxed containers, a monitor periodically checks progress, and a reporter compiles findings. Each role is a prompt template driving whatever model you wire in.

Upstream is pivoting toward a paid product — as they should; the tool is valuable. The open multi-agent pentest space is crowded for good reason. Adjacent projects I am watching (and will likely try and write up next) include PentestGPT, Strix, and CAI (Cybersecurity AI). The prompt-discipline lesson in this post is not PentAGI-specific.

Pointed at GLM-5.2, PentAGI tested beautifully. The model reasoned about attack surface, chained tool calls, and wrote genuinely good findings. What it would not do is conclude. In that fifteen-hour run it kept probing a single vector long past the point of proof, never called the hack_result tool that marks a subtask done, and so never advanced the plan or compiled the report. Ten findings on disk, zero subtasks closed, and a flow that would have run until I killed it.

The reflex diagnosis is "the model's too weak to know when it's finished." That's wrong, and the distinction is the entire point of this post.

The Diagnosis: A Capable Model With No Stopping Rule

GLM-5.2 does exactly what it's told, thoroughly, and it does not self-limit. The prompt templates were written for a model that quietly decides "I have enough, move on." GLM-5.2 never makes that call on its own, so every soft edge in the prompts turned into an infinite grind. Five specific gaps did the damage:

• Buried completion instruction. Closing a subtask depends on calling hack_result, but that instruction sat at the bottom of a long list. The model treated it as optional trivia, not the exit condition.
• Too-soft self-check. The periodic execution monitor asked "should I continue, or ask for help?" Given a genuine choice, a thorough model always picks continue.
• No hard turn terminator. Even when hack_result did fire, it isn't a hard stop, so the agent kept emitting commands right past its own verdict.
• Fuzzy subtask boundaries. The planner sometimes bundled several attack vectors into one broad subtask with no clear finish line, which invited endless over-grinding.
• No evidence-sufficiency rule. With nothing telling it when proof was enough, it over-enumerated — at one point guessing more than forty portal tokens long after the pattern was obvious and one working example already settled the question.

None of these is a model deficiency. Every one is a missing instruction. The harness was written against an assumption the model doesn't satisfy.

The Fix: Curate the Prompts to Stop and Decide

One rule for every change: test until you can make the CONFIRMED / NOT-CONFIRMED decision, then stop. Decisiveness, not exhaustiveness. State the stopping rule instead of assuming the model carries it. That turned into four edits in backend/pkg/templates/prompts/:

• generator / refiner — atomic subtasks. One vector per subtask. Never combine, never emit an umbrella "do X across all vectors" task. Broad subtasks create fuzzy finish lines; atomic ones have obvious ones. (generator.tmpl)
• task-assignment wrapper — close-out, yield, and sufficiency. Test the vector, write the finding, call hack_result, then stop. Plus an evidence rule: one working proof-of-concept is CONFIRMED; a blocked primary path is NOT-CONFIRMED; stop enumerating once you can make that call. (task_assignment_wrapper.tmpl)
• execution monitor — closure bias. The periodic self-check now pushes decisively toward calling hack_result and terminating, rather than offering "keep going" as an equal option. (question_execution_monitor.tmpl)
• pentester — report-first. When findings exist and no report does, compile the report before running any more tests. (pentester.tmpl)

Write it down. Literally — the diffs are the product: FORK-NOTES.md on main.

The Payoff

Same model, same cluster, curated prompts, clean start. This time the flow closed subtasks, advanced through every vector in the plan, and reached its report. Given distinct multi-class auth tokens (admin, limited, multi-org), it confirmed issues the first run never proved: unauthenticated PII via long-lived share tokens, a missing rate limit and permission gate on an MCP-style endpoint, and a latent stored-XSS path that stayed NOT-CONFIRMED as a primary exploit but was real code-level risk. Intelligence was never the missing piece. Permission to stop and decide was.

Favorite ops detail: mid-engagement I had to move the Sparks. Full tilt they run hot enough that you want gloves if you're carrying one more than half a minute, and my teenage son had a fair point that parking four of them in his closet to spread household power load was a bad roommate move. Working data lived on a persistent volume, so nothing was lost — once GLM-5.2 came back up the flow auto-resumed. You can pick the inference cluster up mid-pentest and put it somewhere with better airflow. Try that with a cloud dependency.

The Bigger Point

Near-frontier inference at home is no longer a demo. Serving a model people compare to Sonnet and Opus off four GB10s would have read as fantasy a year ago. The bottleneck moved off the silicon and into the harness. The limits live in the orchestration prompts, not the wafer fab.

If you are building agentic systems on capable local models — or on cloud GLM with the same harness — and the agent will not finish, do not reach for a bigger model first. Check whether you ever told it what "done" looks like. A model that does exactly what you ask will grind forever if you never wrote the stopping condition down. Write it down.

One caveat: upstream PentAGI has slowed down — last commit about six weeks before the fork — which is part of why forking made sense, especially as they move toward paid offerings. And one of these fixes really belongs in the source, not the prompt layer. Making hack_result a hard turn-terminator is a change to the tool-execution loop; papering over it with prompt language works, but the durable fix lives in code. That one I'd like to send upstream if the project picks back up.

Curated prompts: crawlsys/pentagi main — star it if it saves you a fifteen-hour non-closing run. Serving recipe (Tony's base + the landmines we hit): crawlsys/GLM-5.2-655K-MTP-4x-DGX-Spark. I am on GitHub, X, and LinkedIn if you want the follow-ups when I run Strix / CAI / PentestGPT through the same stop-and-decide lens.

Related Posts

This is the second time PentAGI has shown up here on local hardware. Turning Honeypot Noise into PentAGI Investigations wires an FTP honeypot into bounded PentAGI flows backed by a smaller Qwen3-Coder model on a single RTX 5090 — the low-cost, high-volume end of the same idea. Stop Burning Frontier Tokens on Routine Inference covers the general principle: keep the routine work on hardware you own, and spend real money only on the part of the job that needs it.

---