SABER: Benchmarking Operational Safety of LLM Coding Agents in Stateful Project Workspaces
Abstract
Large language models deployed as coding agents exhibit significant safety violations in realistic project environments, necessitating new evaluation approaches beyond simple prompt refusal assessments.
Large language models are increasingly deployed as coding agents, shifting safety from individual responses to action sequences. Existing benchmarks, however, primarily assess whether models refuse unsafe prompts, leaving impacts on stateful workspaces largely unexamined. We present SABER, a benchmark for environment-aware operational safety that places models in realistic agent-style projects and evaluates safety from the final environment state after a sequence of actions. Beyond binary safety-violation reports, SABER categorizes violations by cause, enabling analysis of model-specific safety profiles. Our evaluations show that even the best-performing model has more than a 54% harmful safety-violation rate (HSR), suggesting that current alignment remains insufficient for realistic project environments. SABER further reveals distinct safety profiles across models. Our benchmark is publicly available at https://github.com/sssr-lab/saber.
Community
SABER shifts coding-agent safety evaluation from single-turn refusal behavior to the final state of a realistic, stateful workspace after multi-step agent actions. This is an important benchmark direction because many safety failures in coding agents emerge operationally, through file edits, commands, and environment changes, rather than in isolated model responses. The reported >54% harmful safety-violation rate, even for the best model, is a strong signal that current alignment is still not sufficient for realistic project settings.
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
- SeClaw: Spec-Driven Security Task Synthesis for Evaluating Autonomous Agents (2026)
- Taxonomy and Consistency Analysis of Safety Benchmarks for AI Agents (2026)
- SkillSafetyBench: Evaluating Agent Safety under Skill-Facing Attack Surfaces (2026)
- When Agents Overtrust Environmental Evidence: An Extensible Agentic Framework for Benchmarking Evidence-Grounding Defects in LLM Agents (2026)
- Measuring Safety Alignment Effects in Autonomous Security Agents (2026)
- Benchmarking Autonomous Agents against Temporal, Spatial, and Semantic Evasions (2026)
- Red-Teaming Agent Execution Contexts: Open-World Security Evaluation on OpenClaw (2026)
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on Hugging Face checkout this Space
You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: @librarian-bot recommend
Moving the safety signal from refusal to the final workspace state is the right reframe — operational harm in coding agents lives in the action sequence, not the individual response. One boundary worth naming about what the final-state view can certify.
Reading safety off the end state conflates "did harm happen" with "is the end state bad," and for stateful workspaces those come apart exactly where it matters: irreversible actions. An agent that runs a destructive command and then repairs the file ends in a clean state, but the force-push already landed, the secret was already sent, the row was already dropped — the residue is clean, the harmful action still happened. The mirror case slips through too: a final state can look harmed when every step was within what was actually authorized. Operational safety is really a property of each action at the moment it executes, not of what's left at the end.
The other half is who computes the "safe final state" and the cause categories. If that verdict comes from a model or heuristic, the benchmark inherits that judge's blind spots — the same problem one level up. The signal that's hard to fake sits at the action boundary: gate each irreversible action on an attributable grant (who × what × which resource), fail-closed, and you get per-action ground truth that doesn't depend on reconstructing intent from the end state. Final-state HSR is the right cross-model outcome measure; pairing it with an execution-time authorization check is what catches the recovered-but-unauthorized action and the irreversible-already-done case the end state can't see.
Get this paper in your agent:
hf papers read 2606.01317 Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash Models citing this paper 0
No model linking this paper
Datasets citing this paper 1
Spaces citing this paper 0
No Space linking this paper
Collections including this paper 0
No Collection including this paper