TRAP: Benchmark for Task-completion and Resistance to Active Privacy-extraction

The Privacy Paradox in AI Agents: Why TRAP Matters

A new benchmark called TRAP (Task-completion and Resistance to Active Privacy-extraction) has been released on arXiv, addressing a critical blind spot in current AI safety evaluations. Unlike existing benchmarks that test models in sanitized environments where private data is clearly marked and separated from task objectives, TRAP simulates realistic document-heavy workflows—such as booking flights with passport numbers or processing medical records—where sensitive information is integral to the task itself.

This distinction is crucial. Most privacy benchmarks evaluate whether models can avoid leaking data when explicitly told not to, or whether they resist adversarial prompts in isolation. TRAP instead measures the tension between two competing requirements: completing a task that requires private data, and resisting attempts to extract that same data. The benchmark creates scenarios where an agent must use passport numbers, financial details, or health information to fulfill a user’s request, while simultaneously facing active extraction attempts from malicious actors embedded in the workflow.

Why This Matters for Deployed Agents

The timing of TRAP is significant. Enterprises are rapidly deploying AI agents into customer service, HR, and healthcare workflows—precisely the domains where private data is routine rather than exceptional. Current safety measures often fail in these contexts because they treat privacy as a binary state (data is either public or private) rather than a contextual requirement. An agent that refuses to process a passport number cannot book a flight; an agent that processes it without safeguards can be tricked into leaking it.

TRAP exposes a fundamental design flaw in how we train and evaluate agents. Most models are optimized for helpfulness (task completion) or harmlessness (privacy protection) as separate objectives, but real-world deployment requires balancing both simultaneously. The benchmark reveals that many state-of-the-art agents either over-refuse (failing tasks that require private data) or under-protect (completing tasks but leaking data under extraction pressure).

Implications for AI Practitioners

For teams building production agents, TRAP offers both a warning and a methodology. First, it demonstrates that standard red-teaming or adversarial testing is insufficient—agents need evaluation in task-integrated privacy scenarios where extraction attempts are woven into legitimate workflows. Second, it suggests that privacy safeguards must be context-aware: an agent should recognize when private data is necessary for a task versus when it is being extracted.

Practitioners should consider implementing differential privacy techniques at the agent level, not just the model level, and building explicit data lifecycle management into agent architectures—tracking what private data has been accessed, why, and ensuring it is not retained or transmitted beyond the immediate task. TRAP also highlights the need for human-in-the-loop verification for high-stakes private data operations, at least until agent privacy reasoning matures.

Key Takeaways

• TRAP fills a critical gap by evaluating agents on the simultaneous demands of task completion and privacy protection in realistic, data-intensive workflows
• Current AI safety benchmarks fail to capture the privacy paradox where agents must use sensitive data to function, creating new attack surfaces
• Many existing agents either over-refuse legitimate tasks or under-protect against extraction, indicating a need for context-aware privacy reasoning
• Practitioners should adopt task-integrated privacy evaluations and implement data lifecycle controls rather than relying on static privacy filters