Beyond expert users: agents should help users construct preferences, not just elicit them

The Flawed Assumption of the Expert User

A new paper from arXiv (2606.30863v1) challenges a foundational assumption embedded in most AI agent systems: that users arrive with well-formed, stable preferences. The authors argue that agents default to asking clarifying questions when tasks are underspecified, but this approach fails when users lack the domain knowledge to articulate what they actually want. Instead of merely eliciting pre-existing preferences, the paper proposes that agents should actively help users construct their preferences through interaction.

This is a subtle but significant shift. Current systems—from travel planners to code assistants—operate on a model where the user is an expert who simply needs to specify parameters. The agent’s job is to ask the right questions to surface those parameters. But the reality is messier. A user booking a complex business trip may not know the trade-offs between flight times, hotel proximity, and cost. A junior developer may not know which code architecture patterns best suit their project. In both cases, the user’s preferences are not hidden; they are incomplete.

Why This Matters

The paper’s critique has practical weight. When agents treat users as experts, they create friction: endless clarification loops, user frustration, and brittle outputs that fail when preferences are genuinely unknown. This is especially problematic for AI agents moving beyond narrow, well-defined tasks into broader, open-ended assistance.

The proposed solution—preference construction—aligns with research in behavioral economics and decision science showing that people often form preferences through the decision process, not before it. An agent that helps a user explore options, understand trade-offs, and iteratively refine their goals could deliver more satisfying outcomes than one that simply polls for inputs.

For AI practitioners, this means rethinking agent design. Instead of a linear “ask → answer → execute” pipeline, agents need capabilities for: (1) surfacing relevant domain knowledge, (2) presenting comparative options, and (3) allowing preference revision as understanding deepens. This is not about adding more questions, but about making the interaction a collaborative discovery process.

Implications for AI Practitioners

• Interaction design must evolve. Systems should detect when a user’s preferences are likely underdeveloped (e.g., novel task, complex domain) and shift from interrogation to guided exploration.
• Data requirements change. Preference construction requires models that can generate and compare alternatives, not just parse user intent. This may demand different training data and evaluation metrics.
• Trust and transparency become critical. If an agent is shaping user preferences, it must do so without manipulation. Practitioners need guardrails to ensure the agent’s suggestions are informative, not prescriptive.
• Evaluation frameworks need updating. Current benchmarks test how well agents follow explicit instructions. New benchmarks should measure how well agents help users arrive at better decisions when starting from uncertainty.

The paper’s core insight is that the “expert user” is a convenient fiction. Building agents that acknowledge and work around this fiction is not just a nicety—it is a prerequisite for AI systems that truly augment human decision-making.

Key Takeaways

• The paper challenges the assumption that users have well-formed preferences, arguing agents should help construct them rather than just elicit them.
• Preference construction reduces friction from clarification loops and improves outcomes in complex, unfamiliar tasks.
• Practitioners must redesign interactions to support guided exploration, not just intent parsing.
• New evaluation metrics are needed to measure an agent’s ability to improve user decision quality, not just follow instructions.