UCOB: Learning to Utilize and Evolve Agentic Skills via Credit-Aware On-Policy Bidirectional Self-Distillation

The Credit Assignment Problem in Agentic Skill Memories

A new paper from arXiv (2606.29502) tackles a fundamental weakness in reinforcement learning agents that use skill memories: retrieved skills are not uniformly beneficial. The proposed method, UCOB (Utilize and COntrol Bidirectional), introduces credit-aware on-policy bidirectional self-distillation to address the fact that a skill that works in one context can actively mislead an agent in another.

What Happened

The researchers identify a critical oversight in current skill-memory architectures. Existing systems treat retrieved skills as uniformly helpful, but in practice, a skill’s utility is highly state-dependent. A navigation skill that works in an open field may fail catastrophically in a corridor. UCOB’s innovation is a bidirectional self-distillation framework that evaluates skills on-policy—meaning it judges a skill’s actual contribution while the agent is acting, not just based on past performance. The “credit-aware” component assigns differential importance to skill segments based on their demonstrated utility in the current state, allowing the agent to selectively amplify helpful patterns and suppress harmful ones.

Why It Matters

This addresses a persistent blind spot in agentic systems. The industry has focused heavily on building larger skill libraries and better retrieval mechanisms, but has largely ignored the reliability of retrieved skills. For AI practitioners deploying agents in production—whether for robotics, automated customer service, or code generation—this is a practical bottleneck. An agent that occasionally executes a counterproductive skill is not just inefficient; it can be dangerous. UCOB’s on-policy evaluation is particularly significant because it moves beyond static skill rankings to dynamic, context-aware filtering. This aligns with a broader trend in reinforcement learning toward online credit assignment, where feedback loops are tightened to prevent learned behaviors from drifting into failure modes.

Implications for AI Practitioners

For teams building agentic systems, UCOB suggests several actionable shifts. First, skill memory architectures should include a utility gate that evaluates each retrieved skill against the current policy’s trajectory, not just historical success rates. Second, the bidirectional distillation approach implies that agents should be trained to both utilize and evolve skills simultaneously—meaning the skill library itself should be updated based on real-time performance, not frozen after initial training. Third, practitioners should expect that as skill libraries grow, the credit assignment problem will worsen, making techniques like UCOB increasingly necessary rather than optional.

The paper also implicitly warns against over-reliance on pretrained skill embeddings. Static representations cannot capture the dynamic interplay between a skill and a policy’s current state. For production systems, this means investing in lightweight on-policy evaluators that run alongside the main agent, rather than assuming retrieved skills are safe by default.

Key Takeaways

• Skill memories in agentic RL suffer from a state-dependent utility problem: the same skill can be helpful or harmful depending on context, and current systems do not account for this.
• UCOB introduces on-policy bidirectional self-distillation with credit awareness, enabling agents to dynamically filter and weight skills based on real-time utility.
• Practitioners should implement utility gates in skill memory architectures and treat skill libraries as evolving, not static, to maintain reliability as agentic systems scale.
• The paper highlights a growing need for online credit assignment mechanisms in production agentic systems, particularly as skill libraries grow in size and complexity.