SkillCoach: Self-Evolving Rubrics for Evaluating and Enhancing Agentic Skill-Use

What Happened

The research introduces SkillCoach, a framework designed to address a critical bottleneck in LLM agent development: reliable skill selection and execution from large, overlapping skill repositories. As agents increasingly rely on reusable "skills"—structured procedures encoding domain rules, tool workflows, and validation logic—the challenge of choosing the correct skill among similar alternatives grows acute. SkillCoach tackles this by implementing self-evolving rubrics that dynamically evaluate and refine an agent’s skill-use behavior, moving beyond static verifiers that cannot adapt to repository drift or nuanced task requirements.

The core innovation lies in the rubric mechanism itself. Rather than relying on fixed scoring criteria, SkillCoach generates and iteratively improves evaluation rubrics based on observed agent performance and task outcomes. This allows the system to capture subtle distinctions between overlapping skills—for example, differentiating a "data cleaning" skill from a "data validation" skill when both involve similar tool calls but serve different purposes. The rubrics evolve through a feedback loop where failed or suboptimal skill selections trigger rubric adjustments, effectively teaching the agent to make better choices over time without manual intervention.

Why It Matters

This work addresses a fundamental tension in agent architecture: the trade-off between skill reusability and selection accuracy. Current approaches often force developers to either maintain small, non-overlapping skill sets (limiting capability) or accept frequent mis-selections in large repositories (degrading reliability). SkillCoach’s self-evolving rubrics offer a third path—one where skill repositories can grow organically while selection accuracy improves autonomously.

For AI practitioners, the implications are significant. First, it reduces the maintenance burden of skill repositories. Instead of manually curating skills to avoid overlap or writing complex selection logic, teams can allow natural repository growth and let the rubric system handle disambiguation. Second, it enables continuous improvement without retraining. As new edge cases emerge in production, the rubrics adapt, making the agent more robust over time. Third, the approach is model-agnostic—it works with any LLM backend, meaning organizations can deploy it without waiting for model updates from providers.

Implications for AI Practitioners

Deploying SkillCoach-like systems will require rethinking evaluation infrastructure. Teams need to instrument their agents to capture not just final outcomes but also skill-selection decisions and their context. The rubrics themselves become a new form of asset—versioned, auditable, and potentially shareable across teams. Practitioners should also anticipate a calibration period where initial rubrics may be overly broad or narrow before converging on effective criteria.

The research also hints at a broader shift: from static evaluation metrics to dynamic, self-improving quality systems. This aligns with the industry trend toward agents that learn from deployment rather than requiring pre-training for every scenario. However, it raises questions about rubric interpretability and safety—if rubrics evolve autonomously, how do we ensure they don’t drift into undesirable behaviors? Responsible deployment will require monitoring rubric evolution and establishing guardrails.

Key Takeaways

• SkillCoach introduces self-evolving rubrics that dynamically evaluate and improve LLM agents’ skill-selection accuracy in overlapping skill repositories, reducing manual curation overhead.
• The framework enables continuous improvement without model retraining, adapting to new edge cases and repository changes in production environments.
• Practitioners should prepare for new evaluation infrastructure requirements, including instrumentation for capturing skill-selection decisions and versioned rubric management.
• The approach signals a shift toward autonomous quality systems, but raises important questions about rubric drift and the need for monitoring guardrails.