Digitizing Coaching Intelligence: An Agentic Framework for Holistic Athlete Profiling using VLM and RAG

What Happened

Researchers have proposed a novel agentic framework that combines Vision-Language Models (VLMs) with Retrieval-Augmented Generation (RAG) to automate and enhance athlete profiling. The system digitizes what has traditionally been a manual, observation-based process—coaches watching athletes and making subjective judgments about physical potential, technical skill, and tactical awareness. By integrating visual analysis from video footage with structured retrieval of historical performance data, the framework aims to produce holistic, reproducible athlete assessments at scale.

The architecture employs a multi-agent design: one agent processes visual input (e.g., running form, agility drills) via a VLM, while another retrieves relevant context from a knowledge base of past assessments, normative benchmarks, and coaching notes. A reasoning agent then synthesizes these inputs into a structured profile. This mirrors how expert coaches mentally combine what they see with what they know from experience—but does so programmatically.

Why It Matters

This work addresses a genuine bottleneck in sports talent identification. Mass recruitment drives—common in soccer academies, military physical tests, or Olympic development programs—currently require dozens of human evaluators, each applying slightly different criteria. The result is inconsistency, scalability limits, and missed talent due to evaluator fatigue or bias.

The framework’s significance extends beyond sports. Any domain requiring holistic human assessment under time constraints—such as physical therapy triage, dance auditions, or ergonomic workplace evaluations—faces the same fundamental challenge: combining visual observation with contextual knowledge in a repeatable way. The agentic VLM+RAG pattern offers a template for solving this class of problems.

Crucially, the approach does not claim to replace human coaches. Instead, it augments them by providing standardized, data-backed profiles that highlight discrepancies between observed performance and historical norms. This shifts the coach’s role from subjective rater to informed decision-maker.

Implications for AI Practitioners

For those building production AI systems, this paper reinforces several practical lessons:

First, multi-agent architectures are maturing. The separation of visual perception (VLM), knowledge retrieval (RAG), and reasoning into distinct agents allows independent optimization and debugging. Practitioners should consider this modular pattern for any task requiring heterogeneous data sources.

Second, domain-specific RAG matters. Generic RAG pipelines fail here because athlete profiling requires nuanced retrieval—not just facts, but comparative benchmarks (e.g., “how does this 40-yard dash compare to age-matched elite prospects?”). Building custom embedding strategies and retrieval filters for domain-specific ontologies is a non-trivial but necessary engineering investment.

Third, evaluation remains the hardest problem. How do you measure whether an AI-generated athlete profile is “correct”? The researchers likely rely on coach agreement studies, but practitioners should anticipate that ground truth in subjective assessment tasks is inherently noisy. Designing evaluation protocols that account for inter-rater variability is as important as the model itself.

Finally, latency and cost constraints are real. VLMs are computationally expensive, and running them on live video feeds during mass screenings requires careful batching or edge deployment strategies.

Key Takeaways

• A multi-agent VLM+RAG framework can automate holistic athlete profiling, replacing subjective manual observation with reproducible, scalable assessments.
• The architecture is transferable to any domain requiring combined visual and contextual human evaluation, from physical therapy to talent scouting.
• AI practitioners should invest in domain-specific RAG pipelines and modular agent designs to handle heterogeneous data sources effectively.
• Evaluating subjective assessment outputs remains a key challenge, requiring careful alignment with human expert judgment rather than simple accuracy metrics.