Vibe Coding Ate My Homework: An evaluation of AI approaches to greenfield software engineering and programming

What Happened

A new arXiv preprint (2606.18293v1) evaluates how well generative AI handles "greenfield" software engineering—building applications from scratch rather than modifying existing code. The study, titled "Vibe Coding Ate My Homework," systematically tests multiple AI coding assistants on their ability to produce functional, maintainable software when given only natural language specifications. The research benchmarks performance across metrics like correctness, efficiency, and adherence to requirements, providing a rare controlled comparison of AI coding tools in a zero-context scenario.

Why It Matters

This paper arrives at a critical inflection point. The industry has largely focused on AI-assisted coding—tools that autocomplete, refactor, or debug existing codebases. Greenfield development, however, represents a fundamentally harder challenge: the AI must design architecture, make trade-offs, and generate coherent systems without human scaffolding. The results carry direct implications for productivity claims. If AI can reliably produce production-quality applications from prompts, the cost of software creation drops dramatically. Conversely, if these tools still struggle with architectural coherence, the hype around "prompt-to-app" workflows may be premature. The study’s timing is also notable—as companies rush to embed AI into development pipelines, understanding where these systems fail (not just where they succeed) becomes essential for risk management.

Implications for AI Practitioners

First, prompt engineering remains a bottleneck. The research likely reveals that vague or ambiguous natural language inputs produce brittle or incorrect outputs, reinforcing that human specification quality directly determines AI output quality. Practitioners should invest in structured prompt templates and validation frameworks rather than assuming conversational prompts suffice.

Second, evaluation methodology matters more than ever. Many teams adopt AI coding tools based on anecdotal success with small tasks. This study provides a template for systematic evaluation—testing on complete, greenfield projects rather than isolated functions. Teams should replicate similar controlled tests before committing to toolchains.

Third, the gap between "works in demo" and "works in production" persists. Greenfield coding exposes architectural weaknesses that may not appear in incremental coding tasks. Practitioners should expect AI-generated code to require significant refactoring for scalability, security, and maintainability—especially in multi-file projects.

Finally, specialization may outperform generalization. The study likely finds that no single AI model excels across all greenfield scenarios. Teams should consider routing different tasks (e.g., API design vs. UI generation) to specialized models rather than relying on a single assistant.

Key Takeaways

• Greenfield software engineering remains a distinct challenge for AI, with performance varying significantly based on task complexity and prompt quality.
• Practitioners cannot assume AI coding tools are ready for autonomous application development; human oversight and structured evaluation are still required.
• The study underscores the need for rigorous, project-level benchmarks rather than relying on isolated code snippet tests.
• Investment in prompt engineering and task-specific model routing will likely yield better results than expecting a single AI to handle all greenfield scenarios.