Evolutionary Feature Engineering for Structured Data

The Evolution of Data Preparation: LLMs as Feature Engineers

The paper introduces Evolutionary Feature Engineering (EFE), a framework that leverages large language models as open-ended search operators within evolutionary optimization to automatically discover preprocessing transformations for structured data. Rather than treating LLMs as static prediction engines, EFE positions them as adaptive agents that can propose, mutate, and refine feature engineering steps—treating data preparation as a search problem rather than a manual craft.

Why This Matters

Feature engineering remains one of the most labor-intensive and domain-specific bottlenecks in applied machine learning. While automated machine learning (AutoML) has made strides in model selection and hyperparameter tuning, the preprocessing pipeline—handling missing values, creating interaction terms, binning, scaling, and encoding—has largely resisted automation because it requires understanding both the data semantics and the downstream model's behavior.

EFE addresses this by combining two complementary strengths: the generative flexibility of LLMs (which can propose novel transformations based on learned patterns from code and documentation) and the structured search capability of evolutionary algorithms (which systematically evaluate and combine promising candidates). This hybrid approach could significantly reduce the time data scientists spend on trial-and-error feature engineering.

The use of LLMs as "open-ended search operators" is particularly noteworthy. Traditional evolutionary approaches require pre-defined mutation and crossover rules. EFE instead allows the LLM to generate new transformations contextually, potentially discovering non-obvious preprocessing steps that a human might miss—such as domain-specific aggregations or composite features that combine multiple raw columns in unexpected ways.

Implications for AI Practitioners

For teams working with tabular data—still the dominant format in enterprise settings—EFE suggests a future where data preparation becomes more automated and less reliant on deep domain expertise. Practitioners should watch for several developments:

First, the computational cost of repeatedly querying an LLM during evolution could be substantial. Organizations will need to weigh the time savings in manual feature engineering against increased inference costs. Second, the quality of discovered features will depend heavily on the LLM's training data—models with stronger coding and mathematical reasoning capabilities will likely produce more useful transformations.

Third, interpretability remains a concern. While evolutionary search can produce high-performing feature sets, understanding why a particular transformation works may be harder than with human-designed features. Practitioners should plan for validation workflows that test discovered features on holdout data and against domain constraints.

Finally, EFE opens the door to more dynamic feature engineering—pipelines that continuously evolve as new data arrives or business conditions change. This could be particularly valuable in domains like finance, e-commerce, or healthcare where data distributions shift over time.

Key Takeaways

• EFE combines LLM-based generation with evolutionary search to automate feature engineering for structured data, reducing manual preprocessing effort
• The framework treats data transformation as an open-ended search problem, potentially discovering non-obvious features that human engineers might overlook
• Practitioners should consider the cost-performance tradeoff, as LLM queries during evolution can be computationally expensive
• Validation and interpretability workflows will need to adapt to handle automatically discovered features that lack explicit human rationale