ForAug: Mitigating Biases in Image Classification via Controlled Image Compositions

The Hidden Shortcut in Image Classification

A new paper from arXiv, "ForAug: Mitigating Biases in Image Classification via Controlled Image Compositions," tackles a persistent blind spot in computer vision: compositional bias. The researchers demonstrate that large-scale image classification datasets are riddled with systematic shortcuts. Objects are not randomly distributed—they are typically centered, appear at predictable scales, and co-occur with class-specific backgrounds. A model trained on such data doesn't learn to recognize a "dog" so much as it learns to associate "centered furry object on a grass background" with the label.

The core contribution of ForAug is a data augmentation strategy that explicitly breaks these spurious correlations. Instead of relying on random cropping or color jittering, ForAug uses controlled image compositions—systematically varying object position, scale, and background context during training. This forces the model to rely on the object's intrinsic features rather than environmental shortcuts.

Why This Matters

This research addresses a fundamental flaw in how we evaluate and deploy image classifiers. Standard benchmarks like ImageNet inadvertently reward models for exploiting these biases. A model that achieves 90% accuracy may fail catastrophically when deployed in the real world, where a dog might appear at the edge of a frame, in an office, or at an unusual scale. This is not a theoretical concern—autonomous vehicles misclassifying pedestrians in unusual poses or medical imaging systems failing on atypical patient demographics are direct consequences of compositional bias.

ForAug’s approach is particularly significant because it does not require relabeling data or collecting new datasets. It is a training-time intervention, making it practical for existing pipelines. The method is also model-agnostic, meaning it can be applied to ResNets, Vision Transformers, or any architecture.

Implications for AI Practitioners

For practitioners, this paper offers a concrete tool to improve model robustness without sacrificing accuracy. The key insight is that standard data augmentation is insufficient—it randomizes context but does not systematically ensure that the model sees objects in all plausible compositions. ForAug provides a structured way to do this.

However, there are practical considerations. Controlled composition requires segmentation masks or bounding boxes for objects during training, which may not be available for all datasets. The paper’s method also increases training complexity and computational cost, as it involves generating multiple compositions per image. Practitioners will need to weigh these costs against the gains in out-of-distribution generalization.

The broader lesson is that dataset bias is not just about demographic representation or class imbalance. Compositional bias is equally pernicious and often invisible in standard evaluation metrics. ForAug serves as a reminder that the data distribution itself encodes shortcuts that models eagerly exploit.

Key Takeaways

• Compositional bias—where objects appear at predictable positions, scales, and backgrounds—is a major source of overfitting in image classifiers, leading to poor real-world generalization.
• ForAug introduces a controlled image composition augmentation that systematically breaks these spurious correlations during training, improving robustness without new data collection.
• The method is practical for existing pipelines but requires segmentation masks and increases computational overhead, limiting immediate applicability for all teams.
• Practitioners should audit their datasets for compositional shortcuts and consider structured augmentation as a standard part of the training pipeline, not an afterthought.