PEBS: Per-rater Empirical-Bayes Shrinkage for RLHF Reward-Model Calibration

A Statistical Fix for the Crowd Problem in RLHF

The paper "PEBS: Per-rater Empirical-Bayes Shrinkage for RLHF Reward-Model Calibration" tackles a subtle but critical flaw in how reward models are trained for Reinforcement Learning from Human Feedback (RLHF). Currently, most RLHF pipelines aggregate preference data from thousands of annotators and fit a single global calibrator—essentially assuming all raters use the same internal scale when judging model outputs. This assumption is false in practice.

Different annotators have systematically different rating-scale offsets (some are generous, others strict) and slopes (some compress their ratings into a narrow range, others spread them wide). Collapsing these into one calibrator introduces systematic bias: the reward model learns to favor outputs that appeal to the "average" rater's idiosyncrasies rather than reflecting genuine quality differences. PEBS addresses this by applying per-rater Empirical Bayes shrinkage—a technique that estimates individual rater calibration parameters while pulling extreme estimates toward the population mean to prevent overfitting from sparse data.

Why This Matters

The implications extend beyond a technical improvement. First, reward hacking becomes more insidious when the reward model is calibrated on pooled, unadjusted data. If a subset of raters consistently prefers verbose, hedging responses, the global calibrator will inflate the reward for verbosity—even if most raters penalize it. PEBS reduces this by isolating per-rater biases before aggregation.

Second, scaling RLHF to diverse user bases requires handling rater heterogeneity gracefully. As RLHF is applied to multilingual, multicultural contexts, the assumption of a single rater "norm" becomes untenable. PEBS provides a principled statistical framework for this heterogeneity without requiring manual rater clustering or exclusion.

Third, practitioners can implement this with minimal overhead. Empirical Bayes shrinkage is computationally cheap—it requires only per-rater mean and variance estimates plus a global prior, which can be computed from the same preference data already collected. This is not a new training pipeline; it is a calibration layer inserted between raw preferences and reward model training.

Implications for AI Practitioners

• Data quality audits should include rater calibration checks. Before deploying a reward model, examine per-rater rating distributions. High variance in offsets or slopes signals that PEBS (or similar methods) is necessary.
• Reward model evaluation should test for rater-bias robustness. Standard validation sets that pool all raters may mask calibration issues. Practitioners should create held-out sets that preserve rater identity to measure how well the model generalizes across different annotation styles.
• PEBS is a low-risk, high-upside intervention. It does not require re-collecting data or retraining from scratch. Adding a per-rater shrinkage step to the calibration pipeline can reduce systematic bias without increasing model complexity.

Key Takeaways

• Current RLHF reward models assume all annotators rate on the same scale, introducing systematic bias from rater heterogeneity.
• PEBS applies per-rater Empirical Bayes shrinkage to estimate individual calibration parameters, reducing bias without overfitting sparse data.
• The method is computationally cheap and can be added to existing RLHF pipelines as a calibration layer.
• Practitioners should audit rater calibration before deployment and evaluate reward models on rater-aware validation sets.