ProRL: Effective Reinforcement Learning for Proactive Recommendation via Rectified Policy Gradient Estimation
Abstract
Proactive recommender systems using reinforcement learning face challenges with gradient estimation bias and variance, which are addressed through stepwise reward centering and position-specific advantage estimation mechanisms.
Proactive Recommender Systems (PRSs) aim to guide user preference shift toward target items by generating paths of intermediate recommendations. Reinforcement learning (RL) provides a principled framework for optimizing such sequential decision tasks, as path rewards can naturally capture both short-term acceptance and long-term guidance effectiveness. However, naively applying policy gradients to PRS results in deficient gradient estimation. We identify two deficiencies: (1) path-level rewards decompose into step-level rewards with positive mean, creating a length-dependent bias that causes gradients to favor path extension over meaningful exploration; (2) weighting each step by the entire path-level reward ignores the decomposition structure, leading to high gradient variance. To rectify these two deficiencies, we propose an effective RL framework ProRL with two novel mechanisms for proactive recommendation. First, Stepwise Reward Centering subtracts expected rewards to neutralize length-dependent bias, ensuring that path extension yields zero expected gradient signal. Second, Position-Specific Advantage Estimation leverages the reward decomposition structure to compute step-dependent baselines, reducing gradient variance. Together, these mechanisms yield policy gradients that precisely target path quality. Our experiments on three real-world datasets demonstrate that ProRL significantly outperforms state-of-the-art PRSs. Our code is available at https://github.com/hongruhou89/ProRL.
Community
Standard policy gradients are fundamentally broken for proactive recommendation. ProRL fixes them with rectified gradient estimation.
Proactive Recommender Systems aim to gradually shift user preferences toward target items through carefully designed paths of intermediate recommendations. Think of guiding a Sci-Fi fan toward comedy by recommending WALL-E → Zootopia → The Secret Life of Walter Mitty. RL should be the perfect framework for this sequential planning problem. But applying standard policy gradients fails completely.
We traced this failure to two deficiencies in gradient estimation that are specific to the proactive recommendation setting.
The length shortcut. Path-level rewards decompose into step-level rewards that have positive expected value at every step. Longer paths trivially accumulate higher rewards. Standard policy gradients pick up on this signal immediately, and within a few hundred updates the model converges to generating identical maximum-length paths for every user, with near-zero diversity.
High gradient variance. Standard estimation weights each step's log-probability by the full path reward. But the action at step t only affects rewards from t onward. Including earlier rewards injects pure noise into the gradient signal.
ProRL rectifies both problems through two mechanisms. Stepwise Reward Centering subtracts the expected per-step reward so that extending a path yields zero expected gradient signal. The optimizer can no longer exploit the length shortcut and is forced to improve actual path quality. Position-Specific Advantage Estimation computes step-adapted baselines from the reward-to-go at each position, achieving roughly 5% of the gradient variance of REINFORCE without requiring a learned critic (which, we show, drifts and degrades over training).
On MovieLens-1M, Steam, and Amazon-Book, ProRL significantly outperforms all baselines (sequential, heuristic, supervised, and LLM-based) across all four metrics. Notably, ProRL generalizes well to reward signals it was never trained on, suggesting the policy has internalized genuine principles of user preference guidance rather than simply fitting the training objectives. Cross-evaluator analysis with unseen recommendation models (GRU4Rec, BERT4Rec, LightSANs) confirms the learned guidance strategy generalizes beyond the training environment.
Happy to discuss and comment :)
📄 Paper accepted at ICML 2026
💻 Code available at https://github.com/hongruhou89/ProRL
Get this paper in your agent:
hf papers read 2605.28293 Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash Models citing this paper 0
No model linking this paper
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 0
No Space linking this paper