How RLHF works: the three-stage pipeline and four-model architecture

Reinforcement Learning from Human Feedback aligns LLM outputs with human preferences through a three-stage pipeline: supervised fine-tuning (SFT) on high-quality demonstrations, explicit reward model training on human-annotated preference pairs, and PPO-based reinforcement learning optimization. At each stage, the engineering complexity compounds — and by the time the RL phase begins, the system must simultaneously run four distinct model instances.

The reward model training stage is the central engineering challenge. As documented in a 2024 patent from Beijing Baichuan Intelligent Technology, the four concurrent components are: a reward model (scores responses), a critic model (evaluates action quality), an actor model (generates candidate responses), and the frozen reference LLM. The KL divergence between the initial policy and the actor’s policy is added as a regularization term to prevent the fine-tuned model from drifting too far from the pre-trained distribution — a defining architectural feature of all standard RLHF implementations.

PPO (Proximal Policy Optimization) is the dominant RL algorithm within RLHF implementations. As described in a 2025 patent from Suzhou Yuannao Intelligent Technology, PPO samples prompts from a pool, generates responses, computes immediate rewards from the reward model alongside KL-regularized future returns, and iteratively updates the policy using gradient descent on the advantage function. The multi-model requirement is one of RLHF’s most frequently cited engineering drawbacks: a 2024 patent from China Construction Bank explicitly notes that traditional PPO-based RLHF involves numerous hyperparameters and can lead to training instability and collapse.

Tsinghua University’s 2024 filing confirms the full pipeline: pre-training, then reward model training on human-labeled comparison data (using ranking rather than absolute scores to reduce annotator calibration noise), followed by PPO-based RL fine-tuning. The use of rankings rather than absolute scores is a deliberate design choice to reduce variance introduced by differences in individual annotator calibration.

How DPO works: reward-free preference optimization

Direct Preference Optimization eliminates the explicit reward model entirely by reformulating preference alignment as a binary classification problem: instead of asking “which response maximizes reward?”, DPO asks “is this response’s reward high or low?” and optimizes that objective directly over preference pair data using a cross-entropy-type loss function. This single reformulation removes the need for the RL training loop and reduces the required model instances from four to two.

The standard DPO pipeline is two-stage: SFT followed by DPO preference optimization over chosen/rejected pairs. Zhejiang University’s 2024 patent on medical LLM fine-tuning explicitly recommends DPO over RLHF precisely because it does not require an explicit reward function or a separately trained reward model — a significant advantage in data-constrained or domain-specific deployment scenarios where the engineering overhead of RLHF is prohibitive.

Despite its simplicity, DPO has documented limitations. A 2025 patent from Midu Technology reports that DPO is susceptible to overfitting on limited human preference datasets, reducing generalization to unseen prompts. Additionally, the reference model itself must be trained and maintained, and the model’s performance ceiling is bounded by the capability of that reference model. The temperature parameter β introduces its own sensitivity: it controls the influence of the reference policy, and the balance between overfitting to the preference dataset and retaining pre-trained capabilities must be carefully managed.

A structurally important limitation is DPO’s offline nature. A 2025 Alipay patent notes that vanilla DPO requires upfront human annotation — design of questions and answers plus manual scoring before training begins — which limits training efficiency. Their proposed multi-round self-iterative DPO approach addresses this by using the model itself to generate and score candidate answers iteratively, partially simulating the online dynamics of RLHF without the full four-model infrastructure.

“Although DPO provides a simpler solution, RL-based training remains indispensable for generalization and universality on complex tasks, as it guides the model through reward mechanisms to make decisions more consistent with human logic.”

Head-to-head: architecture, stability, and failure modes compared

The core tradeoffs between RLHF and DPO span four dimensions: training architecture complexity, reward signal fidelity, stability under training, and the online versus offline learning distinction. Each dimension reveals a different set of engineering risks that practitioners must weigh against their deployment context.

Training architecture complexity

RLHF requires constructing, training, and running inference on multiple concurrent models during the RL phase. As documented in a 2025 patent from Du Xiaoman Technology, coordinated updates between the actor and critic models are necessary to prevent instability and oscillation, and reward signals must be calibrated at both sentence and token levels. A 2025 patent from Beijing Zhongke Wenge Technology explicitly states that RLHF requires large-scale labeled data with human expert ranking followed by complex RL pipeline execution, whereas DPO uses offline preference pair data to directly optimize model parameters — a contrast that makes DPO significantly more accessible for teams without large-scale annotation infrastructure.

Reward signal fidelity and reward hacking

RLHF’s explicit reward model is both its strength and its vulnerability. The reward model learns a rich, continuous scoring function over responses, which in principle provides nuanced feedback. However, as flagged in a 2025 patent from Miaomu Ltd., reward models in RLHF are sensitive to noise in human annotations, exhibit limited generalization in deployment settings, and are susceptible to reward hacking — where the policy model exploits artifacts in the reward model rather than genuinely improving quality. This is a well-documented failure mode also tracked by researchers at DeepMind and Anthropic.

Stability and hyperparameter sensitivity

RLHF implementations using PPO are widely noted for training instability. A 2025 patent from CloudWalk Technology confirms that RLHF training consumes large computational resources, is prone to convergence difficulties due to RL’s inherent instability, and is extremely sensitive to hyperparameters such as the KL divergence coefficient — small changes can produce dramatic performance swings. DPO’s loss function is more stable, but introduces its own sensitivity through the β temperature parameter. According to WIPO‘s patent filings database, the volume of patents addressing RLHF instability has increased substantially in the 2023–2026 period, reflecting the scale of the engineering challenge.

Online vs. offline learning

Standard RLHF is an online learning process — the policy model generates new responses, receives reward signals, and updates iteratively in a feedback loop. DPO is primarily an offline learning process that optimizes over a fixed dataset of preference pairs. This distinction has significant implications for generalization: RLHF’s online loop allows continuous policy improvement beyond the static training set, while DPO’s performance ceiling is bounded by the coverage of the preference dataset. For complex tasks requiring nuanced reasoning or multi-objective optimization, a 2026 patent from Suzhou Yuannao Intelligent Technology states explicitly that RL-based training remains indispensable for generalization and universality.

Application domains, hybrid approaches, and the next generation

Both RLHF and DPO are deployed across diverse domains in the patent corpus, but the choice between them is rarely arbitrary — it reflects specific constraints around data availability, engineering capacity, and task complexity. Meanwhile, a third category of approaches is emerging that moves beyond both paradigms.

Domain-specific deployments

Financial applications include work from Alipay and China Construction Bank, where RLHF-based reward model training uses preference rankings from domain experts. Actimize Ltd.’s 2026 patent applies the full RLHF pipeline with a reward model and PPO for fraud detection in financial tabular data. Medical LLM fine-tuning, as addressed in Zhejiang University’s 2024 patent, explicitly prefers DPO for reduced engineering overhead. Google LLC’s 2026 patent extends RLHF by replacing human annotators with search engine feedback signals, demonstrating that the reward signal source can be automated without fundamentally changing the RLHF architecture — a finding consistent with alignment research published by Nature on automated feedback mechanisms.

Hybrid SFT+DPO+PPO unified frameworks

A significant patent cluster involves hybrid methods that integrate SFT, DPO, and PPO within unified training pipelines. Baidu’s 2024 patent proposes combining DPO and PPO algorithms for SFT training into a unified alignment learning framework, explicitly noting that current frameworks merging only SFT and DPO lose the advantages of online reinforcement learning, and that methods merging DPO and PPO represent the next generation of alignment training. Qingdao Ant Robot’s 2025 patent addresses the conflict between SFT (which encourages imitation of expert behavior) and RL (which encourages policy exploration) by employing a dual-path parallel architecture with a dynamic weight fusion mechanism that smoothly transitions from imitation learning to exploration.

Next-generation alternatives: Bayesian and tied preference methods

Novel alternatives to both RLHF and DPO are emerging from large-lab research divisions. Google LLC’s Posterior Preference Optimization (2025) takes a Bayesian approach, training the model to predict posterior token probabilities conditioned on human preferences, thereby preserving pre-trained prediction distributions while adding sequential preference-tuned predictions. This addresses the RLHF weakness that balancing multiple preference models is difficult without degrading individual compliance. DeepMind Technologies’ Tied Preference Optimization (2025) extends DPO by explicitly modeling tied preferences — cases where neither response is preferred — which standard DPO’s binary formulation cannot handle. DeepMind also filed Direct Posterior Preference Fine-Tuning (2025), a supervised variant that directly predicts posterior token probabilities conditioned on positive preferences without requiring additional per-vocabulary-token inference at decoding time.

Federated and privacy-preserving RLHF

Google LLC’s Federated RLHF patent demonstrates RLHF extended to federated learning settings, where multiple user devices each run local reward models and aggregate scores to a central server for model training — preserving privacy while expanding the diversity of human feedback signals. This architecture is structurally more compatible with RLHF’s explicit reward model than with DPO’s offline preference dataset approach, since federated reward scoring maps naturally to the reward model training stage. The implications for privacy-sensitive domains such as healthcare and finance are significant, aligning with standards tracked by ISO on federated and privacy-preserving machine learning.

Who is filing what: key players and innovation trends across the patent corpus

The patent landscape surveyed spans filings from 2023 through 2026 across China, the United States, Korea, and WIPO. The dominant assignees by filing volume include Alipay (Hangzhou), Google LLC, DeepMind Technologies, Microsoft Technology Licensing, Baidu, Tencent, and multiple Chinese universities. Their strategic focus areas are distinct and reveal the competitive shape of the field.

Alipay (Hangzhou) / Ant Group’s multiple CN filings focus on unified multi-stage fine-tuning pipelines integrating RLHF and DPO, iterative self-training DPO, and token-level reward granularity improvements for RLHF. Google LLC’s US and WO filings expand RLHF beyond human feedback to automated feedback sources (search engines), federated reward aggregation, and Bayesian posterior preference optimization. DeepMind Technologies is advancing the theoretical frontier with tied preference handling and direct posterior fine-tuning. Microsoft Technology Licensing covers in-situ user interaction feedback as a continuous alignment signal. Baidu, Beijing Baichuan, Tencent, and Zhipu AI are focused on multi-model ensemble reward learning, actor-critic stabilization, and integrated alignment frameworks combining SFT, DPO, and PPO.

Chinese universities — Peking University, Zhejiang University, Harbin Institute of Technology, Southeast University, Tsinghua University, and Northeast University — are contributing theoretical extensions including game-theoretic DPO alignment, multi-reward confidence-weighted DPO, bias-intensity-weighted DPO loss functions, and minimax robustness frameworks. These academic filings, tracked through PatSnap’s patent search platform, represent a distinct innovation layer from the production-focused industry filings.

A clear trend across the corpus is the migration from pure RLHF implementations toward hybrid or DPO-centric pipelines for production deployment, while RLHF retains primacy for complex multi-objective generalization tasks. Simultaneously, the academic and large-lab patent filings show rapid innovation in replacing both approaches with principled alternatives grounded in Bayesian inference, game theory, and multi-objective optimization. Practitioners seeking to navigate this landscape can access the full corpus through PatSnap Eureka’s AI-native innovation intelligence tools.