Optimization of LLM for human alignment using strengthening learning
Large languages often require an additional alignment phase to optimize them for human use. In this phase, the learning of strengthening plays a central role by allowing models to make decisions based on human feedback or accuracy based on tasks. This fine setting allows models to line up more closely with user expectations, which makes them more suitable for apps based on precise mathematical instruction or tasks.
Challenges in the choice of online and online online learning strategies
A major difficulty occurs when choosing the most effective means of carrying out this fine setting. The training methods are divided into two extreme approaches – to be offered that depend on static and pre -generated data and entirely online approaches which are constantly updated at each new interaction. Each method has distinct challenges. Offline models cannot adapt during training, which limits performance, while online models often require more IT resources. In addition, ensuring that the models work well through mathematical (verifiable) and open (non -watchable) tasks add additional complexity to this choice.
Overview of alignment algorithms: DPO and GRPO
Historically, tools such as direct optimization of preferences (DPO) and the relative optimization of group policies (GRPO) have been used for the alignment of the model. DPO works offline and is designed to operate with preference -based data pairs. It is appreciated for its simplicity and effectiveness of data but does not have the adaptability of online methods. GRPO is based on the PPO algorithm and manages the fine online adjustment by comparing output groups to calculate the relative advantages. While GRPO adapts in real time and is suitable for dynamic reward systems, its policy of policy increases the calculation charge and makes experimentation more demanding.
A balanced alternative for LLM alignment
The research introduced by Meta and Nyu explored a method to overcome these limitations thanks to a semi-unique training configuration. This technique modulates how often the generation and formation components of the model are synchronized, rather than updating at each training stage, as in fully online methods, or not at all, as in offline configurations. The semi-online method hits common ground by adjusting the synchronization rate. Researchers have designed this approach to reduce training time and maintain adaptability of high models. The modular configuration also allowed them to apply DPO or GRPO with specific reward models in a flexible way.
Next instruction and mathematical reasoning
The methodology involved the fine adjustment of the LLAMA-3.1-8B-ISTRUCT model using two types of tasks: the following open education and the solving mathematical problems. For non-veritifiable tasks, the user prompts were sampled from the WildChat-1m data set and evaluated using the ATHENE-RM-8B reward model, which attributes scalar scores to each prompt. For verifiable tasks, the team used the Numinamath data set in conjunction with the Math-Verify toolbox, which checks if the generated responses line up with the expected outputs. Training experiences were carried out on 32 NVIDIA H200 GPU for training and 8 GPU for inference, with different configurations comparing offline, semi-online and online synchronization intervals.
Performance gains on verifiable and non -verifiable tasks
Performance differences have been observed. On Math500, the offline DPO reached a precision of 53.7%, while the semi-in-line DPO with a synchronization interval of S = 100 reached 58.9%. Online DPO and GRPO showed results similar to 58.7% and 58.1%, respectively. Similar trends have been observed on the Numinamath reference, where the offline DPO reached 36.4%, and semi-one variants increased this to 39.4% (S = 10). Performance gains were not limited to mathematical tasks. When non -veritifiable tasks were evaluated with Alpacaeval 2.0 and hard references of the arena, the models formed with types of mixed awards have been better performed. The combination of verifiable and non -verifiable rewards in a single training configuration has resulted in stronger average scores, which indicates that the generalized method effectively.
A flexible and evolving approach for learning to strengthen in LLM
This study shows that the models of large refiners do not require strict adhesion to offline or online configurations. By introducing a flexible synchronization scheme, Meta and Nyu's research team has effectively increased the efficiency of training while maintaining or improving performance. The results show that the types of rewards and the frequency of carefully balanced training synchronization lead to models that work well between the types of tasks without incurring high calculation costs.
Discover the Paper. All the merit of this research goes to researchers in this project. Also, don't hesitate to follow us Twitter,, YouTube And Spotify And don't forget to join our Subseubdredit 100k + ml and subscribe to Our newsletter.
Nikhil is an intern consultant at Marktechpost. It pursues a double degree integrated into materials at the Indian Kharagpur Institute of Technology. Nikhil is an IA / ML enthusiast who is still looking for applications in fields like biomaterials and biomedical sciences. With a strong experience in material science, he explores new progress and creates opportunities to contribute.
