Baidu Open Sources Ernie 4.5: LLM Series Masking from 0.3b to 424b Settings

Baidu officially opened its last Ernie 4.5 series, a powerful family of foundation models designed for a better understanding of language, reasoning and generation. The version includes ten model variants ranging from 0.3b -compact dense models to massive mixing mixing mixtures (MOE), with the largest variant totaling 424b settings. These models are now available for free to the global research and developer community by hugging the face, allowing open experimentation and wider access to the technology of the Chinese and multilingual language.

Technical overview of ERNIE 4.5 architecture

The Ernie 4.5 series is based on the previous iterations of Baidu of Ernie models by introducing architectures of advanced models, including dense and activated MOE conceptions by clear route. The MOE variants are particularly notable for the counting of scaling parameters: the ERNIE 4.5-MOE-3B and ERNIE 4.5-MOE-47B variants activate that an expert subset per input token (generally 2 of the 64 experts), keeping the number of active parameters manageable while retaining the expressiveness and generalization capacities of the model.

ERNIE 4.5 models are formed using a supervised fine setting mixture (SFT), learning by strengthening with human feedback (RLHF) and contrasting alignment techniques. The formation corpus extends over 5.6 billions of tokens in various fields in Chinese and English, using the Baidu multi-stain pre-election pipeline. The resulting models demonstrate a high fidelity in the monitoring of instructions, multi-turn conversation, long generation and marks of reasoning.

Model variants and open source version

The Ernie 4.5 version includes the following ten variants:

• Dense models: Ernie 4.5-0.3b, 0.5b, 1.8b and 4b
• Moe models: Ernie 4.5-MOE-3B, 4B, 6B, 15B, 47B and 424B Total parameters (with variable active parameters)

The MOE-47B variant, for example, only activates 3B parameters during inference while having a total of 47B. Likewise, the 424B model – the largest ever published by Baidu – uses clear activation strategies to make the inference feasible and evolving. These models support the FP16 and Int8 quantification for effective deployment.

Performance benchmarks

ERNIE 4.5 models show significant improvements in several Chinese and multilingual NLP. According to the official technical report:

• On CmmluErnie 4.5 goes beyond the previous versions Ernie and reaches advanced precision in understanding the Chinese language.
• On MmluThe multilingual reference, Ernie 4.5-47b shows competitive performance with other leading LLMs such as GPT-4 and Claude.
• For long generationErnie 4.5 reaches higher coherence and invoice scores when evaluated using internal baidu measures.

In instructions monitoring tasks, models benefit from a contrasting fine adjustment, showing improved alignment with the intention of the user and reduced hallucination rates compared to previous ERNIE versions.

Applications and deployment

ERNIE 4.5 models are optimized for a wide range of applications:

• Chatbots and assistants: Multilingual support and instructions monitoring make it suitable for AI assistants.
• Research and answer to questions: High loyalty and generation loyalty allow integration with CLOTH pipelines.
• Content generation: The long text and the generation of knowledge -rich content is improved with better factual earthing.
• Multimodal code and extension: Although the current version focuses on the text, Baidu indicates that Hernie 4.5 is compatible with multimodal extensions.

With a support for a context length of up to 128k in certain variants, the Ernie 4.5 family can be used in tasks requiring memory and reasoning on long documents or sessions.

Conclusion

The Ernie 4.5 series represents an important step in the development of Open Source, offering a versatile set of models adapted to evolutionary tasks, multilingual and aligned by instruction. Baidu's decision to publish models ranging from light variants of 0.3b to a MOE model of parameter 424B highlights its commitment to research on inclusive and transparent AI. With complete documentation, open availability on embraces and support for effective deployment, Ernie 4.5 is positioned to accelerate the global progress of the understanding and generation of natural language.

---

Discover the Paper And Models on the embraced face. All the merit of this research goes to researchers in this project. Also, don't hesitate to follow us Twitter And don't forget to join our Subseubdredit 100k + ml and subscribe to Our newsletter.