Phi-4-Mini-Flash-RestadingMicrosoft's latest addition to the family of Phi-4 models is a light and light language model designed to excel in long-term reasoning while maintaining high inference efficiency. Released FaceThis 3.8B parameter model is a distilled version of Phi-4-Mini, refined for dense reasoning tasks such as mathematical problem solving and the answer to multi-hop questions. Built using Microsoft's new Sambay Hybrid decoder architecture, it achieves advanced performance among compact models and operates up to 10 × faster than its predecessor on long generation tasks.
Architecture: closed memory meets a hybrid decoding
At the heart of the revival of Phi-4-Mini-Flash is the Sambay Architecture, a new model of a hybrid decoder that integrates State space models (SSMS) with attention layers using a light mechanism called Closed memory unit (GMU). This structure allows an effective memory sharing between layers, considerably reducing the inference latency in the scenarios with a long and long generation context.
Unlike architectures based on transformers who are strongly based on attention calculations with high memory intensity, Sambay operates Samba (a hybrid SSM architecture) in the self -decorator and replaces about half of the diapers of cross -attention in the decoder crossed by GMU. GMUs serve as cheap trigger and element functions that reuse the hidden state of the final SSM layer, thus avoiding redundant calculation. This results in a complexity of a linear prefiless and the lower decoding I / S, which gives substantial accelerated during inference.
Training pipeline and reasoning capacities
The Phi-4-Flash model is pre-formulated on 5T tokens from real synthetic data and high-quality filtered, in accordance with the rest of the Phi-4-Mini family. After pre-training, he undergoes Fine refinement supervised in several stages (SFT) And Direct preferences optimization (DPO) Use of set -up data sets on reasoning. Unlike, unlike the Phi-4-Mini silk race, it entirely excludes learning to strengthen (RLHF).
Despite this, the implementation of Phi-4-Mini-Flash surpasses the seizure of Phi-4-Mini on a series of complex reasoning tasks. On the MATH500 reference, he obtained a pass at @ 1 precision of 92.45%, outperforming the seasoning racing of Phi-4-Mini (91.2%) and exceeding other open models like QWEN-1.5B and Sur-Stratos-7B. On AIME24 / 25, it also shows strong gains, with more than 52% precision on AIME24.
This performance jump is attributed to the capacity of the architecture for Long generation of the chain of thoughts (CO). With a 64K context length support and an inference optimized under the vllm Framework, the model can generate and reason in contexts of several thousand people without a neck of strangulation. In latency references with 2k-token prompts and 32k-token generations, Phi-4-Mini-Flash-Seasoning 10 × higher flow than his predecessor.
Effective long -term context treatment
The efficiency gains in the revival of Phi-4-Mini-Flash are not only theoretical. Thanks to the design of the decoder-hybrid-decooder, the model obtains competitive performance on long-context references such as the repertoire and the rule. For example, with a Watch out for the sliding window (SWA) Size as small as 256, it maintains a high recovery accuracy, indicating that long -range tokens dependencies are well captured via SSMS and memory sharing on GMU.
These architectural innovations lead to a reduction in general calculation and memory costs. For example, during decoding, the GMU layers replace attention operations that would otherwise cost o (n · d) time per token, reducing this to O (d), where n is the length of sequence and D is a hidden dimension. The result is real-time inference capacity even in multi-tours scenarios or at the document level.
Open weight and use cases
Microsoft has opened the weights and configuration of the opening model by hugging the face, offering full access to the community. The model supports the length of the context of 64K, works under the standard side of the hugs and the VLLM times, and is optimized for the quick token flow on the A100 GPUs.
Potential use cases for the transmission of Phi-4-Mini-Flash include:
- Mathematical reasoning (for example, Sat, A like level problems)
- Qa Multi-Hop
- Analysis of legal and scientific documents
- Autonomous agents with long -term memory
- High speed cat systems
Its combination of free access, capacity for reasoning and effective inference makes it a solid candidate for deployment in environments where calculation resources are limited, but the complexity of tasks is high.
Conclusion
Phi-4-minini-Flash-Seasoning illustrates how architectural innovation-in particular hybrid models taking advantage of SSM and effective triggers-can provide transformer gains in reasoning performance without size or cost of Montgolfière model. It marks a new direction in effective long -context language modeling, opening the way to real -time reasoning agents on devices and open source alternatives to commercial LLM.
Discover the Paper,, Codes,, Model on the embraced face And Technical details. All the merit of this research goes to researchers in this project. Also, don't hesitate to follow us Twitter,, YouTube And Spotify And don't forget to join our Subseubdredit 100k + ml and subscribe to Our newsletter.
Nikhil is an intern consultant at Marktechpost. It pursues a double degree integrated into materials at the Indian Kharagpur Institute of Technology. Nikhil is an IA / ML enthusiast who is still looking for applications in fields like biomaterials and biomedical sciences. With a strong experience in material science, he explores new progress and creates opportunities to contribute.
