Great language models (LLMS), with billions of parameters, feed many AI -oriented services in all industries. However, their massive size and their complex architectures make their calculation costs during inference an important challenge. As these models evolve, the optimization of the balance between calculation efficiency and exit quality has become a crucial research area.
The basic challenge lies in how LLM manages inference. Each time an entry is processed, the entire model is activated, which consumes in -depth calculation resources. This complete activation is not necessary for most tasks, because only a small subset of neurons contributes significantly to the final exit. Existing sparse activation methods try to solve this problem by selectively deactivating less important neurons. However, these approaches often focus only on the extent of hidden states while ignoring the critical role of weight matrices in the propagation of errors through the network. This surveillance leads to high approximation errors and deteriorates from the performance of the model, in particular at higher rarity levels.
Large activation techniques have included methods such as the mixture of experts (MOE) used in models such as GPT-4 and Mistral, which are based on additional training to know which experts activate for each entry. Other approaches, such as the Sarcelle and cats, aim to reduce calculation using the size of hidden activations to cut neurons, but they always leave room for improvement. These methods often fight on the balance between rarity and precision, as they can wrongly deactivate important neurons or keep those which have a minimum influence. In addition, they require a threshold adjustment specific to the model, which makes them less flexible in different architectures.
Microsoft researchers, the University of Renmin of China, New York University and the University of Technology of Southern China have proposed a new method called Wina (informed weight of neurons) to solve these problems. Wina introduces a clear activation technique without training which uses both hidden state amplitudes and ℓ2 standards of the weight of weight matrices to determine the neurons to be activated during inference. Considering the combined impact of input amplitudes and the importance of weight, Wina creates a more efficient sparsification strategy that adapts to different layers of the model without requiring recycling or fine adjustment.
The Wina method is built on a simple but powerful idea: neurons that have strong activations and significant weight amplitudes are more likely to influence downstream calculations. To operationalize this, Wina calculates the product of elements of the hidden states and weight standards, by selecting the upper components according to this combined metric. This strategy allows Wina to build a sparse subnet which preserves the most important signals while ignoring redundant activations. The method also includes a transformation of the tensor which applies orthogonality by column in weight matrices, ensuring that the theoretical error limits are effectively translated into real performance. By combining these steps, Wina maintains a tight approximation error while offering significant calculation savings.
The research team has evaluated Wina on several models of large languages, including Qwen-2.5-7b, Llama-2-7b, Llama-3-8b and Phi-4-14b, on various tasks and levels of rarely. Wina surpassed the Sarcelle and cats on all the models tested and the rarity parameters. For example, on Qwen-2.5-7b to 65% of the rarity, Wina reached an average performance up to 2.94% higher than the Sarcelle and 1.41% better than the transformation of the Turchas. On LLAMA-3-8B, WINA delivered gains from 1.06% to 50% rarity and 2.41% to 65% of rarity. Even at high levels of rarity, Wina has retained stronger performance on stains with high intensity of reasoning like GSM8K and Arc Challenge. Wina also saved coherent calculation, reducing floating commas to 63.7% on LLAMA-2-7B and 62.7% on Phi-4-14B.
In summary, Wina offers a robust and without training solution for sparse activation in large -language models by combining hidden state amplitudes with weight matrix standards. This approach addresses the limits of previous methods, such as the Sarcelle, resulting in lower approximation errors, better precision and significant calculation savings. The work of the research team represents an important step in the development of more efficient LLM inference methods which can adapt to various models without requiring additional training.
Discover the Paper And GitHub page . All the merit of this research goes to researchers in this project. Also, don't hesitate to follow us Twitter And don't forget to join our 95K + ML Subdreddit and subscribe to Our newsletter.
Sana Hassan, consulting trainee at Marktechpost and double -degree student at Iit Madras, is passionate about the application of technology and AI to meet the challenges of the real world. With a great interest in solving practical problems, it brings a new perspective to the intersection of AI and real life solutions.
