Meta AI researchers have introduced a self-regressive U-Net model of evolutionary bytes which surpasses transformers based on tokens through the modeling of the language modeling

Language modeling plays a fundamental role in the treatment of natural language, allowing machines to predict and generate text that resembles human language. These models have evolved significantly, starting with statistical methods and progressing through neural architectures to systems based on today's transformers. At the center of many applications, such as chatbots, translation tools and text completion engines, language models interpret and generate sequences of words or bytes. Their efficiency largely depends on the underlying architecture and the representations of the data used. As the demand for more efficient and scalable models increases, researchers continue to explore new structures and training methods to improve performance, manage longer contexts and reduce the calculation load. Among these efforts, the combination of convolutionary architectural ideas with an autoregressive prediction has become an intriguing approach.

Challenges with token -based language models and transformers

One of the main problems of language modeling is the excessive use of token models and models of transformers, which are expensive in calculation and generally ineffective for treatment at the byte level or even between languages. Techniques such as the pair of bytes coding for the lengths of control sequence but create inconsistencies between languages ​​and domains. The transformers, although precise, lack scalability due to their quadratic complexity. Although competing approaches, such as clear attention, try to solve this problem, they generally do so to the detriment of simplicity or performance. Modeling at the byte level with flat transformers has only demonstrated partial success, highlighting the need for new architectures that can treat the inputs of raw bytes without tokenization while achieving excellent performance.

Presentation of Au-Net: a tongue model in the bytes without a token

Researchers from Fair in Meta, Tau, Inria and Lisn, CNRS & Université Paris-Saclay, INSA Rouen Normandy, Litis, Rouen, France, presented a new U-Net (Au-Net). This model incorporates the ideas of the conceptions of Net in U Convolutionalal with self -regressive decoding processes. Unlike transformer systems, Au-Net does not require tokenization and operates directly on bytes. Architecture is designed to allow a parallel and effective generation, the autonomy to incorporate self -regressive capacities. He succeeded by coding for downward convolutions, then increased sampling stages, which restore the original sequence size. In particular, Au-Net presents a division mechanism which makes it possible to carry out predictions on the sequence of the sequence, improving scalability. This design change also guarantees that the complexity of the model increases linearly with the sequence length, rather than quadratic. The researchers have deployed this model through several landmarks of languages ​​and multilingual tasks to test its effectiveness in parameters with low resources and large -scale.

Architecture Au-Net: Coding with several scales and parallel inference

Au-clean architecture is implemented with stages with several scales which reduce and then rebuild the entry sequences using convolutions with progress. During training, each segment of the entry sequence is predicted in a masked manner to maintain the self -regressive property. The model uses a division function learned to divide the input sequences into non -horse groups, which are then planned simultaneously and combined in a full output. It supports configurations that are both shallow and deep, with models ranging from 3% to 75% of the training calculation budget compared to standard basic lines. For example, a configuration formed on 200B tokens with 8 billion parameters obtained very competitive results. Another version, formed on 60 billion tokens with a billion parameters model, obtained a 35.7 blue score on standard translation tasks, outperforming basic models formed on the same data. In addition, Au-Net has demonstrated faster generation speeds due to its parallel decoding, offering a significant advantage for latency sensitive applications.

Reference results show a competitive advantage over transformers

Experimental results have shown high performance in a wide range of tasks. On ENWIK8, a compression reference at the byte level, Au-Net reached 1.01 bits per byte, exceeding a basic line of the transformer which has only reached 1.02 bits per byte. On PG-19, a long-context language modeling task, the model reached 2.61 bits per byte against 2.75 from standard transformers. Au-Net has also evolved effectively in calculation budgets, reaching 43.3 blue on the Flores-200 translation with an 8B model size formed on 200B tokens. In a multilingual assessment using Flores-200, the model has outperformed token transformers through pairs of languages ​​with low resources. It has also demonstrated better inter-lingual generalization in language families, reaching a blue score of up to 33.0 in several configurations. When evaluated in equal calculation and data budgets, Au-Net has equaled or surpassed transformers, generation speeds improving from 20% to 30% in certain contexts.

Key contributions and information on the yield of Au-Net

• Au-Net eliminates the need for token by operating directly on the inputs of raw bytes.
• On Enwik8, Au-Net obtained a BPB of 1.01, exceeding the baselines of the transformer with 1.02 BPB.
• On PG-19, it reached 2.61 BPB, improving the 2.75 BPB of standard transformers.
• The multilingual evaluation of Flores-200 showed 33.0 blue blue systems, outperforming tokens.
• The models at the bytes formed with Au-Net have maintained high performance on the parameters with high resources and low resources.
• The generation speed improved from 20% to 30%, supporting rapid and parallel inference.
• The laws of enhancement held; Performance has improved with increased size and model data.
• The model has shown better cross -generalization and noise robustness.
• Effective use of calculation; Au-Net has paired or exceeded the performance of the transformer to lower calculation budgets.
• Au-Net is a viable alternative for large-scale language modeling tasks, including multilingual applications and by bytes.

Conclusion: the practical advantages and the scalability potential of Au-Net

In conclusion, the researchers provided detailed scaling analyzes showing that the Au-Net adheres to the predictable hyperparameters. It benefits from an increase in the size of the model and training tokens in a consistent manner with the practices observed in the models of transformers. For example, in calculation training parameters, Au-net performance has improved regularly with an increased given ratio, corresponding to the gains observed in the processor of the transformer. Above all, Au-Net was able to move to models with 8 billion parameters, demonstrating effective training and showing that architecture is capable of supporting high-capacity systems. In extended evaluations, the model has maintained its effectiveness when applied to downstream tasks, showing strong performance in the language generation, translation and prediction references at the level of bytes. Au-Net has also proven easier to train and more robust in noisy input conditions compared to token-based models.

Why is this research important?

This research is important because it calls into question long-standing dependence on the language models based on the tokens by introducing au-clean, an autoregressive architecture at the byte level which eliminates the general tokenization costs while achieving competitive or higher performance. By dealing with raw bytes directly and at the scale effectively with linear complexity, Au -Net deals with the key limits of transformer models – namely their quadratic scale and dependence on fixed vocabularies. Its strong results between multilingual and long context benchmarks, in particular in low -resources parameters, highlight its potential for creating more efficient, inclusive and generalized NLP systems. This positions at a promising alternative for future large-scale language modeling efforts.

---

Discover the Paper And GitHub page. All the merit of this research goes to researchers in this project. Also, don't hesitate to follow us Twitter And don't forget to join our Subseubdredit 100k + ml and subscribe to Our newsletter.