Research has shown that important language models (LLMS) tend to overestimate information at the beginning and at the end of a document or a conversation, while neglecting the environment.
This “position bias” means that, if a lawyer uses a virtual assistant fueled by LLM to recover a certain sentence in a 30 -page affidavit, the LLM is more likely to find the right text if it is on the initial or final pages.
MIT researchers discovered the mechanism behind this phenomenon.
They created a theoretical framework to study how information crosses the automatic learning architecture which forms the backbone of LLMS. They found that certain design choices which control how the model processes the input data can cause a position bias.
Their experiences have revealed that the architectures of the model, in particular those affecting the way in which information is distributed over the input words in the model, can give rise to or intensify positioning biases, and that training data also contributes to the problem.
In addition to pivoting the origins of position biases, their frame can be used to diagnose and correct it in future designs of models.
This could lead to more reliable chatbots that remain on the subject during long conversations, to medical AI systems which reason more equitably when managing a data mine on patients and code assistants who pay particular attention to all parts of a program.
“These models are black boxes, so as a LLM user, you probably don't know that positions can make your model incoherent. You just feed your documents in any order you want and you expect it to work. But by understanding the underlying mechanism of these black box models, we can improve them decision-making systems (cover) and first author of a paper on this research.
His co-authors include Yifei Wang, a postdoc MIT; and the main authors Stefanie Jegelka, associate professor of electrical and computer engineering (CEE) and member of the IDSS and the computer and artificial intelligence laboratory (CSAIL); and Ali Jadbabaie, professor and chief of the civil and environmental engineering department, basic member of the teaching room of the IDSS and principal researcher in the lids. Research will be presented at the international conference on automatic learning.
Analyze attention
The LLMs like Claude, Llama and GPT-4 are fed by a type of neural network architecture known as the transformer. The transformers are designed to process sequential data, coding for a phrase in pieces called tokens, then learn the relationships between the tokens to predict what words come next.
These models have become very good on this subject because of the attention mechanism, which uses interconnected layers of data processing nodes to give meaning to the context by allowing tokens to concentrate selectively or to take care of related tokens.
But if each token can take care of all the other tokens in a 30 -page document, it quickly becomes intractable. Thus, when engineers build models of transformers, they often use attention masking techniques that limit the words that a token can assist.
For example, a causal mask only allows words to take care of those who preceded it.
Engineers also use position encodings to help the model understand the location of each word in a sentence, improving performance.
MIT researchers built a theoretical framework based on graphics to explore how these modeling choices, attention masks and position codings could affect position biases.
“Everything is coupled and tangled in the attention mechanism, so it is very difficult to study. The graphics are a flexible language to describe the dependent relationship between words in the attention mechanism and trace them on several layers, ”explains Wu.
Their theoretical analysis suggested that causal masking gives the model an inherent bias towards the start of an input, even when this bias does not exist in the data.
If the previous words are relatively unimportant for the meaning of a sentence, the causal masking can ensure that the transformer pays more attention to its beginning in any case.
“If it is often true that the previous words and the subsequent words of a sentence are more important, if an LLM is used on a task which is not a generation of natural language, such as the classification or recovery of information, these biases can be extremely harmful,” explains Wu.
As a model develops, with additional layers of attention mechanism, this bias is amplified because the previous parts of the input are used more frequently in the model reasoning process.
They also found that the use of position encodings to connect the words more strongly to the nearby words can mitigate the position bias. The technique refocuses the attention of the model in the right place, but its effect can be diluted in models with more layers of attention.
And these design choices are only a cause of position bias – some can come from training data that the model uses to learn to prioritize words in a sequence.
“If you know that your data is biased in a certain way, you also need your model in addition to adjusting your modeling choices,” explains Wu.
Lost in the middle
After establishing a theoretical framework, the researchers carried out experiments in which they systematically varied the position of the correct answer in the text sequences for a task of recovery of information.
The experiments showed a phenomenon “lost in the environment”, where the precision of recovery followed a mot in U. The models worked better if the correct answer was located at the start of the sequence. The performance has decreased, it approaches the middle before bouncing a little if the right answer was towards the end.
In the end, their work suggests that using a different masking technique, eliminating additional layers of the attention mechanism or the strategic use of positional coding could reduce position bias and improve the accuracy of a model.
“By making a combination of theory and experiences, we were able to examine the consequences of the design choices of models that were not clear at the time. If you want to use a model in high issues applications, you need to know when it will work, when it will not do it and why, ”explains Jadbabaie.
In the future, researchers wish to further explore the effects of positional encodings and study how positioning could be strategically exploited in certain applications.
“These researchers offer a rare theoretical lens in the attention mechanism at the heart of the transformer model. They provide a convincing analysis that clarifies the long -standing quirks in the behavior of the transformer, showing that attention mechanisms, in particular with causal masks, biament of biases to the start of the sequences. Saberi, professor and director of the Stanford University Center for Computational Market Design, who was not involved in this work.
This research is supported, in part, by the American Office for Naval Research, the National Science Foundation and a professor of Alexander von Humboldt.
