Attention Mechanism
The mechanism that determines which parts of the input will focus more on when producing an output of the model.
The attention mechanism (attention Mechanism) is one of the most effective innovations of today’s artificial intelligence architectures, dynamically determines how much the input will focus on what parts of the input when producing an output of a model. The idea was first revealed in the machine translation: when translating a sentence to another language, in order to make equal attention to all words in each word source sentence in the output, it should show more "found" to certain words that are relevant. Technically, the mechanism throws a varying weight according to binding to each input element, these weights are calculated by calculation of similarity between query (query), key (key) and value (value) vectors and converted to a potential distribution, which is equal to the softmax function.
The attention mechanism solved the problem with compression the previous recurrent nerve networks (RNN) to a fixed-sized "eous" vector; no longer the model can access directly and directly when producing a word at the beginning of a long text. This ability forms the core of the Transformer architecture and provides consistent protection of the context in long documents, code files or multi-step conversations. If there was no attention mechanism, the capacity of today’s large language models to establish relationships between long texts and produce consistent, binding sensitive outputs would not be possible.
The attention mechanism solved the problem with compression the previous recurrent nerve networks (RNN) to a fixed-sized "eous" vector; no longer the model can access directly and directly when producing a word at the beginning of a long text. This ability forms the core of the Transformer architecture and provides consistent protection of the context in long documents, code files or multi-step conversations. If there was no attention mechanism, the capacity of today’s large language models to establish relationships between long texts and produce consistent, binding sensitive outputs would not be possible.