Explaining the Self-Attention Mechanism in LLMs

The Self-AttentionO Mecanismo de Atenção (especificamente o Self-Attention ou Autoatenção) é a inovação matemática que tornou as LLMs viáveis. Antes dele, as redes neurais processavam texto de forma sequencial, esquecendo o início da frase ao chegar no final. O mecani... More mechanism is the true heart of the Transformer architecture, the foundation behind every modern Large Language Model (LLM).

Before Self-AttentionO Mecanismo de Atenção (especificamente o Self-Attention ou Autoatenção) é a inovação matemática que tornou as LLMs viáveis. Antes dele, as redes neurais processavam texto de forma sequencial, esquecendo o início da frase ao chegar no final. O mecani... More was introduced in the landmark paper “Attention Is All You Need” (Vaswani et al., 2017), language models processed text sequentially, word by word (or tokenTokens são as unidades básicas de informação que uma LLM processa. Eles não são necessariamente palavras inteiras; podem ser sílabas, caracteres ou partes de palavras. A tokenização é o processo de transformar o texto bruto em uma sequência desses to... More by tokenTokens são as unidades básicas de informação que uma LLM processa. Eles não são necessariamente palavras inteiras; podem ser sílabas, caracteres ou partes de palavras. A tokenização é o processo de transformar o texto bruto em uma sequência desses to... More), using Recurrent Neural Networks (RNNs). This approach caused two major issues: loss of long-range context and an inability to parallelize training.

Self-AttentionO Mecanismo de Atenção (especificamente o Self-Attention ou Autoatenção) é a inovação matemática que tornou as LLMs viáveis. Antes dele, as redes neurais processavam texto de forma sequencial, esquecendo o início da frase ao chegar no final. O mecani... More solved both problems at once by allowing every word in a sequence to connect directly with every other word, evaluating context and relevance in parallel.

What is Self-Attention?

In simple terms, Self-AttentionO Mecanismo de Atenção (especificamente o Self-Attention ou Autoatenção) é a inovação matemática que tornou as LLMs viáveis. Antes dele, as redes neurais processavam texto de forma sequencial, esquecendo o início da frase ao chegar no final. O mecani... More allows a model to calculate how much attention a specific word should pay to all other words in the same sentence to properly understand its meaning.

Consider the sentence:

“The bank of the river was covered in green grass.”

If the model evaluates the word “bank” in isolation, it could mean a financial institution or the side of a river. However, by running the Self-AttentionO Mecanismo de Atenção (especificamente o Self-Attention ou Autoatenção) é a inovação matemática que tornou as LLMs viáveis. Antes dele, as redes neurais processavam texto de forma sequencial, esquecendo o início da frase ao chegar no final. O mecani... More mechanism, the word “bank” calculates high relevance scores with “river” and “grass”, allowing the model to instantly disambiguate its meaning in context.

The Query, Key, and Value (Q, K, V) Paradigm

To understand the mathematics behind Self-AttentionO Mecanismo de Atenção (especificamente o Self-Attention ou Autoatenção) é a inovação matemática que tornou as LLMs viáveis. Antes dele, as redes neurais processavam texto de forma sequencial, esquecendo o início da frase ao chegar no final. O mecani... More, think of the process as a database search query:

Query (Q): Represents the current word seeking context (“What am I looking for?”).
Key (K): Represents every word in the sentence advertising its content (“What information do I hold?”).
Value (V): Represents the actual contextual content of each word that will be extracted once a match is found.

For every word in the input sequence, the model projects its vector representation into three vectors (Q, K, and V) using trainable weight matrices ( $W^Q, W^K, W^V$ ).

The Self-Attention Equation

The entire process is condensed into a single mathematical formula known as Scaled Dot-Product Attention:

\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{Q K^T}{\sqrt{d_k}}\right)V

Step-by-Step Breakdown

1. Compute Raw Attention Scores ( $Q K^T$ )

The model performs a dot product between the Query vector of the target word and the Key vectors of all words in the sentence. Higher dot product values indicate higher semantic alignment and relevance.

2. Scaling ( $\sqrt{d_k}$ )

The result is divided by the square root of the dimension of the key vectors ( $\sqrt{d_k}$ ). This prevents the dot products from growing excessively large for high-dimensional vectors, which would push the Softmax function into regions with near-zero gradients (slowing down model training).

3. Softmax Normalization

The Softmax function converts the scaled scores into a probability distribution (values between 0 and 1 that sum up to 1). These values represent the attention weights—the exact percentage of focus assigned to each word.

4. Weighted Combination with Values (V)

Finally, each Value vector (V) is multiplied by its corresponding attention weight, and the results are summed. The output is a new context-enriched vector representation for the word.

Why Is This Revolutionary?

Parallel Execution: Unlike RNNs, which must process tokens step-by-step, Self-AttentionO Mecanismo de Atenção (especificamente o Self-Attention ou Autoatenção) é a inovação matemática que tornou as LLMs viáveis. Antes dele, as redes neurais processavam texto de forma sequencial, esquecendo o início da frase ao chegar no final. O mecani... More computes connections across all words in a single matrix operation, unlocking massive GPU training speedups.
Direct Connections: The distance between any two words in a sequence becomes 1 step, completely eliminating the forgotten context problem over long texts.