How the scalar product of vectors became the foundation of modern AI

In 2017, Google researchers published a paper with a deliberately modest title—Attention is All You Need. There were no grandiose announcements or robot demonstrations. However, it was this very paper that drew a clear line in the history of neural networks: “before” and “after.”

Today, ChatGPT, Claude, Gemini, Midjourney, and virtually all modern generative AI are based on the transformer architecture described in that paper.

The Wall That Old Neural Networks Hit

Before transformers, recurrent neural networks (RNNs) and their improved variants—LSTM and GRU—dominated the field. They processed text sequentially—word by word—while storing “memory” of previous tokens in a hidden state.

The problem was obvious:

• Long-range dependencies were lost—the network “forgot” the beginning of a sentence by the time it reached the end;
• sequential processing did not allow for parallelizing computations;
• training on large datasets became a nightmare.

The industry hit a ceiling in both performance and quality.

A Revolution Straight Out of a Linear Algebra Textbook

The solution turned out to be surprisingly simple. At the heart of the self-attention mechanism lies the scalar product of vectors—an operation taught in introductory mathematics courses.

The essence of the idea: each word in a sentence is compared to every other word through the scalar product of their vector representations. The higher the result, the stronger the connection between the words. This allows the model to “see” the entire context at once, without losing connections between distant parts of the text.

This made it possible to:

• process all tokens in parallel, rather than one at a time;
• detect dependencies between words at any distance;
• scale models up to billions of parameters without fundamental architectural changes.

Why Simplicity Proved to Be a Strength

This is the main paradox of the Transformer revolution. For years, the industry had been searching for a breakthrough in complexity—biologically plausible models, exotic memory mechanisms, and non-standard inference logic.

But what took off was an operation that can be explained to a schoolchild: multiply numbers and add the results.

Scalar multiplication scales well on GPUs, is easily differentiated during backpropagation, and intuitively measures the “similarity” of vectors. It is precisely this combination of properties that made it the ideal building block for an attention mechanism.

What This Means Today

Transformers have gone far beyond text processing:

• GPT-4, Claude, Gemini — language models;
• Stable Diffusion, Midjourney — image generation;
• AlphaFold — protein structure prediction;
• systems for processing code, audio, and video — transformers are everywhere.

An architecture born from a single linear algebraic operation has become the universal language of modern machine learning. And this is perhaps the best proof that, in science, elegant simplicity often trumps sophisticated complexity.

Source: Habr