From output prediction to the probabilistic map: autoregression, self-attention and embeddings

Ask a fraud model about a payment and it answers with one of two labels it was trained to tell apart. Ask a language model to continue "the detective" and it does not answer with a label at all: it answers with a distribution — opened is likely, found is plausible, sang is unlikely but not impossible — and then commits to one of them, moves on, and asks the question again with one more word in the context. That shift, from a mapping between input and expected output to a map of probabilities over what could come next, is what this article is about, together with the three mechanisms that make the map usable: autoregression, self-attention and embeddings.^[4]

1. From a mapping to a map of probabilities

Traditional AI, as defined in the previous article, learns a mapping between an input and its expected output. Generative AI changes the focus: instead of looking only for the correct answer to an input, it learns probability distributions over patterns, sequences and possible combinations. That learning builds a probabilistic map of what tends to make sense in a given context. The result is not a choice among known labels but a navigation through plausible paths to produce a new, coherent output.^[4,5]

The difference shows in what the model returns. A classifier returns one point of a closed set. A generative model returns, at every step, a whole distribution over the next piece — and the output is the path it walks through those distributions. Figure 1 puts the two side by side.

Practise: open in the Lab: Exercise 1, Map versus mapping

2. Underlying patterns and coherent generation

Training on large volumes of data serves to capture the implicit structures that organise information, even when they are not explicitly labeled. The model learns regularities of language, style, association and continuity, which raises its ability to tell probable sequences from sequences that are foreign to the context.^[5] That is the basis of coherent generation: the output seems to make sense because it respects recurring patterns learned during training. Nobody wrote down that a sentence in English tends to put the verb after the subject, that a recipe lists ingredients before steps, or that a function's closing brace comes after its body; the model absorbed those regularities from millions of examples in which they hold.

The same fact explains the failure mode. The better the training, the higher the chance that the model produces something consistent and the lower the chance that it drifts into a hallucination — an output that follows the patterns of plausible text without being anchored in anything true.^[7] A hallucinated citation has the shape of a citation: authors, a title, a year, a venue. The map says "this is what a citation looks like here"; it does not say "this citation exists". Coherence and truth are different properties, and only the first one is what the map optimises.

3. Two analogies: the artist and the cartographer

The artist analogy reduces the abstraction of the process. An artist observes many works, internalises styles, compositions and relations, and then produces something new without literally copying a specific piece. The generative model operates in a similar way: it uses a learned statistical repertoire to compose an original output conditioned on the context it received. Originality arises from the coherent recombination of patterns, not from creation out of nothing.

The cartographer analogy explains the map itself. A cartographer does not only record the mountains, rivers and forests already seen; they learn the logic of the terrain and start to infer where new elements could exist. The model likewise does not merely memorise examples: it learns the structure of the space of possibilities. When asked to generate something similar to what it knows, it projects a new configuration compatible with the regularities it observed. That is how it can draw an "imaginary island" coherent with the islands it has seen — a harbour on the sheltered side, a river coming down from the highlands, a village at the river mouth — even though that particular island never existed.

Table 1 — The two analogies describe the same object from two sides: the artist stresses recombination, the cartographer stresses the structure that makes a never-seen output coherent.

Practise: open in the Lab: Exercise 2, The cartographer

4. Autoregressive models: one token at a time

Autoregressive models generate content token by token, choosing each next element on the basis of the previous ones.^[4,5] In text this means that each word produced changes the context available for the next decision, forming a chain of local dependencies. Formally, the probability of a whole sequence is the product of the probability of each token given everything before it:

Take the scenario of generating a novel in the style of a famous detective series. The model uses the learned repertoire of style and structure — a crime, a list of suspects, a clue the reader overlooked, a reveal in the last chapter — but the concrete construction happens step by step during generation. Coherence emerges because each new choice must be compatible with what has already been written: once the detective has opened something, the next token is very probably something that can be opened. Figure 2 shows one step of the chain.

Practise: open in the Lab: Exercise 3, Softmax

5. Autoregression as a process of writing

The writer analogy shows why autoregression works well for language. A writer does not materialise the whole text at once; they write a sentence, use that sentence to guide the next one, and adjust course as the narrative advances. The model does the same at statistical scale: it keeps autocompleting itself until the sequence is done. This mechanism explains both the fluency of generation and the possibility of drift: when the chain starts to follow a path that is plausible but incorrect, every later step is conditioned on that error and tends to make it look consistent rather than to fix it.^[6]

• Fluency. Each token is chosen to fit the ones already written, so the surface is smooth by construction.
• Drift. A single unlikely choice — a wrong name, an invented number — becomes context for everything after it. The map has no separate notion of "true"; it only has "consistent with what came before".
• Why review stays mandatory. Reading the output is the only step in which something other than the map judges it.

6. Self-attention: keeping the relevant parts of the context

Attention and context were introduced before; the new point here is self-attention as the way of preserving relevant relations along the sequence. Self-attention lets the model weigh which earlier parts of the text matter more for interpreting or generating the next token.^[1] In a long text this helps the model keep links between distant elements — a character introduced at the beginning and picked up many paragraphs later. Without this mechanism, generation would lose continuity far more easily, because a model that only looked at the last few tokens could not know who she is.

In practical terms, self-attention works as a system of selective focus over the available history. The model does not treat all previous tokens as equally important; it assigns larger weights to the relations most useful for the current decision. Each position builds a query, compares it with the keys of every position before it, turns the comparisons into weights that sum to one and uses them to blend the values:

Back to the book: this is what lets the model sustain the identity of the main character and the logic of the narrative even when many events happen in between. The pronoun on page two hundred attends to the name on page one, and the clue being examined attends to the letter opened three sentences earlier. Figure 3 shows the weights for one such decision.

Practise: open in the Lab: Exercise 4, Selective focus

7. Embeddings: from symbols to vectors

Before operating on words, images or code, the model converts those elements into numeric vectors called embeddings.^[2,4] This vector representation turns symbolic items into points of a mathematical space where relations can be measured computationally. The model does not "read Portuguese" or "read English" the way a human does; it processes numeric patterns that encode semantic and contextual properties. That is what makes it possible to compare meaning by geometric proximity between vectors.

The embeddings are not designed by hand. They are learned together with the rest of the model, adjusted so that words used in similar ways end up with similar vectors — which is exactly the property the probabilistic map needs, because a distribution learned for one context can then apply to another context that is close in this space even if the words differ.^[4] Real models use hundreds or thousands of dimensions; Figure 4 uses two so that the geometry can be seen.

8. Semantic similarity by vectors

When two terms appear in similar contexts, their vectors tend to end up close in the semantic space; when they appear in very different contexts, they tend to end up far apart.^[2,10] That is why pairs such as Paris and Rome occupy nearby regions, while Paris and spoon usually sit further apart. The analogy with geographic coordinates helps: vector proximity works like proximity of location, but in a space of meaning. The standard measure is the cosine of the angle between two vectors:

Equation (3) gives 1 for vectors pointing the same way, 0 for unrelated ones and −1 for opposites, regardless of their length. Equation (4) is the famous consequence: directions in the space carry relations, so subtracting France from Paris and adding Italy lands near Rome.^[3] This structure is one of the bases on which the model recognises relations, keeps context and chooses more plausible continuations.

The whole of Figure 4 fits in a few lines. The vectors are the ones plotted; the outputs in the comments are what the code prints:

import math

vec = {                                   # toy embeddings in two dimensions; real models use hundreds
    "paris":  (0.35, 0.85),  "rome":  (-0.35, 0.85),
    "france": (0.45, 0.10),  "italy": (-0.45, 0.10),
    "spoon":  (0.05, -0.60), "fork":  (-0.05, -0.55),
}

def norm(u):
    return math.sqrt(sum(a * a for a in u))

def cosine(u, v):
    # equation (3): 1 = same direction, 0 = unrelated, -1 = opposite; length does not matter
    dot = sum(a * b for a, b in zip(u, v))
    return dot / (norm(u) * norm(v))

def neighbours(word, k=2):
    others = [w for w in vec if w != word]
    return sorted(others, key=lambda w: cosine(vec[word], vec[w]), reverse=True)[:k]

def analogy(a, b, c):
    # equation (4): a - b + c, then the closest word that is not one of the three
    target = tuple(x - y + z for x, y, z in zip(vec[a], vec[b], vec[c]))
    candidates = [w for w in vec if w not in (a, b, c)]
    return max(candidates, key=lambda w: cosine(target, vec[w]))

print(neighbours("paris"))                # ['rome', 'france']
print(neighbours("rome"))                 # ['paris', 'italy']
print(round(cosine(vec["paris"], vec["spoon"]), 2))   # -0.89: far apart
print(analogy("paris", "france", "italy"))   # rome

Practise: open in the Lab: Exercise 5, Cosine and analogy

9. Conclusion

A generative model does not store answers; it stores a map. Training on large volumes of data turns the implicit structure of language, code and images into probability distributions over what tends to make sense in a given context. Autoregression walks that map one token at a time, which is why the output is fluent and why it can drift; self-attention decides which parts of the history matter for each step, which is why a long text keeps its characters and its thread; embeddings give the map its coordinates, so that meaning can be compared as distance and relations as direction. Keep the cartographer in mind: the model infers terrain it never saw, coherent with terrain it did see — and only a reader can check whether the city is really there.