How Large Language Models "Think": Unveiling the Complete Process from Tokenization to Reasoning

Input text – The model receives a raw text string.
Tokenization – Breaking down the text into a sequence of tokens.
Token mapping to ID – Each token corresponds to an integer ID in the model's vocabulary.
Embedding – Using an embedding matrix to convert each token ID into a high-dimensional vector representation. This step enables the computer to numerically "understand" text semantics.
Model inference – The model (e.g., a Transformer neural network) receives these token vectors, passes them through multiple layers of computation, and produces prediction scores (logits) for the next token.
Softmax probability distribution – Converting logits through the Softmax function to obtain a probability distribution over all tokens in the vocabulary.
Sampling for generation – Selecting the next output token from the probability distribution according to a decoding strategy (such as greedy search, Top-k, Top-p, or temperature sampling). The selected token is then appended to the output sequence, and steps 5–7 repeat until a complete response is generated.

Below, we follow this "thinking" process step by step to reveal how large language models work from tokenization through reasoning to final text generation.

Text Tokenization: From Strings to Token Sequences

Why tokenize? Large language models cannot process raw text characters directly—they only understand numbers. Accordingly, input sentences must first be converted into numbers before being fed to the model. The first step in this conversion is tokenization. Simply put, tokenization is the process of splitting a string of text into smaller units called tokens. Each token typically corresponds to a word, a subword segment, or possibly even a character or punctuation mark, depending on the tokenization algorithm and vocabulary used. Tokenization transforms text into a token sequence.

How tokenization works. To efficiently represent different languages, modern LLMs commonly use subword tokenization algorithms such as Byte-Pair Encoding (BPE) or WordPiece. Taking BPE as an example, it builds a vocabulary by repeatedly merging frequent character sequences, making the vocabulary neither excessively large nor unable to represent common words or segments as single tokens. For instance, in English, a word like "international" might be split into subwords such as "inter," "na," and "tion." In Chinese, common phrases like "机器" and "学习" can also be represented as single tokens. The BPE tokenizer constructs its vocabulary by analyzing character pair frequencies in the training corpus and iteratively merging them into new tokens. This means the model's vocabulary contains both complete words and high-frequency subword or character segments.

Tokens and IDs. Regardless of the tokenization algorithm used, the result is a series of tokens. Each token is then mapped by lookup to a unique integer ID for the model to process. Think of the model's vocabulary as a massive "dictionary" containing all tokens the model can recognize along with their corresponding numbers. For example, GPT-2's vocabulary has roughly 50,257 entries. Every token—whether a complete word like "apple," a subword like "ap," or a punctuation mark—corresponds to an ID between 0 and 50,256.

A concrete example: Take the English input "I have a dream." Using GPT-2's BPE tokenizer produces the token sequence: ["I", " have", " a", " dream"] (where spaces are included as part of tokens). For the Chinese sentence "我有一个梦想," using a Chinese tokenizer might produce: ["我", "有", "一个", "梦想"] (depending on the vocabulary and algorithm). Each token is then mapped to a corresponding ID—for instance, "I" might map to ID 40 and " dream" to ID 14324 (these are illustrative). The key point about tokenization is that the model can only accept tokens within its fixed vocabulary. If the input contains a string not in the vocabulary (such as a rare word), the tokenizer will further split it into smaller known segments or handle it with a special unknown token marker.

After tokenization and ID mapping, we have successfully converted text into a numerical sequence. Next, we must convert these discrete numbers into vector form for convenient computation—this is token embedding.

Token Embedding: Enabling Computers to "Understand" Natural Language

After obtaining the token ID sequence, the model does not directly compute using these ID numbers. The reason is straightforward: using IDs directly (such as the number 14324) has no semantic meaning—the magnitude relationship between IDs does not reflect the relationship between word meanings. The model requires a representation that captures semantic information. Accordingly, the first layer of an LLM is typically an embedding layer that maps each token ID to a dense vector. We call this process token embedding.

What are embedding vectors? In brief, an embedding vector is a high-dimensional vector (typically hundreds of dimensions) representing a token. Unlike simple one-hot encoding, each dimension of an embedding vector is a continuous value. These vectors are learned during model training such that tokens with similar semantics have similar vector representations in the embedding space. In other words, embedding maps discrete symbols (words or characters) to a continuous space, allowing the model to use mathematics to "understand" relationships between them. For example, in a well-trained embedding space, vectors representing "king" and "queen" might lie close together, and the relationship "king − man + woman" ≈ "queen" can be approximately captured through vector arithmetic—this demonstrates the power of embeddings in capturing semantics.

The embedding process. The embedding layer can be viewed as a lookup operation: it maintains a matrix $E$ of shape $(|V|, d)$, where $|V|$ is the vocabulary size and $d$ is the embedding dimension (such as 512, 768, or higher). When a token has ID $i$, the embedding layer outputs row $i$ of matrix $E$, denoted $E[i]$. For example, the token sequence "I have a dream" is converted to a set of vectors $[\mathbf{e}{I}, \mathbf{e}{have}, \mathbf{e}{a}, \mathbf{e}{dream}]$. These vectors typically consist of floating-point values like [0.12, −0.45, ...]; in high-dimensional space, each vector may not be directly intuitive, but the model leverages them for subsequent computation.

After the embedding layer, the original text has become a series of vectors that the model can "digest." As noted in the Hugging Face blog, these embedded vector sequences are the true input to the neural network. With them, the model can then perform complex reasoning calculations on text meaning in continuous vector space. This step enables computers to mathematically "understand" natural language, laying the foundation for subsequent reasoning.

Model Inference: Contextual Understanding and Next-Token Prediction

Once we have the token vector sequence, these vectors are fed into the language model for inference computation. Current mainstream LLMs (such as GPT series, BERT, etc.) employ the Transformer architecture. The Transformer comprises multiple layers of self-attention mechanisms and feedforward neural networks, allowing efficient modeling of long-range dependencies in sequential data. Although this article does not focus on Transformer details, understanding its role helps explain how the model "thinks":

Self-attention allows the model to dynamically focus on relationships between different tokens in the sequence at each layer. For instance, when processing a sentence, attention mechanisms let words "see" other relevant words in the sentence, adjusting their own representation in context.
Multiple stacked layers and nonlinear transformations allow the model to progressively abstract higher-level semantic representations. The initial input consists of word embedding vectors; after passing through the first attention and feedforward layer, we obtain new vector representations incorporating contextual information. As computation progresses through layers, semantic information is gradually aggregated and refined.

After multiple layers of Transformer computation, the model produces its prediction for the next token at the output layer. Specifically, the model does not directly output a word or sentence but rather outputs a score for each token in the vocabulary—what we call logits. Logits can be understood as "unnormalized scores": higher-scoring tokens are more likely to be the next word according to the model, while lower scores indicate lower likelihood.

For example, if we ask the model to continue after "I have a dream," the final layer might compute scores for tens of thousands of tokens such as "of," ",", "..." A high score for "of" indicates the model has largely completed its "next-token prediction" work, but these logits must still be converted to probability form for final output decisions. Note: This process of assigning scores to each word repeats at each step; after generating one token, it is added to the input for the next prediction. This is the fundamental way autoregressive language models generate text: predicting the probability distribution of the next word step by step.

Probability Distribution and Sampling: How Models Choose the Next Token

Converting to probability with Softmax. After the model obtains scores for each token, it must convert them to intuitive probabilities through the Softmax function. Softmax considers scores for all words, exponentiates and normalizes them to produce a probability distribution in which probabilities for all possible tokens sum to 1. The Softmax formula is generally expressed as:

$$ P(w_i) = \frac{\exp(\text{logit}i)}{\sum{j}\exp(\text{logit}_j)} $$

where $P(w_i)$ is the model's estimated probability that token $w_i$ is the next word. Continuing the "I have a dream" example, if Softmax calculation yields a 17% probability for the token "of," we write: $P(\text{"of"} \mid \text{"I have a dream"}) = 0.17$. The higher the probability of a token, the more the model considers it a fitting continuation given the context.

Greedy or random? Given the probability distribution for the next word, one might intuitively select the highest-probability token as output (known as greedy search or greedy decoding). The greedy strategy selects the most likely word at each step; it is straightforward and, if we always pick the highest probability, identical inputs always produce identical outputs (completely deterministic output). This sounds good initially, but greedy selection often causes models to produce monotonous or mechanically repetitive content. Consider a person who always chooses the most likely word without variation—the speech would sound extremely dull. Similarly, if a large language model always picks the highest-probability word, the generated text may lack variety and risk falling into repetitive loops. Consequently, when generating longer texts, we typically introduce controlled randomness to achieve more diverse and natural results.

Sampling strategies introducing randomness. To make model outputs more flexible and varied, the field has developed multiple sampling strategies that introduce random selection while maintaining coherent text. Major strategies include temperature sampling, Top-k sampling, and Top-p (nucleus) sampling. Their common principle is: rather than rigidly always picking the maximum probability word, give several top-ranked candidate words a chance of being selected. Below we briefly explain each strategy:

Temperature sampling: The temperature parameter affects how "sharp" the probability distribution is. The temperature $T$ appears in the formula as dividing logits by $T$ before applying Softmax. When $T$ is higher (e.g., $T = 1.0$ is the default), the probability distribution is smoother, and the model has greater opportunity to select words with originally lower but nonzero probabilities. Conversely, when $T$ is very low (approaching 0), Softmax makes the highest-probability word monopolize nearly all probability mass, making output nearly equivalent to greedy selection. Therefore, higher temperature produces more random output, while lower temperature produces more conservative and determined output. Typically, if the same prompt is used multiple times, you find the model gives somewhat different answers each time; but if temperature is set to 0, you get exactly the same answer every time. This is because $T = 0$ effectively always selects the highest-probability token (complete elimination of randomness).
Top-k sampling: Top-k considers only the $k$ most probable candidate tokens. The model originally produces a probability distribution over the entire vocabulary, but many words have such low probability they are essentially impossible; we need not consider them. By setting a value $K$ (say, $K = 5$), we take the five most likely tokens, then randomly sample one from them according to their relative probabilities. This ensures we only select from words the model is fairly confident about, avoiding strange low-probability words, while introducing some variation compared to picking only the highest. The impact of parameter $K$ is: larger $K$ gives the model output more diversity, but may introduce lower-quality words; smaller $K$ yields more stable output but may be overly conservative. In the extreme, $K = 1$ reduces to greedy decoding (selecting the single most likely token each time).
Top-p sampling (nucleus sampling): Top-p sampling does not fix how many words to select but instead sets a cumulative probability threshold $p$. It accumulates probability starting from the highest, and once the cumulative probability exceeds $p$, it takes all candidates within that range as the selection set. In other words, Top-p dynamically determines the size of the candidate set to ensure the total probability of these candidates is at least $p$. For example, if $p = 0.15$ is set, the model might need to include the first two or three words (reaching 15% cumulative probability with words such as "United" and "Netherlands"), then randomly select one from them according to relative probabilities. Top-p's advantage is that regardless of distribution shape, it ensures considering sufficient probability mass. When the distribution is highly skewed, candidates may be few; when the distribution is flatter, candidates increase accordingly. This produces more flexible results than Top-k and is commonly used to increase diversity in generated text.

It is important to note that these strategies can be combined. For example, we often set both a higher $p$ (such as 0.9) and a moderate $k$ (such as 50); the model first selects Top-k, then applies Top-p constraints within that set, creating a "double filter." Tuning these parameters balances text quality and diversity—overly low temperature or $p$ makes output highly determined but potentially dull, while excessively high values may cause the model to output irrelevant or even absurd content.

Why different outputs for the same input? Based on these sampling mechanisms, we can now understand: if random sampling is introduced, each generation may draw different words from high-probability candidates, so even with identical prompts, the model's responses may differ. This does not mean the model is "unreliable" but rather is normal for generative tasks—fundamentally, LLMs generate probabilistically-driven plausible answers, not a single determined answer. Only under extremely deterministic configurations, such as pure greedy search with temperature 0, will identical inputs always produce identical outputs. In fact, conversational systems like ChatGPT can give somewhat different responses to the same question precisely because they employ temperature sampling and similar mechanisms to avoid generating the same response every time. This randomness stems from sampling strategies during model inference, not from the model "forgetting" previous responses.

A small example: Suppose the model's current context is "the weather today" and it predicts the next word might be: "very nice" (40% probability), "bad" (30%), "okay" (25%), and other words (5% combined). Using greedy strategy, the model selects "very nice" with the highest probability, and the output might be "the weather today is very nice." But with sampling strategy, the model has some chance of selecting "bad" or "okay," potentially outputting "the weather today is bad" or "the weather today is okay." This creates different sentence directions. Such random sampling makes the model's output richer and prevents monotony. Of course, if we want to ensure consistent responses, we can lower the temperature or fix the random seed to obtain reproducible results.

Explanation from an Engineering Practice Perspective

Having understood these principles, let's examine how to apply this knowledge in engineering practice. Usually we employ existing deep learning frameworks and model libraries (such as Hugging Face Transformers) to handle tokenization and text generation. These libraries have already encapsulated most details, letting us call them conveniently:

Using a tokenizer: Each pretrained model comes with a corresponding tokenizer. Developers first load the tokenizer, then use its encode or __call__ method to convert input text to a token ID list. For example, in Transformers:

from transformers import AutoTokenizer  
tokenizer = AutoTokenizer.from_pretrained("gpt2")  
ids = tokenizer("I have a dream", return_tensors='pt').input_ids

This gives us the corresponding token ID sequence. In real applications, ensure you use the same tokenizer and vocabulary as the model was trained with.

Embedding and model forward pass: For users, this step is typically hidden inside the model. After loading the model, we pass the token IDs tensor obtained in the previous step to the model to perform forward inference and get the output. For example, in Transformers:

from transformers import AutoModelForCausalLM  
model = AutoModelForCausalLM.from_pretrained("gpt2")  
output = model.generate(input_ids=ids, max_new_tokens=50, temperature=0.8, top_p=0.9)

The generate method handles model forward computation, Softmax, and sampling strategies internally. We only need to control sampling strategies through parameters like temperature, top_p, and top_k. In the code above, temperature 0.8 and Top-p 0.9 are set for text generation. Setting temperature=0 and top_p=1.0 (or do_sample=False) would instead use greedy generation for deterministic output.

Decoding results: The model generates a sequence of token IDs that we must convert back to readable text using the tokenizer's decode method. For example:

result_text = tokenizer.decode(output[0], skip_special_tokens=True)  
print(result_text)

This yields the model's final generated natural language sentence.

In engineering practice, several other considerations deserve attention: setting maximum length and end-of-sequence tokens to prevent infinite generation; truncating or padding to fit the model's context window size; and so on. Choices about these hyperparameters significantly affect model performance and generation quality. In practice, parameter tuning is often necessary to find a good balance between output quality and diversity.

Conclusion

Through the steps analyzed above, we have unveiled the complete process of large language models from tokenization through embedding to reasoning and generation. In essence, an LLM's "thinking" is: converting input into numerical representation, computing the probability distribution of the next word within a massive neural network based on knowledge learned during training, selecting output according to a particular strategy, and thus progressively generating text responses.

This generation mechanism allows models to produce coherent and semantically rich language, yet it also determines that fundamentally they are probabilistic models. We have corrected the misconception that "identical inputs must produce identical outputs" and understood the role of random sampling in generation. We have also emphasized the importance of embedding vectors in enabling the model to "understand" language: without appropriate vector representations, computers cannot process human language signals mathematically.

For developers, understanding these principles helps optimize model behavior more effectively. For example, knowing how to adjust temperature allows you to control the randomness of responses; understanding tokenization mechanisms helps avoid unnecessary length increase or unexpected token splits. As the Hugging Face blog notes, the remarkable performance of large language models is grounded in definite logic and mathematical foundations. We can both leverage their powerful pretrained knowledge and use engineering techniques to precisely control their output.

The development of large language models continues, with abundant research emerging in improved tokenization methods and more sophisticated sampling strategies. We hope this article has provided you with a clear panoramic perspective, helping technical practitioners understand how LLMs work more deeply and offering some guidance for practical applications and optimization. In the future, we have reason to believe that as our understanding of these mechanisms deepens, we will be able to construct more efficient and more intelligent language models.