We are now intertwined with large language models (LLMs) — ChatGPT, Gemini, Claude, and Llama to name just three, marvelled at their ability to understand, generate and converse in human language. But what’s actually going on under the hood? While the complete story involves advanced mathematics and vast swathes of data, we can dissect the core functions of these AI marvels.
- Tokenization: Language Breakdown While raw text is being input, the LLM needs to see what we are getting at. Computers read numeric information, so we must convert the text. Tokenization is the practice of breaking up a sentence or passage into smaller units, or ‘tokens’. Tokens can be whole words, or parts of words such as subwords; they could even consist only of a single character. For instance, “understanding for example” might be come tokens like [“understand”, “ing”, “Ġfor “, “example”] (the “Ġ” often marks a space). This enables the model to deal with a large lexicon, including words which have been newly coined or are seldom used, by putting together the known subword pieces.
- Embedding: Numbers to Meanings Once tokenized, every token is converted to a numerical representation called an embedding. This is not just random digital noise, it is the resolving of each word down into a high-dimensional dynamic vector (in your mind’s eye, think of a row which is very long and reaches far back). Vectors like this can keep semantically similar words or words that are often used together very close together in space, so that “king” and “queen” and “running” and “jogging” all cluster together. This allows the model to understand meanings/uses of language: the most important way that an embedded word vector is separated from the others.
Embedded word vectors constitute the primary way that the model’s “knowledge” of language is established.
The real powerhouse behind most modern LLMs is the Transformer architecture, which was introduced in the seminal 2017 paper “Attention Is All You Need.” Before Transformers, it was hard for an AI to process long sequences of text (like paragraphs), because understanding the context across distant words was difficult. This was totally transformed by transformers. A typical Transformer consists of two main parts:
Encoder: Reads the input sequence (usually represented by embeddings) in its entire form and builds up rich context that gives every word meaning more than just its dictionary definition. The whole process is more holistic: it looks at all tokens simultaneously and sees how they interact with each other. Decoder: With the encoder’s understanding in mind the decoder generates output sequence (for example, an answer to a question or translation) token by token based on the input context and what it has created for itself so far. (And it should be noted: Some models, such as GPT, mostly use the decoder portion, while others — like BERT — mainly concentrate on the encoder. Models that are sequence-to-sequence minded, such as T5, use both parts.) 4. The Attention Mechanism: Focusing on What Matters
Within the Transformer, the key aspect is the Attention Mechanism (especially, “self-attention”). Picture reading a sentence such as: “Chased the mouse, the cat, sat on the mat. “In comprehending the word” sat, “your brain will pay more attention to”cat “than”chased “or”mouse “since it’s interested in who sat. Self-attention allows LLM to do something similar computationally. As it deals with each token, the attention mechanism offsets the significance of all the other tokens from that sequence relative to present one. It learns which words are most important for understanding any given act of context, no matter where they may be in that given sentence. Being able to weigh token importance in this way dynamically is what enables Transformers to handle long-range dependencies and effectively grasp complex grammatical structures, as well as with it meanings. Once we have fed your question through our pipeline (tokenization, embeddings, and the transformer), please choose the next answer, whatever you do want now! Its attention mechanism let the encoder learn more easily what’s going on in your question or statement part by examining words nearby it and their dependencies. Using this information, and a gigantic set of training data, the decoder can then generate an answer according to that understanding. Every token is made paying attention to what parts of the original input or its own output so far have been most relevant in order to create a cohesive and contextually appropriate result.The core–tokenization, embeddings, the Transformer architecture, and attention mechanism–allows us to understand why LLMs behave as they do, and to see consequences such as the human-like text that can be produced.
