What is an LLM?
A Large Language Model (LLM) is a deep learning model trained on vast amounts of text data to predict the next word in a sequence. Modern LLMs are based on the ...
Definition
A Large Language Model (LLM) is a deep learning model trained on vast amounts of text data to predict the next word in a sequence. Modern LLMs are based on the Transformer architecture—specifically decoder-only autoregressive models.
Why It Exists
Traditional natural language processing (NLP) systems relied on custom models for individual tasks (e.g., one model for translation, another for classification). LLMs act as generalist foundation models; they learn general language representations, allowing a single model to perform diverse tasks simply by altering the input instructions.
How It Works
AUTOREGRESSIVE NEXT-TOKEN GENERATION
┌──────────────┐
│ Input Prompt │ ──► [ Tokenizer ] ──► [ Embedding Layer ] ──► [ Transformer Layers ]
└──────────────┘ │
▼
[ Softmax Probabilities ]
│
┌──────────────┐ ▼
│ Next Token │ ◄───────────────────────────────────────── [ Sampling Stage ]
└──────────────┘
(Appended to Prompt for Next Step)
1. **Embedding**: Input text is broken into tokens and mapped into high-dimensional vectors representing semantic meaning.
2. **Self-Attention**: The transformer layers analyze relationships between all tokens in the sequence concurrently, calculating attention scores to represent contextual meaning.
3. **Autoregressive Decoding**: The model outputs a probability distribution over the entire vocabulary, selects the next token, appends it to the input, and repeats the cycle (autoregression) until it hits an End-of-Sequence (`<EOS>`) token.Real-World Example
In ChatGPT or Claude, the interface is conversational. When you prompt the model, the backend splits your query into tokens, feeds it into the model weights, and runs the autoregressive generation loop, streaming the tokens back to your screen in real-time.
Python Example
Here is a simulation of the core autoregressive next-token prediction logic using a token transition matrix:
import random
from typing import List, Dict
class AutoregressiveMockLLM:
"""Simulates next-token autoregressive decoding using a Markov chain model."""
def __init__(self):
# A vocabulary transition mapping
self.transitions: Dict[str, List[str]] = {
"what": ["is", "are"],
"is": ["your", "the", "artificial"],
"your": ["name", "purpose"],
"name": ["?", "assistant"],
"artificial": ["intelligence"],
"intelligence": ["?", "is", "here"]
}
def generate(self, start_word: str, max_tokens: int = 5) -> str:
current_token = start_word.lower()
output = [current_token]
for _ in range(max_tokens):
next_options = self.transitions.get(current_token)
if not next_options:
break
# Autoregressive step: choose next token based on transition probabilities
current_token = random.choice(next_options)
output.append(current_token)
if current_token == "?":
break
return " ".join(output)
if __name__ == "__main__":
model = AutoregressiveMockLLM()
print("Generation 1:", model.generate("what", max_tokens=5))
print("Generation 2:", model.generate("artificial", max_tokens=5))Interview Questions
- Beginner: What is the core objective of a decoder-only LLM during training?
- Intermediate: What is the role of the Self-Attention mechanism in the Transformer architecture?
- Advanced: Explain the difference between encoder-decoder (e.g., T5) and decoder-only (e.g., Llama) architectures, and why the industry has converged on decoder-only models for Generative AI.
Interview Answers
- Beginner: The core objective of a decoder-only LLM is autoregressive next-token prediction: given a sequence of tokens, the model is trained to predict the most statistically likely next token.
- Intermediate: Self-attention allows the model to relate each token in a sequence to every other token. It computes attention scores that weigh the contextual relationship between tokens (e.g., linking the pronoun "it" to the noun "dog" earlier in the sentence), regardless of their distance.
- Advanced: Encoder-decoder models use separate networks for input comprehension (encoder) and output generation (decoder), which is ideal for translation. Decoder-only models use a unified network to process prompts and generate responses. The industry converged on decoder-only models because they are computationally simpler to scale to billions of parameters, have lower memory requirements during inference, and excel at open-ended creative tasks.
Common Mistakes
- Assuming LLMs search the web: Beginners often think LLMs lookup answers in a database. Explain that LLMs are frozen neural networks; their knowledge is stored inside their parameters (weights), not in an external database.
Assignment
Expand the AutoregressiveMockLLM transitions mapping to support generated sentences that answer: "how does AI work?".