RecallStack
Login

Tokens

MEDIUM3 min readby AdminJune 20, 2026History
0.0|0 ratingsLog in to rate

Tokens are the basic building blocks processed by an LLM. Before text is fed to a model, a **tokenizer** splits the string into numerical IDs representing sub-w...

#ai-engineering#course#tokens

Definition

Tokens are the basic building blocks processed by an LLM. Before text is fed to a model, a tokenizer splits the string into numerical IDs representing sub-word units using algorithms like Byte-Pair Encoding (BPE).

Why It Exists

Computers do not process letters or characters directly; they process numbers. Splitting text into exact words creates massive vocabularies that require excessive GPU memory. Splitting into characters loses semantic meaning. Sub-word tokenization strikes the perfect balance, allowing models to handle out-of-vocabulary words using common character roots.

How It Works

python
1
2
3
4
5
6
7
SUB-WORD TOKENIZATION (BPE)
Raw Text:    "preheating" ──► Tokenizer ──► [ "pre", "heat", "ing" ]
Token IDs:                                   [ 231,   4567,   89 ]

1. **Byte-Pair Encoding**: The tokenizer builds a vocabulary of characters and recursively merges the most frequent pairs of adjacent characters/tokens observed in training.
2. **Conversion**: A word like `"preheating"` is split into sub-words `["pre", "heat", "ing"]` and mapped to corresponding integer IDs: `[231, 4567, 89]`.
3. **Estimation Rules**: As a general rule of thumb, 1 token $\approx$ 4 characters, or $\approx$ 0.75 words in English.

Real-World Example

Example

In Gemini API or OpenAI API, billing and rate limits are calculated based on token counts. If you send a long code file, your API bill is directly proportional to the number of input and output tokens consumed.

Python Example

Example

Here is a simulation of sub-word tokenization and token cost estimation:

python
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
from typing import List, Dict

class MockSubwordTokenizer:
    """Simulates Byte-Pair Encoding (BPE) sub-word splits."""
    def __init__(self):
        # Mock sub-word vocabulary mappings
        self.vocab = {"un": 101, "predict": 102, "able": 103, "ly": 104, "help": 105, "ful": 106}

    def tokenize(self, text: str) -> List[int]:
        tokens = []
        words = text.lower().split()
        
        for word in words:
            # Simple greedy sub-word segmentation simulation
            remaining = word
            while remaining:
                matched = False
                for prefix in sorted(self.vocab.keys(), key=len, reverse=True):
                    if remaining.startswith(prefix):
                        tokens.append(self.vocab[prefix])
                        remaining = remaining[len(prefix):]
                        matched = True
                        break
                if not matched:
                    # Fallback to single character/byte token value
                    tokens.append(ord(remaining[0]))
                    remaining = remaining[1:]
        return tokens

def estimate_billing_cost(token_count: int, model_name: str) -> float:
    # Model price per 1,000,000 tokens (Gemini 1.5 Flash input rate: $0.075)
    rates = {
        "gemini-flash": 0.075 / 1_000_000,
        "premium-model": 5.00 / 1_000_000
    }
    return token_count * rates.get(model_name, 0.0)

if __name__ == "__main__":
    tokenizer = MockSubwordTokenizer()
    sample_text = "unpredictably helpful"
    ids = tokenizer.tokenize(sample_text)
    
    print("Raw text:", sample_text)
    print("Token IDs:", ids)
    print("Total Tokens:", len(ids))
    print("Estimated cost ($):", estimate_billing_cost(len(ids), "gemini-flash"))

Interview Questions

  • Beginner: How many words equal 100 tokens on average?
  • Intermediate: What is Byte-Pair Encoding (BPE) and why is it preferred over character-level tokenization?
  • Advanced: Why does writing code or non-English text consume significantly more tokens than standard English text?

Interview Answers

  • Beginner: On average, 1 token is equivalent to 0.75 words, so 100 tokens is approximately 75 words in English.
  • Intermediate: BPE is a sub-word tokenization algorithm that starts with single characters and iteratively merges the most frequent adjacent byte pairs. It is preferred over character tokenization because it compresses text into fewer tokens, preserving rich semantic meaning while maintaining a manageable vocabulary size.
  • Advanced: Tokenizers are trained on English-centric corpora where common English words are assigned unique token IDs. Non-English languages and code blocks (which contain unusual characters, spaces, and punctuation) do not match these common merges, forcing the tokenizer to split them into individual characters or bytes, which inflates the token count.

Common Mistakes

  • Calculating string length as token count: Treating len(text) as the token count. Characters and tokens are not equivalent; always use the specific model tokenizer (like tiktoken for OpenAI, or google-genai wrappers) to count tokens accurately.

Assignment

Write a Python script that counts the tokens in a text file using a mock tokenizer. If the count exceeds 100 tokens, throw a warning warning that the prompt is too long.

Discussion

Join the discussion! Sign in to leave comments and ask questions.

Loading discussion...