Data and Tokenization

Why Data Is the Real Ingredient

There is a saying in computing that has been around since the 1960s: garbage in, garbage out. Feed a computer bad information, and no matter how sophisticated the software, bad results come out. This principle applies to AI more than to any technology before it, because AI models do not follow rules that humans write. They absorb patterns from data. If the data has bad patterns, the model learns bad patterns.

Think of it this way. Imagine you are trying to teach a child to recognise dogs. You show them 10,000 pictures, but all of them happen to be golden retrievers photographed outdoors in sunlight. The child will learn to recognise golden retrievers in sunlight perfectly. Show them a black poodle indoors and they will be confused. The model has not learned "dog." It has learned "golden retriever in sunlight." The limitation was not the child's intelligence. It was the data.

This is called dataset bias, and it is one of the most common reasons AI systems fail in the real world. The model performs brilliantly in testing because the test data looks like the training data. Then it encounters data from a different hospital, a different city, or a different demographic, and performance collapses. The model was never learning what we thought it was learning.

What Makes Data Good or Bad

Not all data problems are as obvious as only having photos of golden retrievers. Most data issues are subtle, discovered only after a model fails in unexpected ways. Here are the most common ones:

The Problem: Models Cannot Read Text

Everything so far in this course has involved numbers. Weights are numbers. Loss is a number. The forward pass multiplies and adds numbers. But most of the world's valuable information comes as text: emails, articles, medical notes, contracts, customer reviews. Text is not numbers. How does a model learn from it?

The answer is tokenization: the process of breaking text into small pieces called tokens, and then converting each token into a number. Once every word, punctuation mark, or fragment of a word has a number, the model can work with it.

Think of it like a library catalogue system. Every book in the library has a catalogue number. Librarians do not carry the books around when they are discussing the collection. They use the numbers. Tokenization gives every piece of text its catalogue number so the model can refer to it numerically.

Three Ways to Tokenize

There are three main approaches to tokenization, and each makes a different trade-off:

Character-level tokenization splits text into individual characters. The word "hello" becomes five tokens: h, e, l, l, o. Each character gets a number. This is simple and handles any language, but it produces very long sequences and forces the model to learn that h-e-l-l-o spells a word, rather than starting from the word itself.

Word-level tokenization splits text by spaces and punctuation. "The cat sat" becomes three tokens: The, cat, sat. Simpler for the model to work with, but it runs into trouble with rare words, typos, and new words that were not in the training vocabulary. A word it has never seen gets mapped to an "unknown" token, losing all meaning.

Subword tokenization is what most modern language models use. It splits common words into single tokens but breaks rare words into fragments. "Running" might be one token, but "antidisestablishmentarianism" gets split into smaller pieces the model has seen before. This balances vocabulary size with the ability to handle unusual words. The widely-used BPE (Byte Pair Encoding) method builds its vocabulary by repeatedly merging the most common pairs of characters until the vocabulary reaches a target size.

Training, Validation, and Test Sets

Once your data is cleaned and tokenized, you cannot just feed all of it to the model during training. If the model sees every single example while it is learning, you have no way to know whether it has genuinely learned the pattern or simply memorised the answers. You need to hold some data back.

The standard approach splits your data into three separate groups, and they each play a different role:

The training set is what the model actually learns from. This is the data the weights get adjusted on. It typically makes up the largest portion of your data, often around 70 to 80 percent.

The validation set is used during training, but not to update the weights. Instead, you periodically pause training and run the model on the validation set to check whether it is actually getting better at generalising, not just memorising the training examples. If the training loss keeps dropping but the validation loss stops improving or starts rising, the model is overfitting. Time to stop.

The test set is kept completely separate and untouched until training is fully finished. It is used once, at the end, to get a final honest measure of how well the model performs on data it has never seen in any form. Think of the test set as the final exam: you study with the training set, you check your progress with validation, and you sit the exam only when you think you are ready.

The Rule You Must Not Break

There is one rule about the test set that practitioners treat as sacred: never use the test set to make decisions during training. The moment you look at the test set results and then go back and adjust your model, the test set is no longer a fair measure. You have started optimising for it, even indirectly. It has become part of your training process.

This mistake is more common than people admit. A team trains a model, checks the test score, tweaks the model, checks the test score again, and repeats. After enough rounds, the test score looks great. But when the model ships and meets real-world data, it underperforms. The team unknowingly leaked the test set into their decisions.

The solution is discipline. Set the test set aside before you start. Look at it once, when training is done. Report that number. Move on.

How Much Data Do You Actually Need?

This is one of the most common questions in machine learning, and the honest answer is: it depends, and you probably need more than you think. A few general principles hold across most situations.

For simple tasks with clear patterns, a few thousand well-labelled examples can be enough. A spam filter does not need millions of emails. A model predicting house prices from a handful of features can learn from tens of thousands of examples. The simpler the pattern, the less data you need.

For complex tasks, especially those involving language, images, or audio, you typically need hundreds of thousands to millions of examples. The large language models you interact with today were trained on trillions of tokens, drawn from a substantial portion of the text available on the internet plus books, code repositories, and scientific papers.

The most powerful shortcut in modern AI is transfer learning: starting with a model that has already been trained on enormous amounts of data, then fine-tuning it on your specific task with a much smaller dataset. Instead of starting from random weights and learning everything from scratch, you start from a model that already understands language, and you teach it your specific domain. Module 6 covers this in depth.