Text Classification from Start to Finish — lecture notes
Build a real text classifier: framing and labelling, tokenisation and TF-IDF features, Naive Bayes and logistic-regression baselines, cross-validation and metrics, imbalance, fine-tuning transformers, zero-shot LLMs, error analysis and monitoring.
0:001. Introduction
Spam filters, sentiment analysis, routing support tickets, flagging harmful content, detecting fake reviews: text classification is the workhorse of practical NLP. In this deep dive we build a classifier end to end, from collecting labels to choosing between classic models, fine tuned transformers and large language models.
0:202. The task
Text classification assigns one or more labels from a fixed set to a piece of text. A support ticket might be routed to billing, delivery, login or other. A review might be positive or negative. When a text can have several labels at once, it is called multi label classification.
0:413. The pipeline
Every project follows the same pipeline. Frame the problem and define the labels. Collect and label examples. Represent text as numbers. Build a simple baseline model, then try stronger ones. Evaluate carefully, reading the errors, and finally deploy and monitor, because language keeps changing.
1:004. Labelling
Most of the quality comes from the labels. Write clear definitions with examples for every label, including tricky edge cases, and have several people label a shared sample to measure agreement. Is refund not received a billing issue or a delivery issue? Decide, and write it down.
1:205. Pause and think
Pause and think. Two annotators label well, that was something, as positive and negative. What does their disagreement tell you? The text is ambiguous, or the guidelines are unclear. Add guidance, or a mixed label. A model can never be more consistent than the labels it learns from.
1:406. Tokenisation
Before modelling, text is split into tokens. Word level tokens give short sequences but a huge vocabulary with unknown words. Character level needs a tiny vocabulary but very long sequences. Sub word tokenisation, used by modern models, keeps common words whole and splits rare ones into meaningful pieces.
2:007. Preprocessing
Classic models benefit from light preprocessing: lowercasing, removing noise such as HTML tags and URLs, and handling numbers and emojis consistently. Be careful not to remove meaning. Deleting stop words can remove not, flipping the sentiment. Transformers need very little preprocessing beyond their own tokenizer.
2:208. TF-IDF features
A classic representation is TF IDF. Each document becomes a vector of word weights, where words frequent in this document but rare elsewhere score highest, and ubiquitous words like the get zero. The result is a long, sparse vector that works surprisingly well with linear models.
2:399. n-grams
Adding word pairs, called bigrams, as extra features captures short phrases. Not good and good become different features, so a linear model can learn that the phrase is negative even though good alone is positive. The vocabulary grows, but regularisation keeps the model in check.
2:5910. Multi-label problems
Check whether your problem is multi class or multi label. In multi class problems each text has exactly one label, and a softmax picks among them. In multi label problems a text can have several, such as a news article about both politics and the economy, so each label gets its own yes or no decision.
3:2211. Naive Bayes
The simplest strong baseline is Naive Bayes. It assumes that, given the class, words appear independently, which is obviously false, yet it works surprisingly well. It combines each word’s evidence using Bayes’ theorem, trains in seconds, and was the algorithm behind early spam filters.
3:4112. The Naive Bayes rule
The rule multiplies the prior probability of each class by the probability of every word given that class, and picks the class with the highest score. In practice we add log probabilities to avoid underflow, and smooth the counts so an unseen word does not zero out a class.
4:0213. Pause and think
Pause and think. The word refund never appeared in any training email labelled spam. What does unsmoothed Naive Bayes conclude about a spam email containing refund? The probability of refund given spam is zero, which zeroes out the entire spam score. Adding one to every count, called Laplace smoothing, fixes this.
4:2314. Word weights
A linear classifier gives each word a weight. Delicious pushes strongly towards positive, while slow and rude push towards negative. The weights add up, and a sigmoid turns the total into a probability. But look at the last sentence: not bad at all is positive, yet word by word it looks negative.
4:4515. Logistic regression
Logistic regression learns such weights by gradient descent, placing a boundary between the classes and assigning each point a probability. Here accuracy climbs from fifty nine to ninety three percent as training runs. On TF IDF features, it is one of the strongest simple baselines in text classification.
5:0616. A strong baseline
Here is a baseline that is hard to beat for its cost. A pipeline combines a TF IDF vectoriser, using words and bigrams, with logistic regression using balanced class weights. Five fold cross validation reports macro F1. It trains in seconds and gives a reference point for anything fancier.
5:2717. Cross-validation
Always evaluate with cross validation or a held out test set. With five folds, every example is used for validation exactly once. Here the scores range from point eight two to point eight seven, averaging point eight four eight: that spread tells you how much a single split could mislead.
5:4818. Reading the confusion matrix
The confusion matrix shows exactly which mistakes happen. For a spam filter tested on one hundred emails: forty two spam caught, eight real emails wrongly flagged, six spam missed and forty four real emails correctly passed, giving precision of eighty four percent and recall of eighty seven and a half.
6:0919. Imbalanced classes
Real label sets are often imbalanced. If ninety five percent of tickets are other, a model that always says other is ninety five percent accurate and completely useless. Use class weights or resampling, tune decision thresholds, and report per class precision and recall or macro F1.
6:2920. Pause and think
Pause and think. A harmful content classifier flags posts for human review, and missing a harmful post is far worse than reviewing a harmless one. Which metric do you prioritise? Recall for the harmful class, while keeping precision acceptable so reviewers are not overwhelmed.
6:4821. Pre-trained transformers
The next step up is a pre trained transformer such as BERT. It has already learned a great deal about language by predicting masked words from both sides of their context, so it understands negation, word order and synonyms far better than a bag of words.
7:0722. Fine-tuning
We fine tune it for our labels: add a small classification head on top of the pre trained network and train on our labelled examples with a small learning rate. With a few thousand labelled texts, fine tuned transformers usually beat TF IDF baselines clearly, especially on subtle cases.
7:2823. Zero-shot and few-shot
Large language models can classify with no training at all. Describe the labels in a prompt, zero shot, or add a handful of examples, few shot, and ask the model to choose. This is perfect for prototypes, brand new labels or when you have almost no labelled data.
7:4824. Zero-shot in code
Here is zero shot classification with an open model. A model trained on natural language inference judges whether the text entails each candidate label, such as this text is about billing. Give it the text and the candidate labels, and it returns a ranked list with scores, without any training data.
8:1025. Which approach?
Which approach should you choose? A TF IDF linear model trains in seconds, runs anywhere and is transparent, but needs labels and misses subtlety. Fine tuned transformers understand context much better. Prompted language models need no labels at all, but cost more per prediction. Many teams prototype with prompting, then train a cheaper model.
8:3326. Error analysis
The most valuable hour in any project is error analysis. Read a sample of misclassified examples and group them by cause: wrong labels, negation, sarcasm, missing domain vocabulary or genuinely ambiguous categories. Very often the best fix is in the data or the label definitions, not the model.
8:5327. Common failure modes
Common failures include negation and sarcasm, which fool bag of words models; domain shift, as new products and slang appear; label noise from inconsistent annotators; and spurious cues, such as an email signature that happens to predict the label. Each has a matching fix.
9:1228. Labelling smarter
Labelling is expensive, so label smartly. With active learning, the current model chooses which unlabelled texts should be labelled next, usually those it is least certain about. Each new annotation then teaches the model as much as possible, often reaching the same accuracy with far fewer labels.
9:3229. Beyond English
Many applications serve several languages. Multilingual transformers such as XLM R share one model across about a hundred languages, so a classifier trained with English labels can often work on Bangla or French too. Always check performance separately for each language, since quality varies.
9:5130. Deploy and monitor
After deployment, language keeps moving. New products, slang and events change the text your classifier sees. Monitor the mix of predicted labels and the model’s confidence, label a fresh sample regularly, and retrain. A sudden rise in low confidence predictions often signals a brand new topic.
10:1131. Best practices
In short: invest in clear labels, start with a TF IDF and logistic regression baseline, measure with macro F1 and per class metrics, read the errors before changing the model, and keep monitoring and refreshing, because language keeps changing.
10:2832. Recap
To recap. Frame your labels carefully, because label quality caps model quality. TF IDF with n grams and a linear model gives a fast, strong baseline. Cross validate and track precision, recall and macro F1. Fine tuned transformers handle context, language models can classify zero shot, and error analysis plus monitoring keep a classifier useful.
Key takeaways
- Text classification maps text to labels; clear labelling guidelines and annotator agreement set the ceiling on quality.
- TF-IDF with word n-grams plus logistic regression (or Naive Bayes) is a fast, strong baseline.
- Use cross-validation and per-class metrics (precision, recall, macro-F1); accuracy misleads on imbalanced labels.
- Fine-tuned pre-trained transformers capture negation and context; LLMs can classify zero-/few-shot without training.
- Error analysis usually reveals data and label problems: negation, sarcasm, domain shift, label noise, spurious cues.
- Monitor deployed classifiers for drift in language and topics, and retrain with fresh labels.
Check yourself
- Why add bigram features such as “not good”?
Show answer
To capture short phrases and negation a bag of single words misses — “not good” differs from “good”.
- What does Naive Bayes assume?
Show answer
Words are independent given the class — A “naive” but effective independence assumption.
- With 95% of tickets labelled “other”, which metric is most informative?
Show answer
Macro-F1 / per-class recall and precision — Accuracy rewards always predicting “other”.
- What is zero-shot classification?
Show answer
Classifying using label descriptions without task-specific training examples — E.g. an NLI model or a prompted LLM.
- Error analysis mainly involves…
Show answer
Reading misclassified examples and grouping them by cause — It reveals what to fix next.
Go deeper
- Text Classification: From Linear Models to Fine-Tuned Transformers · The AI Lecture Hall
- Sentiment Analysis: Opinions, Aspects and Nuance · The AI Lecture Hall
- Text Preprocessing: Tokenisation, Normalisation, Stemming and Lemmatisation · The AI Lecture Hall
- Fine-Tuning Pretrained Language Models: A Practical Guide · The AI Lecture Hall
- Evaluating NLP Systems: Perplexity, BLEU, ROUGE, BERTScore and Human Judgement · The AI Lecture Hall
© 2026 Janin A Apurba, CSE, AUST · Advanced ICT Officer, CNRS-UNHCR. All rights reserved. Notes for the animated lecture at https://ai-in-motion.vercel.app/watch/text-classification-end-to-end.html