Topic Modeling with LSA in Python: A Practical Guide to Discovering Hidden Themes in Text
Topic: NLP Modeling — Topic Modeling with Latent Semantic Analysis (LSA) in Python
Level: Beginner to Intermediate
Technology: Python, NLP, Scikit-learn, NumPy, Pandas, Matplotlib
Estimated reading time: 18–22 minutes
Introduction: What If a Computer Could Discover What Your Documents Are About?
Imagine you have thousands of documents.
Some are about artificial intelligence. Others discuss programming, cybersecurity, education, sports, business, or technology. You know that different themes exist in the collection, but manually reading every document would take an enormous amount of time.
This is where Topic Modeling becomes useful.
Topic modeling is an unsupervised Natural Language Processing (NLP) technique that attempts to discover hidden themes, or topics, inside a collection of documents.
One of the classic approaches to topic modeling is Latent Semantic Analysis (LSA).
LSA can transform a large collection of text into a mathematical representation and then use linear algebra to uncover relationships between words and documents. Instead of simply counting which words appear, LSA tries to discover deeper patterns in the way words occur across documents.
For example, suppose we have these documents:
"Python is widely used for machine learning."
"Neural networks are important in deep learning."
"The football team won the championship."
"The player scored two goals in the final."
A topic-modeling algorithm might discover themes such as:
Topic 1: Python, machine learning, neural networks, deep learning
Topic 2: football, team, player, goals, championship
The important part is that we did not manually label the documents.
The algorithm discovered the patterns from the text itself.
In this guide, you will learn:
What topic modeling is
What LSA means
How LSA works mathematically
Why Singular Value Decomposition is important
How to preprocess text
How to create a document-term matrix
How to use TF-IDF with LSA
How to implement LSA in Python
How to interpret discovered topics
How to choose the number of topics
How to visualize topics
Common mistakes and limitations
How LSA compares with other topic-modeling methods
How LSA can be used in real-world NLP applications
1. What Is Topic Modeling?
Topic modeling is a family of statistical and mathematical methods used to discover hidden themes in a collection of documents.
The key idea is simple:
Documents contain words, and patterns in those words can reveal underlying topics.
A topic does not necessarily have a predefined name.
For example, if an algorithm repeatedly finds words such as:
python, code, programming, function, software
you might interpret the resulting topic as Programming.
If another topic contains:
neural, network, training, model, learning
you might interpret it as Machine Learning.
The algorithm itself generally does not know that the first topic is "Programming" or that the second is "Machine Learning."
Instead, it gives you a group of words associated with each latent topic.
Humans then interpret those words and assign meaningful labels.
2. Why Is Topic Modeling Useful?
Modern applications can contain enormous amounts of text.
Consider:
News websites
Customer reviews
Research papers
Social media posts
Support tickets
Product descriptions
Books
Legal documents
Scientific publications
Search queries
Reading all of this manually is difficult.
Topic modeling provides a way to automatically explore large collections of text.
For example, a company might have 100,000 customer-support messages.
After applying topic modeling, it could discover themes such as:
Login problems
Payment issues
Account security
Mobile application problems
Delivery questions
Subscription problems
This can help analysts understand what customers are discussing without manually categorizing every message.
3. What Is LSA?
LSA stands for Latent Semantic Analysis.
It is a mathematical technique for discovering hidden relationships between terms and documents.
LSA became popular as a method for analyzing the semantic structure of text.
The word latent means something hidden or not directly observed.
For example, the word "doctor" and the word "hospital" are different words, but they frequently appear in related documents.
Similarly:
Python, programming, code, and software
may frequently occur together.
LSA attempts to capture these hidden relationships by representing documents and words in a lower-dimensional mathematical space.
The central mathematical technique behind LSA is:
Singular Value Decomposition (SVD).
This is one of the most important concepts to understand when working with LSA.
4. The Core Idea Behind LSA
Suppose we have a collection of documents.
We first convert the documents into a numerical matrix.
For example:
| Word | Document 1 | Document 2 | Document 3 |
|---|---|---|---|
| python | 2 | 1 | 0 |
| machine | 1 | 2 | 0 |
| learning | 1 | 2 | 0 |
| football | 0 | 0 | 2 |
| player | 0 | 0 | 1 |
This is called a document-term matrix.
Each row represents a term.
Each column represents a document.
The values indicate how strongly a word occurs in each document.
But this matrix can become extremely large.
Imagine:
10,000 documents
50,000 unique words
The resulting matrix could contain hundreds of millions of entries.
LSA uses dimensionality reduction to compress this large matrix into a smaller representation.
That is where SVD comes in.
5. Understanding Singular Value Decomposition
Singular Value Decomposition decomposes a matrix into three matrices.
If our document-term matrix is represented by:
then SVD decomposes it as:
where:
U represents relationships between terms and latent concepts
Σ contains singular values representing the importance of those concepts
Vᵀ represents relationships between documents and latent concepts
This might look complicated at first, but the main idea is simple.
LSA takes a large text matrix and finds a smaller number of important dimensions that capture major patterns.
For example, instead of representing documents using 10,000 word dimensions, we might represent them using only 20 latent dimensions.
Those dimensions can correspond approximately to hidden topics.
6. Why Reduce the Number of Dimensions?
Suppose we have 20,000 unique words.
Representing every document using all 20,000 dimensions can be inefficient.
Many words may be:
Rare
Unimportant
Redundant
Noisy
Poor indicators of meaning
LSA reduces the representation.
For example:
Original representation
20,000 dimensions
↓
LSA representation
20 dimensions
The smaller representation can capture broad semantic patterns while removing some noise.
This is called dimensionality reduction.
7. LSA vs Simple Word Counting
A basic bag-of-words model primarily asks:
How many times does each word occur?
LSA goes further.
It attempts to identify patterns in word usage across documents.
Consider:
"car"
"vehicle"
"automobile"
These words are different strings.
Traditional word-count methods treat them as separate features.
LSA can potentially place them into related latent dimensions if their distributions across documents are similar.
This is one reason LSA is associated with latent semantic analysis.
However, it is important not to think of LSA as understanding language like a modern large language model.
LSA is fundamentally a mathematical method based on patterns in a matrix.
8. The LSA Pipeline
A typical LSA topic-modeling pipeline looks like this:
Raw Documents
↓
Text Cleaning
↓
Tokenization
↓
Stopword Removal
↓
Vectorization
↓
TF-IDF Matrix
↓
Singular Value Decomposition
↓
Reduced Topic Space
↓
Topic Interpretation
↓
Visualization / Analysis
Each stage has a purpose.
Let's examine them one by one.
9. Step 1: Collect Your Documents
The first step is obtaining a collection of text documents.
For example:
documents = [
"Python is popular for machine learning and data science.",
"Neural networks are widely used in deep learning.",
"Football players train hard before the championship.",
"The team won the football match yesterday.",
"Machine learning models can analyze large datasets."
]
These documents will become our input.
In real projects, documents could come from:
CSV files
Databases
APIs
Websites
PDF files
News articles
Research papers
Customer reviews
The more useful and representative your collection is, the more meaningful your topics are likely to become.
10. Step 2: Clean the Text
Raw text often contains unnecessary information.
For example:
"Python, Python!!! is AMAZING!!!"
We may want to normalize it into something like:
python amazing
Common preprocessing operations include:
Converting text to lowercase
Removing punctuation
Removing unnecessary symbols
Removing stopwords
Removing extremely rare terms
Optional stemming
Optional lemmatization
However, preprocessing should be chosen carefully.
Aggressive preprocessing can remove information that is actually useful.
11. Step 3: Stopword Removal
Stopwords are common words that often contribute little to topic identification.
Examples include:
the
is
a
an
of
to
in
and
For example:
"The model is trained on a large dataset."
After removing common stopwords:
model trained large dataset
The remaining words may provide stronger signals for topic modeling.
Scikit-learn provides built-in English stopwords that can be used with TfidfVectorizer.
12. Step 4: Convert Text into Numbers
Machine-learning algorithms cannot directly process raw sentences.
We need to convert text into numerical vectors.
One popular method is TF-IDF.
TF-IDF stands for:
Term Frequency–Inverse Document Frequency
It gives higher importance to words that are important to a document but not equally common across the entire collection.
13. Understanding TF-IDF
TF-IDF combines two concepts.
Term Frequency
Term frequency measures how frequently a word appears in a document.
If a document repeatedly discusses "Python," then Python receives a higher term-frequency value.
Inverse Document Frequency
Inverse document frequency reduces the importance of words that appear in many documents.
A common word such as "the" appears almost everywhere, so it provides little information.
A specialized word such as "transformer" may appear in fewer documents and therefore carry more information.
Conceptually:
The exact implementation includes additional normalization and smoothing details, but this formula captures the central idea.
14. Why Use TF-IDF Before LSA?
LSA can technically be applied to different matrix representations.
However, TF-IDF is commonly used because it can reduce the influence of extremely common terms.
For example, suppose the word "data" appears in almost every document.
A raw count matrix may give it significant weight.
TF-IDF can reduce its importance if it is common throughout the collection.
This allows LSA to focus more on discriminative terms.
A common pipeline is therefore:
Documents
↓
TF-IDF
↓
SVD
↓
LSA Topics
15. Installing the Required Python Libraries
For a basic implementation, install:
pip install numpy pandas scikit-learn matplotlib
If you are working in Jupyter Notebook:
!pip install numpy pandas scikit-learn matplotlib
Scikit-learn provides the tools we need for:
TF-IDF vectorization
SVD
Topic extraction
Mathematical transformations
16. Building Your First LSA Model
Let's create a small dataset.
documents = [
"Python is widely used for machine learning and artificial intelligence.",
"Machine learning models can analyze data and make predictions.",
"Deep learning uses neural networks to solve complex problems.",
"Neural networks are important in modern artificial intelligence.",
"Football players train every day before an important match.",
"The football team won the championship after a difficult game.",
"The player scored two goals during the football match.",
"Coaches prepare football teams for competitive tournaments."
]
We can now create a TF-IDF vectorizer.
from sklearn.feature_extraction.text import TfidfVectorizer
vectorizer = TfidfVectorizer(
stop_words="english"
)
X = vectorizer.fit_transform(documents)
The matrix X contains numerical representations of our documents.
We can inspect its dimensions:
print(X.shape)
The result represents:
(number of documents, number of terms)
17. Applying SVD
Now we use TruncatedSVD.
from sklearn.decomposition import TruncatedSVD
svd = TruncatedSVD(
n_components=2,
random_state=42
)
lsa_matrix = svd.fit_transform(X)
Here:
n_components=2
means that we want two latent dimensions.
We can think of these dimensions as two major topic directions in our small dataset.
The transformed matrix contains the representation of each document in the reduced topic space.
18. Extracting Topic Words
The most interesting part is discovering which words are strongly associated with each component.
We can retrieve the feature names:
terms = vectorizer.get_feature_names_out()
Then inspect the most important words.
for i, component in enumerate(svd.components_):
terms_in_topic = [
terms[index]
for index in component.argsort()[-10:]
]
print(f"Topic {i + 1}:")
print(terms_in_topic)
You may get output resembling:
Topic 1:
['learning', 'neural', 'networks', 'python', 'data', 'models']
Topic 2:
['team', 'game', 'players', 'goals', 'football', 'match']
The exact results can vary depending on preprocessing and data.
Notice something important:
The algorithm does not automatically name the topics.
It simply gives you important words.
You interpret them.
For example:
learning, neural, networks, python, models
could be labeled:
Topic 1 — Artificial Intelligence
And:
team, players, goals, football, match
could be labeled:
Topic 2 — Football
19. Understanding components_
The following property is extremely important:
svd.components_
It contains the component-to-term weights.
Conceptually:
Term 1 Term 2 Term 3 Term 4
Topic 1 0.10 0.72 0.55 0.03
Topic 2 0.01 0.05 0.11 0.91
A higher absolute weight means the term has a stronger association with that component.
We can therefore sort the terms according to their weights.
20. A Better Topic Extraction Function
Instead of repeating code, we can create a reusable function.
def display_topics(model, feature_names, num_words):
for topic_index, topic in enumerate(model.components_):
top_indices = topic.argsort()[:-num_words - 1:-1]
words = [
feature_names[i]
for i in top_indices
]
print(
f"Topic {topic_index + 1}: "
+ " | ".join(words)
)
Now call:
display_topics(
svd,
terms,
10
)
This gives us the top ten words for each topic.
21. Choosing the Number of Topics
One of the biggest questions in topic modeling is:
How many topics should I create?
Suppose you have 10,000 documents.
Should you use:
2 topics?
5 topics?
10 topics?
20 topics?
50 topics?
There is no universal answer.
The correct number depends on:
Dataset size
Dataset diversity
Desired level of detail
Application
Evaluation method
If you choose too few topics, unrelated subjects may be merged together.
If you choose too many topics, one meaningful subject may be split into several tiny topics.
22. Experimenting with Different Topic Counts
A practical approach is to train several models.
For example:
for k in [2, 3, 4, 5, 6, 8, 10]:
svd = TruncatedSVD(
n_components=k,
random_state=42
)
svd.fit_transform(X)
print(
k,
svd.explained_variance_ratio_.sum()
)
This allows you to examine how much variance the components capture.
However, explained variance alone does not tell you whether the topics are meaningful.
Human interpretation remains important.
23. Explained Variance
SVD provides:
svd.explained_variance_ratio_
This tells us how much variance each component explains.
For example:
Topic 1: 0.21
Topic 2: 0.17
Topic 3: 0.11
The first component captures more variance than the third.
You can calculate the total explained variance:
variance = svd.explained_variance_ratio_.sum()
print(variance)
A larger value means the reduced representation captures more of the variance in the original matrix.
But remember:
More variance does not automatically mean better topics.
Topic quality and interpretability matter too.
24. Visualizing LSA Topics
One major advantage of dimensionality reduction is visualization.
If we reduce documents to two dimensions, we can plot them.
import matplotlib.pyplot as plt
plt.figure(figsize=(10, 7))
plt.scatter(
lsa_matrix[:, 0],
lsa_matrix[:, 1]
)
plt.xlabel("LSA Component 1")
plt.ylabel("LSA Component 2")
plt.title("Documents in LSA Topic Space")
plt.show()
Each point represents a document.
Documents located close to each other have similar representations in the reduced semantic space.
This can help reveal clusters.
25. Labeling Documents with Their Dominant Topic
LSA provides a topic representation for each document.
For example:
topic_scores = svd.transform(X)
Then:
dominant_topics = topic_scores.argmax(axis=1)
This identifies the strongest component for each document.
We can print the results:
for index, topic in enumerate(dominant_topics):
print(
f"Document {index + 1}: "
f"Topic {topic + 1}"
)
This can be useful when organizing large collections of documents.
26. Calculating Topic Scores
Suppose a document has:
Topic 1 = 0.82
Topic 2 = 0.13
Topic 3 = 0.21
Topic 1 has the strongest score.
Therefore, we might classify that document as primarily belonging to Topic 1.
However, a document can contain multiple themes.
For example:
Artificial intelligence is being used in sports analytics.
This document could relate to both:
Artificial Intelligence
Sports
That is one reason topic modeling should not always be treated as simple single-label classification.
27. A Complete LSA Example
Here is a compact end-to-end implementation.
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.decomposition import TruncatedSVD
documents = [
"Python is used for machine learning and data science.",
"Machine learning models analyze data and make predictions.",
"Deep learning uses neural networks.",
"Neural networks are important in artificial intelligence.",
"Football teams train for competitive matches.",
"The football player scored two goals.",
"The team won the championship.",
"Football coaches prepare players for tournaments."
]
vectorizer = TfidfVectorizer(
stop_words="english"
)
X = vectorizer.fit_transform(documents)
svd = TruncatedSVD(
n_components=2,
random_state=42
)
lsa = svd.fit_transform(X)
terms = vectorizer.get_feature_names_out()
for topic_index, topic in enumerate(svd.components_):
top_indices = topic.argsort()[-10:][::-1]
print(f"\nTopic {topic_index + 1}")
for index in top_indices:
print(terms[index], end=" ")
print()
This is enough to build a basic LSA topic-modeling system.
28. Improving the Preprocessing Pipeline
Real-world text is usually much messier than our examples.
Consider:
"AI is AMAZING!!! Check www.example.com #technology"
A more advanced preprocessing pipeline might:
Convert text to lowercase
Remove URLs
Remove punctuation
Remove unnecessary symbols
Remove stopwords
Remove very short tokens
Normalize whitespace
Optionally lemmatize words
For some datasets, you may also want to preserve specific domain terms.
For example, in software engineering:
C++
Node.js
React
PyTorch
OpenAI
Removing punctuation blindly could damage these important terms.
Therefore, preprocessing should always depend on your dataset.
29. N-Grams with LSA
Single words are called unigrams.
But phrases can sometimes be much more meaningful.
For example:
machine learning
deep learning
neural network
natural language
We can capture these using n-grams.
With Scikit-learn:
vectorizer = TfidfVectorizer(
stop_words="english",
ngram_range=(1, 2)
)
This includes:
Unigrams
Bigrams
For example:
machine
learning
machine learning
This can improve topic interpretability in some datasets.
However, adding many n-grams increases the feature space.
30. Should You Use Stemming or Lemmatization?
Stemming reduces words to approximate roots.
For example:
playing
played
plays
might be reduced to a common stem.
Lemmatization attempts to convert words to their dictionary form.
For example:
running → run
better → good
depending on the linguistic model.
LSA does not require stemming or lemmatization.
You should test whether they improve your particular dataset.
Sometimes TF-IDF plus SVD already produces useful topics without complicated linguistic preprocessing.
31. LSA for News Articles
Imagine a news website with 50,000 articles.
The articles could cover:
Politics
Technology
Sports
Business
Science
Health
Entertainment
You can use LSA to discover recurring themes.
For example:
Topic 1:
market, company, investment, financial, business
Topic 2:
team, player, match, league, football
Topic 3:
software, technology, computer, AI, digital
Editors could use these topics to explore their content library.
32. LSA for Research Papers
LSA can also be useful for academic literature.
Suppose you have 20,000 research papers.
Potential topics might include:
machine learning
computer vision
natural language processing
robotics
optimization
medical imaging
Topic modeling can help researchers:
Explore large literature collections
Discover related papers
Identify research trends
Organize documents
Find broad thematic areas
LSA is particularly useful as an exploratory technique.
33. LSA for Customer Reviews
Consider an online store with thousands of reviews.
Instead of reading every review, LSA could reveal recurring themes.
For example:
Topic 1:
battery, charging, power, hours
Topic 2:
screen, display, brightness, resolution
Topic 3:
delivery, shipping, package, arrived
Topic 4:
price, expensive, value, money
This can help businesses identify the areas customers discuss most often.
34. LSA for Search and Information Retrieval
LSA has also historically been associated with information retrieval.
Imagine a user searches:
"vehicle safety"
A document might contain:
"automobile crash protection"
The exact words are different.
A purely lexical matching system might struggle.
LSA can potentially identify relationships between words based on their distribution across documents.
This can improve semantic matching in certain traditional information-retrieval settings.
Modern search systems often use more advanced embedding techniques, but LSA remains an important conceptual foundation.
35. LSA vs LDA
LSA is often compared with Latent Dirichlet Allocation (LDA).
Both can be used for topic modeling, but they work differently.
LSA
LSA uses:
Matrix representation
SVD
Linear algebra
Dimensionality reduction
LDA
LDA is a probabilistic generative model.
It represents:
Documents as mixtures of topics
Topics as probability distributions over words
For example:
Document:
70% Technology
30% Business
LDA can naturally represent this mixture.
LSA instead produces continuous latent component scores.
Neither approach is universally best.
The choice depends on your dataset and objective.
36. LSA vs Modern Embeddings
Modern NLP often uses embeddings generated by transformer-based models.
Examples include sentence embeddings and document embeddings.
These approaches can capture semantic relationships more effectively in many applications.
For example:
"car"
"automobile"
"vehicle"
can be represented as vectors that are close in semantic space.
LSA is much simpler.
Its advantages include:
Easy implementation
Low computational requirements
Mathematical simplicity
Fast experimentation
No need for a pretrained language model
This makes LSA excellent for learning NLP fundamentals and for certain lightweight applications.
37. Important Limitation: LSA Does Not Understand Context Like Transformers
Consider:
"I deposited money in the bank."
and:
"I sat beside the river bank."
The word "bank" has different meanings.
Traditional LSA relies heavily on statistical co-occurrence patterns.
It does not dynamically understand context in the way modern transformer models can.
Therefore, LSA should not be confused with contextual language models.
It is a powerful classical technique, but it has clear limitations.
38. Important Limitation: Topics Can Be Difficult to Interpret
Sometimes LSA produces strange topic combinations.
For example:
python
football
database
goal
server
player
You might wonder:
What topic is this?
This can happen because the mathematical components maximize certain properties of the matrix, not human interpretability.
A topic model is useful only if the resulting patterns make sense for your application.
39. Important Limitation: Choosing Too Many Topics
Suppose your dataset contains only 500 documents.
If you create 100 topics, the resulting topics may become fragmented.
You might get:
Topic 1:
python, code, function
Topic 2:
python, package, library
Topic 3:
code, software, programming
These may all represent essentially the same broader theme.
Start with a reasonable number of components and experiment.
40. Important Limitation: Choosing Too Few Topics
The opposite problem can also occur.
Suppose your dataset covers:
Sports
Programming
Finance
Medicine
Education
If you use only two components, very different subjects may become mixed.
For example:
Topic 1:
team, software, finance, players
Topic 2:
medicine, programming, education
This is not very useful.
The number of topics should reflect the diversity of the dataset.
41. Practical Tips for Better LSA Topics
Here are several useful strategies.
1. Clean the text carefully
Remove obvious noise but preserve meaningful domain-specific terms.
2. Use TF-IDF
TF-IDF often works better than raw word counts for LSA.
3. Experiment with topic counts
Try several values instead of choosing one number immediately.
4. Inspect top words
Always examine the most important terms in each component.
5. Use n-grams
Bigrams can make topics easier to interpret.
6. Remove extremely rare terms
Very rare words can introduce noise.
7. Evaluate manually
Human interpretation remains important.
8. Visualize documents
Two-dimensional representations can reveal clusters.
42. Removing Extremely Rare Terms
A word appearing in only one document may not be useful for discovering broad topics.
Scikit-learn allows you to control document frequency.
For example:
vectorizer = TfidfVectorizer(
stop_words="english",
min_df=2
)
Here:
min_df=2
means that a term must appear in at least two documents to be included.
You can also use:
max_df=0.8
to exclude terms that appear in a very large percentage of documents.
These parameters can significantly affect topic quality.
43. Creating a Reusable LSA Class
For larger projects, you can organize the model into a class.
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.decomposition import TruncatedSVD
class LSATopicModel:
def __init__(self, n_topics=5):
self.vectorizer = TfidfVectorizer(
stop_words="english"
)
self.svd = TruncatedSVD(
n_components=n_topics,
random_state=42
)
def fit(self, documents):
X = self.vectorizer.fit_transform(documents)
self.svd.fit(X)
def transform(self, documents):
X = self.vectorizer.transform(documents)
return self.svd.transform(X)
def get_topics(self, n_words=10):
terms = self.vectorizer.get_feature_names_out()
topics = []
for topic in self.svd.components_:
indices = topic.argsort()[-n_words:][::-1]
words = [
terms[i]
for i in indices
]
topics.append(words)
return topics
Now you can create:
model = LSATopicModel(n_topics=5)
model.fit(documents)
topics = model.get_topics(
n_words=10
)
for i, topic in enumerate(topics):
print(
f"Topic {i + 1}:",
", ".join(topic)
)
This structure is easier to reuse in applications.
44. Building a Topic Explorer
You can turn your LSA model into a simple topic-exploration tool.
A useful interface could display:
Topic 1 — Artificial Intelligence
---------------------------------
machine
learning
neural
model
data
Documents:
Document 1
Document 2
Document 4
Then:
Topic 2 — Sports
----------------
football
player
team
match
goals
Documents:
Document 5
Document 6
Document 7
This makes topic modeling useful beyond a simple Python experiment.
45. How LSA Can Help Your NLP Projects
If you're building an NLP application, LSA can serve as a lightweight baseline.
For example, you might use it for:
Document clustering
Content organization
Search exploration
Recommendation systems
Article categorization
Research discovery
Review analysis
News analysis
Text summarization support
Dataset exploration
A strong NLP workflow often begins with a simple baseline.
LSA is an excellent baseline because it is relatively easy to understand and implement.
46. A Practical Workflow for a Real Dataset
Suppose you have a CSV file:
articles.csv
with a column:
text
You could load it with Pandas:
import pandas as pd
df = pd.read_csv("articles.csv")
documents = df["text"].dropna().tolist()
Then create the vectorizer:
vectorizer = TfidfVectorizer(
stop_words="english",
min_df=2,
max_df=0.8,
ngram_range=(1, 2)
)
Transform the documents:
X = vectorizer.fit_transform(documents)
Apply LSA:
svd = TruncatedSVD(
n_components=10,
random_state=42
)
lsa = svd.fit_transform(X)
Now each document has a 10-dimensional topic representation.
47. Saving the Model
If you want to use the trained LSA model later, you can save it with joblib.
import joblib
joblib.dump(
vectorizer,
"tfidf_vectorizer.pkl"
)
joblib.dump(
svd,
"lsa_model.pkl"
)
Later:
vectorizer = joblib.load(
"tfidf_vectorizer.pkl"
)
svd = joblib.load(
"lsa_model.pkl"
)
This allows your application to reuse the trained model without rebuilding it every time.
48. Using LSA on New Documents
Suppose the model has already been trained.
You receive:
new_documents = [
"Neural networks are used to train modern AI models."
]
Transform it:
X_new = vectorizer.transform(
new_documents
)
Then project it into the LSA space:
new_topics = svd.transform(
X_new
)
Now the new document has a position in the learned latent topic space.
You can determine which topic has the strongest score:
topic = new_topics.argmax(axis=1)[0]
print(
"Dominant topic:",
topic + 1
)
This allows an existing LSA model to analyze incoming documents.
49. LSA as a Learning Tool for NLP
Even if you eventually plan to use advanced transformer models, learning LSA is valuable.
Why?
Because LSA teaches several fundamental concepts:
Text vectorization
Sparse matrices
TF-IDF
Dimensionality reduction
SVD
Latent representations
Topic modeling
Document similarity
Linear algebra in NLP
These ideas provide useful foundations for understanding more advanced NLP systems.
50. The Mathematical Picture
Let's summarize the mathematical process.
Start with documents.
Convert them into a matrix:
Apply SVD:
Keep only the first components:
The result is a lower-rank approximation of the original matrix.
In practical terms:
Huge text matrix
↓
SVD
↓
Keep important components
↓
Smaller latent representation
↓
Discover semantic patterns
This is the mathematical heart of LSA.
51. Why LSA Is Still Worth Learning
You may wonder:
If modern transformer models are more powerful, why learn LSA?
Because not every NLP problem requires a massive model.
LSA can be:
Fast
Lightweight
Easy to deploy
Easy to explain
Computationally inexpensive
Useful for exploratory analysis
It also teaches concepts that remain relevant in machine learning.
Understanding how classical NLP methods work gives you a stronger foundation for understanding modern approaches.
52. LSA Project Ideas
Once you understand the basics, try building one of these projects.
Project 1: News Topic Detector
Collect news articles and automatically discover their major themes.
Project 2: Research Paper Explorer
Use a collection of academic papers and visualize topic clusters.
Project 3: Book Topic Analyzer
Analyze chapters from books and identify recurring themes.
Project 4: Customer Review Explorer
Discover what customers frequently discuss.
Project 5: Blog Topic Organizer
Automatically group hundreds of blog posts into broad themes.
Project 6: Search Engine Prototype
Use LSA representations to explore semantic document similarity.
Project 7: NLP Dataset Explorer
Create a dashboard showing:
Topics
Top words
Documents per topic
Topic distributions
Two-dimensional visualization
These projects can turn theoretical knowledge into practical NLP experience.
53. Common Mistakes Beginners Make
Mistake 1: Expecting the model to name topics
LSA produces mathematical components, not human-readable topic names.
You must interpret the top words.
Mistake 2: Using an arbitrary number of topics
Always experiment with different component counts.
Mistake 3: Ignoring preprocessing
Poor text preprocessing can produce poor topics.
Mistake 4: Assuming high variance means perfect topics
Explained variance is useful, but interpretability matters.
Mistake 5: Treating LSA like a modern language model
LSA does not have the contextual understanding of transformer models.
Mistake 6: Using too little data
A tiny dataset may not contain enough information for meaningful topics.
Mistake 7: Removing useful domain words
Aggressive cleaning can destroy important signals.
54. LSA Checklist
Before considering your topic model complete, ask:
Do I have enough documents?
Have I cleaned obvious noise?
Have I considered stopwords?
Am I using TF-IDF?
Did I remove extremely rare terms?
Did I test different numbers of topics?
Did I inspect top topic words?
Are the topics interpretable?
Did I visualize the document space?
Did I evaluate the model on representative documents?
If the answer is yes, you have a strong starting point for an LSA-based topic-modeling project.
55. Final Complete Example
Here is a polished implementation that brings the main ideas together:
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.decomposition import TruncatedSVD
import matplotlib.pyplot as plt
documents = [
"Python is widely used for machine learning and artificial intelligence.",
"Machine learning models analyze data and make predictions.",
"Deep learning uses neural networks to solve complex problems.",
"Neural networks are important in modern artificial intelligence.",
"Football players train every day before important matches.",
"The football team won the championship after a difficult game.",
"The player scored two goals during the football match.",
"Coaches prepare football teams for competitive tournaments."
]
# Step 1: Convert documents to TF-IDF vectors
vectorizer = TfidfVectorizer(
stop_words="english",
ngram_range=(1, 2),
min_df=1
)
X = vectorizer.fit_transform(documents)
# Step 2: Create the LSA model
svd = TruncatedSVD(
n_components=2,
random_state=42
)
# Step 3: Transform documents
lsa_matrix = svd.fit_transform(X)
# Step 4: Display topics
terms = vectorizer.get_feature_names_out()
for topic_index, topic in enumerate(
svd.components_
):
top_indices = topic.argsort()[-10:][::-1]
words = [
terms[index]
for index in top_indices
]
print(
f"Topic {topic_index + 1}:",
", ".join(words)
)
# Step 5: Display dominant topic
dominant_topics = lsa_matrix.argmax(axis=1)
for i, topic in enumerate(
dominant_topics
):
print(
f"Document {i + 1}: "
f"Topic {topic + 1}"
)
# Step 6: Visualize
plt.figure(figsize=(10, 7))
plt.scatter(
lsa_matrix[:, 0],
lsa_matrix[:, 1]
)
for i in range(len(documents)):
plt.annotate(
str(i + 1),
(
lsa_matrix[i, 0],
lsa_matrix[i, 1]
)
)
plt.xlabel("LSA Component 1")
plt.ylabel("LSA Component 2")
plt.title("LSA Document Topic Space")
plt.show()
This example demonstrates the complete journey:
Documents
↓
TF-IDF
↓
Document-Term Matrix
↓
SVD
↓
Latent Components
↓
Topic Words
↓
Document Topic Scores
↓
VisualizationTo fully understand this topic, we recommend reading the previous lesson first. It explains the core concepts that this article builds upon.
Read the previous article here:
https://khayyamshah2007.blogspot.com/2026/08/lsa-topic-modeling-in-nlp-complete.html
Conclusion
Latent Semantic Analysis (LSA) is one of the classic techniques for topic modeling and dimensionality reduction in Natural Language Processing.
Its central idea is elegant:
Convert text into a mathematical matrix, decompose that matrix with SVD, and use the resulting latent dimensions to discover hidden patterns.
The process starts with raw documents.
Those documents are transformed into numerical vectors, commonly using TF-IDF. The resulting matrix is then decomposed using Singular Value Decomposition. By keeping a smaller number of components, LSA creates a compressed representation of the original text collection.
These latent components can reveal meaningful themes such as:
Topic 1 → Artificial Intelligence
Topic 2 → Sports
Topic 3 → Finance
Topic 4 → Education
The model does not actually know these labels. Instead, it provides groups of important words that humans interpret.
LSA is not a replacement for modern transformer-based NLP systems in every application. It has limitations, especially when dealing with context, polysemy, subtle semantics, and highly complex language. Nevertheless, it remains valuable because it is simple, fast, interpretable, and mathematically elegant.
More importantly, learning LSA builds a foundation for understanding how text can be represented as vectors and how mathematical transformations can reveal hidden structures inside language.
If you are learning NLP, LSA is an excellent bridge between traditional text-processing methods and modern semantic representation techniques.
Once you understand the progression:
Text
↓
Tokenization
↓
TF-IDF
↓
Matrix Representation
↓
SVD
↓
Latent Space
↓
Topics
you have learned one of the fundamental ideas behind classical NLP topic modeling.
And that foundation makes the next steps—such as LDA, word embeddings, Word2Vec, Doc2Vec, sentence embeddings, and transformer-based models—much easier to understand.
The most important lesson is not simply how to run TruncatedSVD.
It is understanding the idea behind it:
Large collections of text contain hidden patterns, and mathematics can help us uncover them.
Quick Reference
Main Libraries
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.decomposition import TruncatedSVD
TF-IDF
vectorizer = TfidfVectorizer(
stop_words="english"
)
X = vectorizer.fit_transform(documents)
LSA
svd = TruncatedSVD(
n_components=5,
random_state=42
)
lsa = svd.fit_transform(X)
Topic Words
terms = vectorizer.get_feature_names_out()
for topic in svd.components_:
indices = topic.argsort()[-10:][::-1]
print([
terms[i]
for i in indices
])
Dominant Topic
dominant_topics = lsa.argmax(axis=1)
Core Formula
Remember
LSA = TF-IDF + SVD + latent semantic representation
That simple combination makes LSA one of the most useful classical techniques to learn when beginning your journey into NLP.
Comments
Post a Comment