Subject modeling stays a vital instrument within the AI and NLP toolbox. Whereas massive language fashions (LLMs) deal with textual content exceptionally nicely, extracting high-level subjects from huge datasets nonetheless requires devoted matter modeling methods. A typical workflow contains 4 core steps: embedding, dimensionality discount, clustering, and matter illustration.
frameworks at this time is BERTopic, which simplifies every stage with modular elements and an intuitive API. On this put up, I’ll stroll by way of sensible changes you may make to enhance clustering outcomes and increase interpretability primarily based on hands-on experiments utilizing the open-source 20 Newsgroups dataset, which is distributed underneath the Artistic Commons Attribution 4.0 Worldwide license.
Undertaking Overview
We’ll begin with the default settings really useful in BERTopic’s documentation and progressively replace particular configurations to focus on their results. Alongside the way in which, I’ll clarify the aim of every module and learn how to make knowledgeable selections when customizing them.
Dataset Preparation
We load a pattern of 500 information paperwork.
import random
from datasets import load_dataset
dataset = load_dataset("SetFit/20_newsgroups")
random.seed(42)
text_label = listing(zip(dataset["train"]["text"], dataset["train"]["label_text"]))
text_label_500 = random.pattern(text_label, 500)Because the information originates from informal Usenet discussions, we apply cleansing steps to strip headers, take away litter, and protect solely informative sentences.
This preprocessing ensures higher-quality embeddings and a smoother downstream clustering course of.
import re
def clean_for_embedding(textual content, max_sentences=5):
traces = textual content.cut up("n")
traces = [line for line in lines if not line.strip().startswith(">")]
traces = [line for line in lines if not re.match
(r"^s*(from|subject|organization|lines|writes|article)s*:", line, re.IGNORECASE)]
textual content = " ".be part of(traces)
textual content = re.sub(r"s+", " ", textual content).strip()
textual content = re.sub(r"[!?]{3,}", "", textual content)
sentence_split = re.cut up(r'(?<=[.!?]) +', textual content)
sentence_split = [
s for s in sentence_split
if len(s.strip()) > 15 and not s.strip().isupper()
]
return " ".be part of(sentence_split[:max_sentences])
texts_clean = [clean_for_embedding(text) for text,_ in text_label_500]
labels = [label for _, label in text_label_500]Preliminary BERTopic Pipeline
Utilizing BERTopic’s modular design, we configure every element: SentenceTransformer for embeddings, UMAP for dimensionality discount, HDBSCAN for clustering, and CountVectorizer + KeyBERT for matter illustration. This setup yields only some broad subjects with noisy representations, highlighting the necessity for fine-tuning to realize extra coherent outcomes.
from bertopic import BERTopic
from umap import UMAP
from hdbscan import HDBSCAN
from sentence_transformers import SentenceTransformer
from sklearn.feature_extraction.textual content import CountVectorizer
from bertopic.vectorizers import ClassTfidfTransformer
from bertopic.illustration import KeyBERTInspired
# Step 1 - Extract embeddings
embedding_model = SentenceTransformer("all-MiniLM-L6-v2")
# Step 2 - Cut back dimensionality
umap_model = UMAP(n_neighbors=10, n_components=5, min_dist=0.0, metric='cosine', random_state=42)
# Step 3 - Cluster diminished embeddings
hdbscan_model = HDBSCAN(min_cluster_size=15, metric='euclidean', cluster_selection_method='eom', prediction_data=True)
# Step 4 - Tokenize subjects
vectorizer_model = CountVectorizer(stop_words="english")
# Step 5 - Create matter illustration
ctfidf_model = ClassTfidfTransformer()
# Step 6 - (Non-obligatory) High-quality-tune matter representations with
# a `bertopic.illustration` mannequin
representation_model = KeyBERTInspired()
# All steps collectively
topic_model = BERTopic(
embedding_model=embedding_model, # Step 1 - Extract embeddings
umap_model=umap_model, # Step 2 - Cut back dimensionality
hdbscan_model=hdbscan_model, # Step 3 - Cluster diminished embeddings
vectorizer_model=vectorizer_model, # Step 4 - Tokenize subjects
ctfidf_model=ctfidf_model, # Step 5 - Extract matter phrases
representation_model=representation_model # Step 6 - (Non-obligatory) High-quality-tune matter representations
)
subjects, probs = topic_model.fit_transform(texts_clean)This setup yields only some broad subjects with noisy representations. This outcome highlights the necessity for finetuning to realize extra coherent outcomes.
Parameter Tuning for Granular Matters
n_neighbors from UMAP module
UMAP is the dimensionality discount module to scale back origin embedding to a decrease dimension dense vector. By adjusting UMAP’s n_neighbors, we management how regionally or globally the information is interpreted throughout dimensionality discount. Decreasing this worth uncovers finer-grained clusters and improves matter distinctiveness.
umap_model_new = UMAP(n_neighbors=5, n_components=5, min_dist=0.0, metric='cosine', random_state=42)
topic_model.umap_model = umap_model_new
subjects, probs = topic_model.fit_transform(texts_clean)
topic_model.get_topic_info()
min_cluster_size and cluster_selection_method from HDBSCAN module
HDBSCAN is the clustering module set by default for BerTopic. By modifying HDBSCAN’s min_cluster_size and switching the cluster_selection_method from “eom” to “leaf” additional sharpens matter decision. These settings assist uncover smaller, extra targeted themes and steadiness the distribution throughout clusters.
hdbscan_model_leaf = HDBSCAN(min_cluster_size=5, metric='euclidean', cluster_selection_method='leaf', prediction_data=True)
topic_model.hdbscan_model = hdbscan_model_leaf
subjects, _ = topic_model.fit_transform(texts_clean)
topic_model.get_topic_info()The variety of clusters will increase to 30 by setting cluster_selection_method to leaf and min_cluster_size to five.
Controlling Randomness for Reproducibility
UMAP is inherently non-deterministic, which means it could produce completely different outcomes on every run except you explicitly set a hard and fast random_state. This element is commonly omitted in instance code, so remember to embody it to make sure reproducibility.
Equally, in the event you’re utilizing a third-party embedding API (like OpenAI), be cautious. Some APIs introduce slight variations on repeated calls. For reproducible outputs, cache embeddings and feed them instantly into BERTopic.
from bertopic.backend import BaseEmbedder
import numpy as np
class CustomEmbedder(BaseEmbedder):
"""Lightweight wrapper to name NVIDIA's embedding endpoint by way of OpenAI SDK."""
def __init__(self, embedding_model, consumer):
tremendous().__init__()
self.embedding_model = embedding_model
self.consumer = consumer
def encode(self, paperwork): # sort: ignore[override]
response = self.consumer.embeddings.create(
enter=paperwork,
mannequin=self.embedding_model,
encoding_format="float",
extra_body={"input_type": "passage", "truncate": "NONE"},
)
embeddings = np.array([embed.embedding for embed in response.data])
return embeddings
topic_model.embedding_model = CustomEmbedder()
subjects, probs = topic_model.fit_transform(texts_clean, embeddings=embeddings)Each dataset area might require completely different clustering settings for optimum outcomes. To streamline experimentation, contemplate defining analysis standards and automating the tuning course of. For this tutorial, we’ll use the cluster configuration that units n_neighbors to five, min_cluster_size to five, and cluster_selection_method to “eom”. This can be a mixture that strikes a steadiness between granularity and coherence.
Enhancing Subject Representations
Illustration performs a vital position in making clusters interpretable. By default, BERTopic generates unigram-based representations, which regularly lack adequate context. Within the subsequent part, we’ll discover a number of methods to counterpoint these representations and enhance matter interpretability.
Ngram
n-gram vary
In BERTopic, CountVectorizer is the default instrument to transform textual content information into bag-of-words representations. As a substitute of counting on generic unigrams, swap to bigrams or trigrams utilizing ngram_range in CountVectorizer. This straightforward change provides a lot wanted context.
Since we’re solely updating illustration, BerTopic gives the update_topics operate to keep away from redoing the modeling once more.
topic_model.update_topics(texts_clean, vectorizer_model=CountVectorizer(stop_words="english", ngram_range=(2,3)))
topic_model.get_topic_info()
Customized Tokenizer
Some bigrams are nonetheless laborious to interpret e.g. 486dx 50, ac uk, dxf doc,… For larger management, implement a customized tokenizer that filters n-grams primarily based on part-of-speech patterns. This removes meaningless mixtures and elevates the standard of your matter key phrases.
import spacy
from typing import Listing
class ImprovedTokenizer:
def __init__(self):
self.nlp = spacy.load("en_core_web_sm", disable=["parser", "ner"])
self.MEANINGFUL_BIGRAMS = {
("ADJ", "NOUN"),
("NOUN", "NOUN"),
("VERB", "NOUN"),
}
# Hold solely probably the most significant syntactic bigram patterns
def __call__(self, textual content: str, max_tokens=200) -> Listing[str]:
doc = self.nlp(textual content[:3000]) # truncate lengthy docs for velocity
tokens = [(t.text, t.lemma_.lower(), t.pos_) for t in doc if t.is_alpha]
bigrams = []
for i in vary(len(tokens) - 1):
word1, lemma1, pos1 = tokens[i]
word2, lemma2, pos2 = tokens[i + 1]
if (pos1, pos2) in self.MEANINGFUL_BIGRAMS:
# Optionally lowercase each phrases to normalize
bigrams.append(f"{lemma1} {lemma2}")
return bigrams
topic_model.update_topics(docs=texts_clean,vectorizer_model=CountVectorizer(tokenizer=ImprovedTokenizer()))
topic_model.get_topic_info()
LLM
Lastly, you may combine LLMs to generate coherent titles or summaries for every matter. BERTopic helps OpenAI integration instantly or by way of customized prompting. These LLM-based summaries drastically enhance explainability.
import openai
from bertopic.illustration import OpenAI
consumer = openai.OpenAI(api_key=os.environ["OPENAI_API_KEY"])
topic_model.update_topics(texts_clean, representation_model=OpenAI(consumer, mannequin="gpt-4o-mini", delay_in_seconds=5))
topic_model.get_topic_info()The representations at the moment are all significant sentences.
You can even write your individual operate for getting the LLM-generated title, and replace it again to the subject mannequin object by utilizing update_topic_labels operate. Please seek advice from the instance code snippet under.
import openai
from typing import Listing
def generate_topic_titles_with_llm(
topic_model,
docs: Listing[str],
api_key: str,
mannequin: str = "gpt-4o"
) -> Dict[int, Tuple[str, str]]:
consumer = openai.OpenAI(api_key=api_key)
topic_info = topic_model.get_topic_info()
topic_repr = {}
subjects = topic_info[topic_info.Topic != -1].Subject.tolist()
for matter in tqdm(subjects, desc="Producing titles"):
indices = [i for i, t in enumerate(topic_model.topics_) if t == topic]
if not indices:
proceed
top_doc = docs[indices[0]]
immediate = f"""You're a useful summarizer for matter clustering.
Given the next textual content that represents a subject, generate:
1. A brief **title** for the subject (2–6 phrases)
2. A one or two sentence **abstract** of the subject.
Textual content:
"""
{top_doc}
"""
"""
attempt:
response = consumer.chat.completions.create(
mannequin=mannequin,
messages=[
{"role": "system", "content": "You are a helpful assistant for summarizing topics."},
{"role": "user", "content": prompt}
],
temperature=0.5
)
output = response.selections[0].message.content material.strip()
traces = output.cut up('n')
title = traces[0].exchange("Title:", "").strip()
abstract = traces[1].exchange("Abstract:", "").strip() if len(traces) > 1 else ""
topic_repr[topic] = (title, abstract)
besides Exception as e:
print(f"Error with matter {matter}: {e}")
topic_repr[topic] = ("[Error]", str(e))
return topic_repr
topic_repr = generate_topic_titles_with_llm( topic_model, texts_clean, os.environ["OPENAI_API_KEY"])
topic_repr_dict = {
matter: topic_repr.get(matter, "Subject")
for matter in matter.get_topic_info()["Topic"]
}
topic_model.set_topic_labels(topic_repr_dict)Conclusion
This information outlined actionable methods to spice up matter modeling outcomes utilizing BERTopic. By understanding the position of every module and tuning parameters in your particular area, you may obtain extra targeted, secure, and interpretable subjects.
Illustration issues simply as a lot as clustering. Whether or not it’s by way of n-grams, syntactic filtering, or LLMs, investing in higher representations makes your subjects simpler to grasp and extra helpful in apply.
BERTopic additionally gives superior modeling methods past the fundamentals coated right here. In a future put up, we’ll discover these capabilities in depth. Keep tuned!
