Bernoulli Naive Bayes, Defined: A Visible Information with Code Examples for Inexperienced persons | by Samy Baladram

A Light Introduction to Backtracking

Cease Chasing “Effectivity AI.” The Actual Worth Is in “Alternative AI.”

Unlocking Predictive Energy By way of Binary Simplicity

All illustrations on this article have been created by creator, incorporating licensed design components from Canva Professional.

In contrast to the baseline strategy of dummy classifiers or the similarity-based reasoning of KNN, Naive Bayes leverages likelihood principle. It combines the person chances of every “clue” (or function) to make a closing prediction. This easy but highly effective methodology has confirmed invaluable in numerous machine studying purposes.

Naive Bayes is a machine studying algorithm that makes use of likelihood to categorise information. It’s based mostly on Bayes’ Theorem, a components for calculating conditional chances. The “naive” half refers to its key assumption: it treats all options as impartial of one another, even when they may not be in actuality. This simplification, whereas typically unrealistic, enormously reduces computational complexity and works effectively in lots of sensible situations.

Naive Bayes strategies is an easy algorithms in machine studying utilizing likelihood as its base.

There are three primary kinds of Naive Bayes classifiers. The important thing distinction between these sorts lies within the assumption they make concerning the distribution of options:

Bernoulli Naive Bayes: Fitted to binary/boolean options. It assumes every function is a binary-valued (0/1) variable.
Multinomial Naive Bayes: Usually used for discrete counts. It’s typically utilized in textual content classification, the place options is likely to be phrase counts.
Gaussian Naive Bayes: Assumes that steady options comply with a standard distribution.

Bernoulli NB assumes binary information, Multinomial NB works with discrete counts, and Gaussian NB handles steady information assuming a standard distribution.

It’s a good begin to concentrate on the best one which is Bernoulli NB. The “Bernoulli” in its identify comes from the idea that every function is binary-valued.

All through this text, we’ll use this synthetic golf dataset (impressed by [1]) for example. This dataset predicts whether or not an individual will play golf based mostly on climate situations.

Columns: ‘Outlook’, ‘Temperature’ (in Fahrenheit), ‘Humidity’ (in %), ‘Wind’ and ‘Play’ (goal function)

# IMPORTING DATASET #
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score
import pandas as pd
import numpy as npdataset_dict = {
'Outlook': ['sunny', 'sunny', 'overcast', 'rain', 'rain', 'rain', 'overcast', 'sunny', 'sunny', 'rain', 'sunny', 'overcast', 'overcast', 'rain', 'sunny', 'overcast', 'rain', 'sunny', 'sunny', 'rain', 'overcast', 'rain', 'sunny', 'overcast', 'sunny', 'overcast', 'rain', 'overcast'],
'Temperature': [85.0, 80.0, 83.0, 70.0, 68.0, 65.0, 64.0, 72.0, 69.0, 75.0, 75.0, 72.0, 81.0, 71.0, 81.0, 74.0, 76.0, 78.0, 82.0, 67.0, 85.0, 73.0, 88.0, 77.0, 79.0, 80.0, 66.0, 84.0],
'Humidity': [85.0, 90.0, 78.0, 96.0, 80.0, 70.0, 65.0, 95.0, 70.0, 80.0, 70.0, 90.0, 75.0, 80.0, 88.0, 92.0, 85.0, 75.0, 92.0, 90.0, 85.0, 88.0, 65.0, 70.0, 60.0, 95.0, 70.0, 78.0],
'Wind': [False, True, False, False, False, True, True, False, False, False, True, True, False, True, True, False, False, True, False, True, True, False, True, False, False, True, False, False],
'Play': ['No', 'No', 'Yes', 'Yes', 'Yes', 'No', 'Yes', 'No', 'Yes', 'Yes', 'Yes', 'Yes', 'Yes', 'No', 'No', 'Yes', 'Yes', 'No', 'No', 'No', 'Yes', 'Yes', 'Yes', 'Yes', 'Yes', 'Yes', 'No', 'Yes']
}
df = pd.DataFrame(dataset_dict)
# ONE-HOT ENCODE 'Outlook' COLUMN
df = pd.get_dummies(df, columns=['Outlook'],  prefix='', prefix_sep='', dtype=int)
# CONVERT 'Windy' (bool) and 'Play' (binary) COLUMNS TO BINARY INDICATORS
df['Wind'] = df['Wind'].astype(int)
df['Play'] = (df['Play'] == 'Sure').astype(int)
# Set function matrix X and goal vector y
X, y = df.drop(columns='Play'), df['Play']
# Break up the information into coaching and testing units
X_train, X_test, y_train, y_test = train_test_split(X, y, train_size=0.5, shuffle=False)
print(pd.concat([X_train, y_train], axis=1), finish='nn')
print(pd.concat([X_test, y_test], axis=1))

We’ll adapt it barely for Bernoulli Naive Bayes by changing our options to binary.

As all the information must be in 0 & 1 format, the ‘Outlook’ is one-hot encoded whereas the Temperature is separated into ≤ 80 and > 80. Equally, Humidity is separated into ≤ 75 and > 75.

# One-hot encode the categorized columns and drop them after, however do it individually for coaching and take a look at units
# Outline classes for 'Temperature' and 'Humidity' for coaching set
X_train['Temperature'] = pd.lower(X_train['Temperature'], bins=[0, 80, 100], labels=['Warm', 'Hot'])
X_train['Humidity'] = pd.lower(X_train['Humidity'], bins=[0, 75, 100], labels=['Dry', 'Humid'])# Equally, outline for the take a look at set
X_test['Temperature'] = pd.lower(X_test['Temperature'], bins=[0, 80, 100], labels=['Warm', 'Hot'])
X_test['Humidity'] = pd.lower(X_test['Humidity'], bins=[0, 75, 100], labels=['Dry', 'Humid'])
# One-hot encode the categorized columns
one_hot_columns_train = pd.get_dummies(X_train[['Temperature', 'Humidity']], drop_first=True, dtype=int)
one_hot_columns_test = pd.get_dummies(X_test[['Temperature', 'Humidity']], drop_first=True, dtype=int)
# Drop the categorized columns from coaching and take a look at units
X_train = X_train.drop(['Temperature', 'Humidity'], axis=1)
X_test = X_test.drop(['Temperature', 'Humidity'], axis=1)
# Concatenate the one-hot encoded columns with the unique DataFrames
X_train = pd.concat([one_hot_columns_train, X_train], axis=1)
X_test = pd.concat([one_hot_columns_test, X_test], axis=1)
print(pd.concat([X_train, y_train], axis=1), 'n')
print(pd.concat([X_test, y_test], axis=1))

Bernoulli Naive Bayes operates on information the place every function is both 0 or 1.

Calculate the likelihood of every class within the coaching information.
For every function and sophistication, calculate the likelihood of the function being 1 and 0 given the category.
For a brand new occasion: For every class, multiply its likelihood by the likelihood of every function worth (0 or 1) for that class.
Predict the category with the very best ensuing likelihood.

For our golf dataset, a Bernoulli NB classifier take a look at the likelihood of every function taking place for every class (YES & NO) then make choice based mostly on which class has larger likelihood.

The coaching course of for Bernoulli Naive Bayes includes calculating chances from the coaching information:

Class Chance Calculation: For every class, calculate its likelihood: (Variety of cases on this class) / (Complete variety of cases)

In our golf instance, the algorithm would calculate how typically golf is performed total.

from fractions import Fractiondef calc_target_prob(attr):
total_counts = attr.value_counts().sum()
prob_series = attr.value_counts().apply(lambda x: Fraction(x, total_counts).limit_denominator())
return prob_series
print(calc_target_prob(y_train))

2.Characteristic Chance Calculation: For every function and every class, calculate:

(Variety of cases the place function is 0 on this class) / (Variety of cases on this class)
(Variety of cases the place function is 1 on this class) / (Variety of cases on this class)

For every climate situation (e.g., sunny), how typically golf is performed when it’s sunny and the way typically it’s not performed when it’s sunny.

from fractions import Fractiondef sort_attr_label(attr, lbl):
return (pd.concat([attr, lbl], axis=1)
.sort_values([attr.name, lbl.name])
.reset_index()
.rename(columns={'index': 'ID'})
.set_index('ID'))
def calc_feature_prob(attr, lbl):
total_classes = lbl.value_counts()
counts = pd.crosstab(attr, lbl)
prob_df = counts.apply(lambda x: [Fraction(c, total_classes[x.name]).limit_denominator() for c in x])
return prob_df
print(sort_attr_label(y_train, X_train['sunny']))
print(calc_feature_prob(X_train['sunny'], y_train))

The identical course of is utilized to the entire different options.

for col in X_train.columns:
print(calc_feature_prob(X_train[col], y_train), "n")

3. Smoothing (Optionally available): Add a small worth (normally 1) to the numerator and denominator of every likelihood calculation to keep away from zero chances

We add 1 to all numerators, and add 2 to all denominators, to maintain the full class likelihood 1.

# In sklearn, all processes above is summarized on this 'match' methodology:
from sklearn.naive_bayes import BernoulliNB
nb_clf = BernoulliNB(alpha=1)
nb_clf.match(X_train, y_train)

4. Retailer Outcomes: Save all calculated chances to be used throughout classification.

Smoothing is already utilized to all function chances. We are going to use these tables to make predictions.

Given a brand new occasion with options which are both 0 or 1:

Chance Assortment: For every doable class:

Begin with the likelihood of this class occurring (class likelihood).
For every function within the new occasion, acquire the likelihood of this function being 0/1 for this class.

For ID 14, we choose the possibilities of every of the function (both 0 or 1) taking place.

2. Rating Calculation & Prediction: For every class:

Multiply all of the collected chances collectively
The result’s the rating for this class
The category with the very best rating is the prediction

After multiplying the category likelihood and the entire function chances, we choose the category that has the upper rating.

y_pred = nb_clf.predict(X_test)
print(y_pred)

This straightforward probabilistic mannequin give an excellent accuracy for this easy dataset.

# Consider the classifier
print(f"Accuracy: {accuracy_score(y_test, y_pred)}")

Bernoulli Naive Bayes has a number of essential parameters:

Alpha (α): That is the smoothing parameter. It provides a small rely to every function to stop zero chances. Default is normally 1.0 (Laplace smoothing) as what was proven earlier than.
Binarize: In case your options aren’t already binary, this threshold converts them. Any worth above this threshold turns into 1, and any worth beneath turns into 0.

For BernoulliNB in scikit-learn, numerical options are sometimes standardized moderately than manually binarized. The mannequin then internally converts these standardized values to binary, normally utilizing 0 (the imply) as the edge.

3. Match Prior: Whether or not to be taught class prior chances or assume uniform priors (50/50).

For our golf dataset, we would begin with the default α=1.0, no binarization (since we’ve already made our options binary), and fit_prior=True.

Like all algorithm in machine studying, Bernoulli Naive Bayes has its strengths and limitations.

Simplicity: Simple to implement and perceive.
Effectivity: Quick to coach and predict, works effectively with giant function areas.
Efficiency with Small Datasets: Can carry out effectively even with restricted coaching information.
Handles Excessive-Dimensional Knowledge: Works effectively with many options, particularly in textual content classification.

Independence Assumption: Assumes all options are impartial, which is commonly not true in real-world information.
Restricted to Binary Options: In its pure type, solely works with binary information.
Sensitivity to Enter Knowledge: Will be delicate to how the options are binarized.
Zero Frequency Drawback: With out smoothing, zero chances can strongly have an effect on predictions.

The Bernoulli Naive Bayes classifier is an easy but highly effective machine studying algorithm for binary classification. It excels in textual content evaluation and spam detection, the place options are sometimes binary. Recognized for its pace and effectivity, this probabilistic mannequin performs effectively with small datasets and high-dimensional areas.

Regardless of its naive assumption of function independence, it typically rivals extra complicated fashions in accuracy. Bernoulli Naive Bayes serves as a superb baseline and real-time classification device.

# Import wanted libraries
import pandas as pd
from sklearn.naive_bayes import BernoulliNB
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import accuracy_score
from sklearn.model_selection import train_test_split# Load the dataset
dataset_dict = {
'Outlook': ['sunny', 'sunny', 'overcast', 'rainy', 'rainy', 'rainy', 'overcast', 'sunny', 'sunny', 'rainy', 'sunny', 'overcast', 'overcast', 'rainy', 'sunny', 'overcast', 'rainy', 'sunny', 'sunny', 'rainy', 'overcast', 'rainy', 'sunny', 'overcast', 'sunny', 'overcast', 'rainy', 'overcast'],
'Temperature': [85.0, 80.0, 83.0, 70.0, 68.0, 65.0, 64.0, 72.0, 69.0, 75.0, 75.0, 72.0, 81.0, 71.0, 81.0, 74.0, 76.0, 78.0, 82.0, 67.0, 85.0, 73.0, 88.0, 77.0, 79.0, 80.0, 66.0, 84.0],
'Humidity': [85.0, 90.0, 78.0, 96.0, 80.0, 70.0, 65.0, 95.0, 70.0, 80.0, 70.0, 90.0, 75.0, 80.0, 88.0, 92.0, 85.0, 75.0, 92.0, 90.0, 85.0, 88.0, 65.0, 70.0, 60.0, 95.0, 70.0, 78.0],
'Wind': [False, True, False, False, False, True, True, False, False, False, True, True, False, True, True, False, False, True, False, True, True, False, True, False, False, True, False, False],
'Play': ['No', 'No', 'Yes', 'Yes', 'Yes', 'No', 'Yes', 'No', 'Yes', 'Yes', 'Yes', 'Yes', 'Yes', 'No', 'No', 'Yes', 'Yes', 'No', 'No', 'No', 'Yes', 'Yes', 'Yes', 'Yes', 'Yes', 'Yes', 'No', 'Yes']
}
df = pd.DataFrame(dataset_dict)
# Put together information for mannequin
df = pd.get_dummies(df, columns=['Outlook'],  prefix='', prefix_sep='', dtype=int)
df['Wind'] = df['Wind'].astype(int)
df['Play'] = (df['Play'] == 'Sure').astype(int)
# Break up information into coaching and testing units
X, y = df.drop(columns='Play'), df['Play']
X_train, X_test, y_train, y_test = train_test_split(X, y, train_size=0.5, shuffle=False)
# Scale numerical options (for computerized binarization)
scaler = StandardScaler()
float_cols = X_train.select_dtypes(embody=['float64']).columns
X_train[float_cols] = scaler.fit_transform(X_train[float_cols])
X_test[float_cols] = scaler.rework(X_test[float_cols])
# Prepare the mannequin
nb_clf = BernoulliNB()
nb_clf.match(X_train, y_train)
# Make predictions
y_pred = nb_clf.predict(X_test)
# Examine accuracy
print(f"Accuracy: {accuracy_score(y_test, y_pred)}")