Naive Bayes Classifiers
In lecture, we learned about Bayes’ Theorem, an important result in probability theory that links the probability of an event happening given that a related event has happened to the probability of that related event happening. Bayes’ Theorem can be written formally as follows:
where P(A | B) is the probability that event A happens given that event B has happened, P(B | A) is the probability that event B happens given that event A has happened, and P(B) is the probability that event B happens.
Bayes’ Theorem is applied to a wide range of problems in many different disciplines and settings—far too many to enumerate here. One of the most important applications of Bayes’ Theorem, though, is the Naïve Bayes classifier, a simple probabilistic classifier that is used in everything from spam filtering to document classification.
To understand what a Naïve Bayes classifier is, we must first understand how to find the probability that an event A happens given that a series of events, B1 to Bn, has occurred—that is, P(A | B1 to Bn). To do this, we assume that events B1 to Bn are independent, meaning that we assume that the outcome of any of the events in the series does not affect the outcome of any of the other events.
This assumption, although naïve, makes our computations much easier—and it’s the reason why the Naïve Bayes classifier is called naïve. The formal definition of independence states that if are independent, then P(B1 to Bn | A), the probability that events B1 to Bn happen given that event A has happened, equals P(B1 | A)…P(Bn | A). Thus, after a few steps, we find that:
Now that we understand how to find the probability an event A happens given that a series of events B1 to Bn happened, we can understand what a Naïve Bayes classifier is. A Naïve Bayes classifier uses the modified version of Bayes’ Theorem involving a series of events happening to classify entities into different categories based on their characteristics.
For instance, say a Naïve Bayes classifier was being used to classify whether an email is spam or not, then it might try to compute , where “it contains the word X”…“it contains the word Y” are a series of events related to whether a certain word appears in the email. Then, the modified version of Bayes’ Theorem above would be used along with known probabilities about the chance that a spam email contains a given word to figure out what the probability that an email is spam given that that it contains certain words. This, however, is just one example of how to use a Naïve Bayes classifier as a spam filter—and, also, just one example of a Naïve Bayes classifier in general.
Source: http://www.cs.ucr.edu/~eamonn/CE/Bayesian%20Classification%20withInsect_examples.pdf