Generative vs Discriminative Models in Machine Learning

In machine learning, classification is a common job. Examples include churn prediction, spam email detection, and image categorization. Tasks involving classification can be accomplished by a variety of algorithms. These algorithms can be divided into two major categories: discriminative and generative.

 

We will attempt to explain the distinction between generative and discriminative classifiers in this piece, as well as how they classify data. Simply said, a discriminative model can be used to classification or regression problem statements and uses conditional probability to generate predictions on previously unknown data. A generative model, on the other hand, focuses on the dataset's distribution to provide a probability for a specific occurrence.

 

Generative Models

 

Generative models are those in which the learning algorithms typically model the underlying patterns/distribution of the data points and the focus is on the distribution of particular classes in a dataset. These models produce situations in which a given feature (x)/input and the desired output/label (y) exist simultaneously by using the concept of joint probability in theory.

 

To model data points and distinguish between several class labels in a dataset, generative models make use of probability estimates and likelihood. These models have the capacity to produce fresh data instances. They do, however, have a significant flaw. These models are significantly impacted by the existence of outliers.

 

Examples of Generative Models

 

We’ve included some of the examples of Generative Models below :

 

1. LDA, or linear discriminant analysis

 

The variance and mean of the data for each class in the dataset are estimated by LDA models. After calculating the means and variances for each class, predictions may be formed by calculating the likelihood that a specific combination of inputs belongs to a particular class.

 

2. Hidden Markov Models (HMMs)

 

Markov Chains can be compared to probability graphs that show how likely it is for us to go from one chain point, or "state," to another. The chance of changing from state j to state I indicated as p, is calculated using Markov chains (i,j). 

 

Simply put, this is the previously described joint probability. A hidden, unobservable Markov chain is utilized in a hidden Markov model. The probabilities for the present state and the state that came before it are used to compute the most likely outcome when the data inputs are given to the model.

 

3. Bayesian networks

 

An example of a probabilistic graphical model is a Bayesian network. A Directed Acyclic Graph can be used to depict the conditional dependencies between the variables. In a Bayesian network, each node relates to a particular variable, and each edge of the graph denotes a conditional reliance. 

 

The joint distribution of the variables and the joint probability can be calculated using the conditional independence for the special relationships in the graph. To put it another way, a Bayesian network represents a portion of the independent relationships in a certain joint probability distribution.

 

A Bayesian network can be used to calculate the probability of occurrences or outcomes once it has been established and properly characterized, with Random Variables, Conditional Relationships, and Probability Distributions known.

 

A Naive Bayes model is one of the most frequently used varieties of Bayesian networks. By treating all attributes as independent of one another, a Naive Bayes model overcomes the difficulty of computing probability for datasets with several parameters or variables.

 

Also Read | What is Naive Bayes Algorithm in Machine Learning

 

Discriminative Models

 

Conditional models, also known as discriminative models, have a tendency to identify the boundaries between classes and labels in a dataset. Finding the decision border dividing one class from another is the aim here, as opposed to generative models.

 

Consequently, discriminative models separate classes by instead modeling the conditional probability and do not make any assumptions about the data point, in contrast to generative models, which tend to model the joint probability of data points and are capable of producing new instances using probability estimates and maximum likelihood. Additionally, they are unable to produce new data instances.

 

In contrast to generative models, discriminative models have the advantage of being more resistant to outliers. However, a misclassification issue, or incorrectly identifying a data point, is a significant disadvantage.

 

A discriminative model focuses on predicting the labels of the data, whereas a generative model focuses on describing how the data was formed. This is another significant distinction between the two types of models.

 

Examples of Discriminative Models

 

Here are some examples of Discriminative Models below : 

 

1. Support vector machines (SVMs)

 

In order to separate the various classes in the dataset, support vector machines build a decision boundary between each pair of data points. For 2-dimensional and 3-dimensional spaces, the SVM algorithm, respectively, creates lines or hyperplanes that divide points. 

 

By attempting to maximize the margin, or the distance from the line/hyperplane to the closest points, SVM seeks to identify the line/hyperplane that best distinguishes the classes. By employing the "kernel technique" to locate non-linear decision boundaries, SVM models can also be applied to datasets that aren't linearly separable.

 

2. Logistic Regression

 

This algorithm calculates the likelihood that an input will fall into one of two states using the logit (log-odds) function. The probability is "squished" towards either 0 or 1, true or false, using a sigmoid function. Class 1 probabilities are those more than 0.50, whereas class 0 probability are those of 0.49 or less. 

 

Logistic regression is therefore frequently applied to binary classification issues. However, by employing a one vs. all strategy, developing a binary classification model for each class, and calculating the likelihood that a given example belongs to the target class or another class in the dataset, logistic regression can be used to solve multi-class problems.

 

3.  Decision Tree

 

A decision tree model works by gradually dividing a dataset into smaller and smaller subgroups; when the subsets can no longer be divided further, the model produces a tree with nodes and leaves. Decisions concerning data points are made at nodes in a decision tree using various filtering criteria. 

 

The categorized data points are represented as leaves in a decision tree. Both category and numerical data can be handled by decision tree methods, and the splits in the tree are based on particular variables or features.

 

4. Random Forests

 

In essence, a random forest model is merely a number of decision trees, with the predictions from each tree averaged to provide the final result. The random forest algorithm builds the individual trees based on a set of randomly chosen observations and features.

 

Discriminative vs. Generative Models

 

• Assume for the moment that our job is to identify the language of a text document. How can machine learning be used to complete this task? Each language can be learned, and then we can identify it. Generative models operate in this manner.

 

As an alternative, we can study only the linguistic similarities and differences among languages without actually learning any of them. The discriminatory strategy is this. We don't speak any languages here.

 

• In other words, case distinction is the main emphasis of discriminative algorithms. They concentrate on learning a decision limit as a result. Generative algorithms, on the other hand, learn the essential characteristics of the data and how to create it from scratch.

 

The discriminative approach concentrates on a solution, whereas the generative approach concentrates on modeling. Therefore, we can create fresh data points using generative algorithms. Discriminatory algorithms are useless for that.

 

But for classification tasks, discriminative algorithms typically perform better. That's as a result of their direct approach to problem solving rather than their initial focus on more general issues.

 

• The ability of generative algorithms to represent intricate relationships between variables, however, is their true strength. They are hence capable of explanation. As a result, they have applications in NLP and medicine that are successful.

 

Contrarily, discriminative algorithms lack the ability to formally define the limitations of their decisions, making them feel like opaque black boxes. There is no direct explanation for the correlations between the variables. As a result, it is difficult for us to imagine.

 

• Additionally, because generative models are predictive, they can be used to tackle both supervised and unsupervised learning tasks. Discriminative models can't infer information from context; they need labeled datasets. Therefore, generative models have broader applicability in the domains of anomaly identification and monitoring.

 

Additionally, generative algorithms reach convergence more quickly than discriminative ones. Therefore, when we just have a tiny training dataset, we favor generative models.

 

The asymptotic error is higher for the generative models even if they converge more quickly. The discriminative models, on the other hand, converge to a reduced asymptotic error. Therefore, the error rate for the discriminative models lowers as the number of training samples grows.

 

Also Read | Life Cycle of a Machine Learning Model

 

Differences Between Generative and Discriminative

 

The main distinctions between generative and discriminative models are outlined here.

 

Generative models

 

1. The actual distribution of the classes in the dataset is what generative models attempt to represent.

 

2. Using the Bayes Theorem, generative models forecast the joint probability distribution, p(x,y).

 

3. In comparison to discriminative models, generative models require more computational resources.

 

4. Unsupervised machine learning problems benefit from the usage of generative models.

 

5. Outliers have a greater influence on generative models than on discriminative ones.

 

Discriminative Models

 

1. Discriminative models simulate the dataset classes' decision boundaries.

 

2. Discriminative models pick up on p(y|x), or conditional probability.

 

3. In comparison to generative models, discriminative models are computationally less expensive.

 

4. For supervised machine learning tasks, discriminative models are helpful.

 

5. Unlike generative models, discriminative models have the advantage of being more resistant to outliers.

 

6. When compared to generative models, discriminative models are more resistant to outliers.

 

Also Read | Introduction to K-nearest Neighbor (KNN) Algorithm

 

Conclusion

 

There are benefits and drawbacks to both generative and discriminative models. If not, we wouldn't be discussing both.

 

Generative models, for instance, are more data hungry. To accurately reflect the distributions, they require sufficient data. Additionally, computationally speaking, generative models are more expensive than discriminative models.

 

While outliers may have a significant impact on the data distribution and reduce the accuracy of generative models, discriminative models are more resistant to them. Machine learning makes good use of both generative and discriminative models. The task and data characteristics determine the best option.