A Cost function in Machine Learning

We are halfway into 2021 and a lot of our lifestyle changes have become an ode to the ongoing pandemic but one thing that has helped us overcome it is the technology that has equipped us with modern solutions.

 

During this period, Machine learning has come to the foreground and has changed the perception of interaction between humans and machines. Discussing the impact of human interaction with machines, enhance your understanding of Human-Computer  Interaction.

 

With machine learning, features associated with it also have flourished. The driving force behind optimization in machine learning is the response from an internal function of the algorithm, called the cost function.

 

 

Cost Function

 

It's a function that determines how well a Machine Learning model performs for a given set of data. The Cost Function calculates the difference between anticipated and expected values and shows it as a single real number. Cost Functions may be created in a variety of methods depending on the situation.

 

To estimate how poorly models perform, cost functions are employed. Simply put, a cost function is a measure of how inaccurate the model is in estimating the connection between X and y. This is usually stated as a difference or separation between the expected and actual values.

 

The term ‘loss' in machine learning refers to the difference between the anticipated and actual value. The "Loss Function" is a function that is used to quantify this loss in the form of a single real number during the training phase. These are utilised in algorithms that apply optimization approaches in supervised learning.

 

Regression, logistic regression, and other algorithms are instances of this type. The phrases "cost function" and "loss function" are interchangeable. The purpose of Cost Function is to be either:

 

• Minimum- When a value is reduced to its simplest form, it is referred to as a cost, loss, or mistake. The aim is to identify the model parameter settings for which the Cost Function gives the smallest possible number.

• Maximum- When something is maximised, the value it produces is referred to as a reward. The aim is to discover model parameter values with as large a returned number as feasible.

 

Example

 

Take the following scenario: you're trying to solve a classification issue, that is, you're trying to sort data into categories. Assume the data is on the weight and height of two different types of fish, which are represented in the scatter plot below by red and blue dots. 

To categorise fishes into these two groups, you'll need to utilise these two attribute values. A scatter plot depicts the distribution of the two different species of fish. Various solutions to this categorization issue are also presented below, in addition to the scatter plot:

---

Scatter plots, Source: www.kaggle.com

---

Although all three classifiers have a high degree of accuracy, the third solution is the best since it does not misclassify any points and splits the data into two equal halves. If you look closely, you'll notice that the top right image's line misclassified one red point, whereas the bottom left image's line misclassified one blue dot. 

 

The line in the graph's bottom right corner properly identifies all of the points. The line is almost precisely in between the two groups, and not closer to any of them, which is why it classifies all the points properly. This is where the cost function notion comes into play. The cost function assists us in finding the best option.

 

(Related reading: What is Regression Analysis?)

 

The cost function changes depending on the algorithm. The cost function for fitting a straight line is the total of squared errors, although it varies from method to algorithm. We differentiate the sum of squared errors with respect to the parameters 'm' and 'c' to minimise the sum of squared errors and discover the optimal 'm' and 'c'. The values of 'm' and 'c' are then obtained by solving the linear equations. The cost function will almost always have to be minimised.

 

It's quite easy to minimise and maximise a function: (a) Calculate the difference between the function and the parameter and equal to 0,

and (b) Differentiate the function w.r.t the parameter and equate to 0.

• For minimization – the function value of the double differential should be greater than 0.

• For maximization - the function value of the double differential should be less than 0.\

 

(Top read: Machine Learning Algorithms)

 

Applying the Cost Function

 

The Cost Function has many different formulations, but for this example, we wanna use the Cost Function for Linear Regression with a single variable.

 

Where:

• m: Is the number of our training examples.

• Σ: The Summatory.

• i: The number of Examples and the Output.

• h: The Hypothesis of our Linear Regression Model

 

The Cost Function will return a value that matches to our Model error once it has been calculated. The Cost Function must be minimised on a constant basis. When we reduce the Cost Function, we reduce the error and, as a result, our Model's performance improves.

 

But how can we make the Cost Function as little as possible?

 

The Gradient Descent Algorithm is the most common method for minimising the Cost Function.

 

(Also read: Introduction to XGBoost Algorithm)

 

Gradient Descent

 

Gradient Descent is an algorithm for optimising the cost function or the model's error. It's used to identify the smallest amount of inaccuracy in your model.

 

Gradient Descent may be thought of as the path you must travel to make the least amount of mistakes. The inaccuracy in your model might vary at different places, and you must discover the quickest approach to decrease it in order to avoid wasting resources.

 

Gradient Descent is analogous to a ball rolling down a slope. The ball will now roll to the bottom of the hill. It may use this position as the point where the error is the smallest because, in every model, the error will be the smallest at one point before increasing again.

 

Gradient descent is a method for determining the inaccuracy in your model for various input variable values. This is repeated, and you'll see that the error numbers become less and fewer with time. You'll soon arrive at the values for variables with the least amount of error, and the cost function will be optimised.

---

Gradient Descent curve, Source: www.simplilearn.com

---

(Suggested blog: What is LightGBM Algorithm?)

 

 

Types of Cost function

 

There are many cost functions in machine learning and each has its use cases depending on whether it is a regression problem or classification problem.

---

Types of Cost Functions

---

1. Regression cost Function

 

Regression models are used to forecast a continuous variable, such as an employee's pay, the cost of a car, the likelihood of obtaining a loan, and so on. The "Regression Cost Function" is a cost function utilised in the regression issue. They are determined as follows depending on the distance-based error:

 

Error = y-y’

Where, Y – Actual Input and Y’ – Predicted output

 

For obvious reasons, this cost function is also known as the squared error function. Because it is simple and works well, it is the most often used cost function for linear regression.

 

 

2. Binary Classification cost Functions

 

The cost functions used in classification problems are not the same as the cost functions used in regression problems. Cross-entropy loss is a popular classification loss function.

 

Under the maximum likelihood inference paradigm, it is the preferable loss function mathematically. It is the loss function that should be assessed first, and only altered if there is a compelling reason to do so. For predicting class 1, cross-entropy will compute a score that represents the average difference between the actual and predicted probability distributions. A perfect cross-entropy value is 0 when the score is minimised.

 

By minimising the overlap between distributions of the soft output for each class, this cost function seeks to reduce the likelihood of classification mistakes. To estimate the distributions from the training data set, the non-parametric Parzen window technique with Gaussian kernels is employed. 

 

The cost function was implemented in a GRBF neural network and evaluated in a motion detection application using low-resolution infrared pictures, demonstrating certain improvements over the traditional mean squared error cost function as well as the support vector machine, a reference binary classifier.

 

 

3. Multi-class Classification cost Functions

 

Predictive modelling issues involving multi-class categorization are ones in which instances are allocated to one of more than two classes.

 

The issue is frequently presented as predicting an integer value, with each class given a different integer value ranging from 0 to (num classes – 1). Predicting the likelihood of an example belonging to each known class is a common way to solve the problem.

 

Categorical cross-entropy, which is nothing more than the mean of cross-entropy for all N training data, is used to calculate the error in classification for the whole model. For multi-class classification problems, the default loss function is cross-entropy.

 

It is designed for use with multi-class classification in this scenario, when the target values are in the range 0 to 1, 3,..., n, and each class is given a distinct integer value.

 

Under the maximum likelihood inference paradigm, it is the preferable loss function mathematically. It is the loss function that should be assessed first, and only altered if there is a compelling reason to do so.

 

For all classes in the problem, cross-entropy will produce a score that summarises the average difference between the actual and anticipated probability distributions. A perfect cross-entropy value is 0 when the score is minimised.

 

As a concluding note, I can say, Cost function acts as a monitoring tool over the different algorithms and models as it points out the differences between the predicted outputs and the actual outputs and helps in improving the model.