Nand Kishor Contributor

Nand Kishor is the Product Manager of House of Bots. After finishing his studies in computer science, he ideated & re-launched Real Estate Business Intelligence Tool, where he created one of the leading Business Intelligence Tool for property price analysis in 2012. He also writes, research and sharing knowledge about Artificial Intelligence (AI), Machine Learning (ML), Data Science, Big Data, Python Language etc... ...

Full Bio 
Follow on

Nand Kishor is the Product Manager of House of Bots. After finishing his studies in computer science, he ideated & re-launched Real Estate Business Intelligence Tool, where he created one of the leading Business Intelligence Tool for property price analysis in 2012. He also writes, research and sharing knowledge about Artificial Intelligence (AI), Machine Learning (ML), Data Science, Big Data, Python Language etc...

3 Best Programming Languages For Internet of Things Development In 2018
871 days ago

Data science is the big draw in business schools
1044 days ago

7 Effective Methods for Fitting a Liner
1054 days ago

3 Thoughts on Why Deep Learning Works So Well
1054 days ago

3 million at risk from the rise of robots
1054 days ago

Top 10 Hot Artificial Intelligence (AI) Technologies

2018 Data Science Interview Questions for Top Tech Companies

Want to be a millionaire before you turn 25? Study artificial intelligence or machine learning

Here's why so many data scientists are leaving their jobs

Google announces scholarship program to train 1.3 lakh Indian developers in emerging technologies

Machine Learning Crash Course: Part 2 Tutorial

By Nand Kishor |Email | May 25, 2017 | 11700 Views

Perceptrons, Logistic Regression, and SVMs

In this post we'll talk about one of the most fundamental machine learning algorithms: the perceptron algorithm. This algorithm forms the basis for many modern day ML algorithms, most notably neural networks. In addition, we'll discuss the perceptron algorithm's cousin, logistic regression. And then we'll conclude with an introduction to SVMs, or support vector machines, which are perhaps one of the most flexible algorithms used today.

Supervised and Unsupervised Algorithms

In machine learning, there are two general classes of algorithms. You'll remember that in our last post we discussed regression and classification. These two methods fall under the larger umbrella of supervised learning algorithms, one of two classes of machine learning algorithms. The other class of algorithms is called unsupervised algorithms.

Supervised algorithms learn from labeled training data. The algorithms are "supervised" because we know what the correct answer is. If the algorithm receives a bunch of images labeled as apples or oranges it can first guess the object in the image, then use the label to check if its guess is correct.

Unsupervised learning is a bit different in that it finds patterns in data. It works similarly to the way we humans observe patterns (or objects) in random phenomena. Unsupervised learning algorithms do the same thing by looking at unlabeled data. Just like we don't have a particular goal when looking at an object (other than identifying it), the algorithm doesn't have a particular goal other than inferring patterns from the data itself.

We'll talk about unsupervised algorithms in a later blog post. For now, let's look at a very simple supervised algorithm, called the perceptron algorithm.



Model of a perceptron

One of the goals of machine learning and AI is to do what humans do and even surpass them. Thus, it makes sense to try and copy what makes humans so great at what they do-their brains.

The brain is composed of billions of interconnected neurons that are constantly firing, passing signals from one neuron to another. Together, they allow us to do incredible things such as recognize faces, patterns, and most importantly, think.

The job of an individual neuron is simple: if its inputs match certain criteria, it fires, sending a signal to other neurons connected to it. It's all very black and white. Of course, the actual explanation is much more complicated than this, but since we're using computers to simulate a brain, we only need to copy the idea of how a brain works.

The perceptron mimics the function of neurons in machine learning algorithms. The perceptron is one of the oldest machine learning algorithms in existence. When it was first used in 1957 to perform rudimentary image recognition, the New York Times called it:

the embryo of an electronic computer that [the Navy] expects will be able to walk, talk, see, write, reproduce itself and be conscious of its existence.

We're still a little far off from that, but the Times did recognize the potential of a perceptron. Perceptrons now form the basis for more complicated neural networks, which we'll be talking about in the next post.

Like neurons, perceptrons take in several inputs and spit out an output. For example, a perceptron might take as input temperature and try to answer the question, "Should I wear a sweater today?" If the input temperature is below a certain threshold (say, 70?F), the perceptron will output 1 (yes). If the temperature is above the threshold, the perceptron will output a 0 (no).

Of course, we could consider more variables than just temperature when deciding whether or not to wear a sweater. Like how a biological neuron typically can have more than one input electrical impulse, we can make our perceptrons have multiple inputs. In that case, we'll have to also weight each input by a certain amount. If we were to use temperature, wind speed, and something completely random like the number of people showering in Hong Kong as our inputs, we would want to use different weights for each input. Temperature would probably have a negative weight, because the lower the temperature the more you should probably wear a sweater. Wind speed should have a positive weight, because the higher the wind speeds the more likely you will need to put on a sweater. And as for the number of people showering in Hong Kong, the weight should probably be zero (unless you normally factor in the number of people taking showers in Hong Kong into your decision making).

But someone from Canada might be very used to the cold, so their threshold for wearing a sweater might be a lower temperature than someone from, say, Australia. To express this, we use a bias to specify the threshold for the Canadian and the Australian. You can think of the bias as a measure of how difficult it is for the perceptron to say ??yes' or ??no'.

The Australian is biased towards hotter temperatures and thus has a higher threshold temperature

Logistic Regression

However, life is not as black and white as perceptrons indicate. There is uncertainty in just about anything, even choosing whether or not to put on a sweater. In reality, we don't immediately put on a sweater the moment it drops below some predefined temperature. It's more as if at any temperature we have a certain "chance" of putting on a sweater. Maybe at 45 F somebody will have a 95% chance of putting on a sweater, and at 60 F the same person will have a 30% chance of putting on a sweater.

To better model life's complexities, we use logistic regression to find these probabilities. This involves fitting a logistic curve (like the one below) to our data. To do this, we again use gradient descent to choose the best parameters for the model. That is, we find parameters that minimize some cost function.

Logistic regression and the perceptron algorithm are very similar to each other. It's common to think of logistic regression as a kind of perceptron algorithm on steroids, in that a logistic model can predict probabilities while a perceptron can only predict yes or no. In fact, taking a logistic model and setting all values less than .5 to zero, and all values above .5 to one gives a very similar result to just the perceptron algorithm.

Support Vector Machines

Support vector machines, or SVMs for short, are a class of machine learning algorithms that have become incredibly popular in the past few years. They are based on a very intuitive idea. Here, we'll introduce SVMs and go through the key ideas in the algorithm.


If you remember the section on classification from our last post, we classify data by drawing a line, called a decision boundary, to separate them. Read More

Source: Berkeley