December 5, 2020

8 Links

8 Links of Separation

The Best Machine Learning Algorithm for Handwritten Digits Recognition | by Mahnoor Javed | Nov, 2020

Implementing Machine Learning Classification Algorithms to Recognize Handwritten Digits

(Image from Pixabay)

Handwritten Digit Recognition is an interesting machine learning problem in which we have to identify the handwritten digits through various classification algorithms. There are a number of ways and algorithms to recognize handwritten digits, including Deep Learning/CNN, SVM, Gaussian Naive Bayes, KNN, Decision Trees, Random Forests, etc.

In this article, we will deploy a variety of machine learning algorithms from the Sklearn’s library on our dataset to classify the digits into their categories.

Let us first look at the dataset:

Let us import the dataset as digits:

from sklearn.datasets import load_digits
digits = load_digits()
Data Downloaded as digits (Image by Author)

The DESCR provides a description of the dataset. The dataset contains images of hand-written digits: 10 classes where each class refers to a digit (0, 1, 2, 3, 4, 5, 6, 7, 8, 9).

Let us visualize the first image of the handwritten digits stored in images, and plot it using matplotlib.

import matplotlib.pyplot as plt 
plt.gray()
plt.matshow(digits.images[0])
plt.show()

The output will be as follows — it shows a handwritten ‘0’

The Output of images[0] (Image by Author)

And it can be verified by opening up the images NumPy array in the variable explorer, and resizing it so it can be easier to visualize:

images[0] (Image by Author)

Our dataset also consists of a NumPy array ‘target’ which stores the labels of all the images. The ‘target_names’ NumPy array stores the label values (the actual number which is the same in this case).

If our images[35] stores the following 8×8 image (which is the number 5):

images[35] (Image by Author)

The target array will store its label in the 35th index place — target[35]:

target[35] (Image by Author)

And the target_names array will be the names of the labels/classes, which in the case of digit recognition, are the digit themselves!

target_names (Image by Author)
from sklearn import metrics
from sklearn.metrics import accuracy_score
from sklearn.model_selection import train_test_split
n_samples = len(digits.images)
data = digits.images.reshape((n_samples, -1))
X_train, X_test, y_train, y_test = train_test_split(
data, digits.target, test_size=0.5, shuffle=False)

Our variable explorer will show the newly declared arrays:

X_test, X_train, y_test & y_train (Image by Author)

The Classification Report will give us the precision, recall, f1-score, support, and accuracy, whereas the Confusion Matrix will show us the number of True Positives, False Positives, and False Negatives for each Classifier.

We will use the following classifiers from Sklearn:

  1. Support Vector Machine
  2. Gaussian Naive Bayes
  3. Decision Trees
  4. Random Forest
  5. K Nearest Neighbours
  6. Stochastic Gradient Descent

1. Support Vector Machines (SVM)

from sklearn import svm
svm_classifier = svm.SVC(gamma=0.001)
svm_classifier.fit(X_train, y_train)
predicted = svm_classifier.predict(X_test)_, axes = plt.subplots(2, 4)
images_and_labels = list(zip(digits.images, digits.target))
for ax, (image, label) in zip(axes[0, :], images_and_labels[:4]):
ax.set_axis_off()
ax.imshow(image, cmap=plt.cm.gray_r, interpolation='nearest')
ax.set_title('Training: %i' % label)

images_and_predictions = list(zip(digits.images[n_samples // 2:], predicted))
for ax, (image, prediction) in zip(axes[1, :], images_and_predictions[:4]):
ax.set_axis_off()
ax.imshow(image, cmap=plt.cm.gray_r, interpolation='nearest')
ax.set_title('Prediction: %i' % prediction)

print("nClassification report for classifier %s:n%sn" % (svm_classifier, metrics.classification_report(y_test, predicted)))
disp = metrics.plot_confusion_matrix(svm_classifier, X_test, y_test)
disp.figure_.suptitle("Confusion Matrix")
print("nConfusion matrix:n%s" % disp.confusion_matrix)
print("nAccuracy of the Algorithm: ", svm_classifier.score(X_test, y_test))
plt.show()

The SVM Classifier gives an Accuracy of 0.9688!

The Classification Report it generates is as follows:

Classification Report for SVM Classifier (Image by Author)

The Confusion Matrix is as follows:

Confusion Matrix for SVM Classifier (Image by Author)

2. Gaussian Naive Bayes

from sklearn.naive_bayes import GaussianNB
GNB_classifier = GaussianNB()
GNB_classifier.fit(X_train, y_train)
predicted = GNB_classifier.predict(X_test)_, axes = plt.subplots(2, 4)
images_and_labels = list(zip(digits.images, digits.target))
for ax, (image, label) in zip(axes[0, :], images_and_labels[:4]):
ax.set_axis_off()
ax.imshow(image, cmap=plt.cm.gray_r, interpolation='nearest')
ax.set_title('Training: %i' % label)

images_and_predictions = list(zip(digits.images[n_samples // 2:], predicted))
for ax, (image, prediction) in zip(axes[1, :], images_and_predictions[:4]):
ax.set_axis_off()
ax.imshow(image, cmap=plt.cm.gray_r, interpolation='nearest')
ax.set_title('Prediction: %i' % prediction)

print("nClassification report for classifier %s:n%sn" % (GNB_classifier, metrics.classification_report(y_test, predicted)))
disp = metrics.plot_confusion_matrix(GNB_classifier, X_test, y_test)
disp.figure_.suptitle("Confusion Matrix")
print("nConfusion matrix:n%s" % disp.confusion_matrix)
print("nAccuracy of the Algorithm: ", GNB_classifier.score(X_test, y_test))
plt.show()

The Classification Report for the Naive Bayes Classifier and the Confusion Matrix are as follows:

Classification Report for Gaussian Naive Bayes Classifier (Image by Author)
Confusion Matrix for Gaussian Naive Bayes Classifier (Image by Author)

The accuracy of the Gaussian Naive Bayes Classifier was found out to be 0.8075. Gaussian Naive Bayes Classifier is less accurate than the SVM Classifier (0.9688) when it comes to recognizing handwritten digits.

3. Decision Trees

from sklearn import tree
dt_classifier = tree.DecisionTreeClassifier()
dt_classifier.fit(X_train, y_train)
predicted = dt_classifier.predict(X_test)_, axes = plt.subplots(2, 4)
images_and_labels = list(zip(digits.images, digits.target))
for ax, (image, label) in zip(axes[0, :], images_and_labels[:4]):
ax.set_axis_off()
ax.imshow(image, cmap=plt.cm.gray_r, interpolation='nearest')
ax.set_title('Training: %i' % label)

images_and_predictions = list(zip(digits.images[n_samples // 2:], predicted))
for ax, (image, prediction) in zip(axes[1, :], images_and_predictions[:4]):
ax.set_axis_off()
ax.imshow(image, cmap=plt.cm.gray_r, interpolation='nearest')
ax.set_title('Prediction: %i' % prediction)

print("nClassification report for classifier %s:n%sn" % (dt_classifier, metrics.classification_report(y_test, predicted)))
disp = metrics.plot_confusion_matrix(dt_classifier, X_test, y_test)
disp.figure_.suptitle("Confusion Matrix")
print("nConfusion matrix:n%s" % disp.confusion_matrix)
print("nAccuracy of the Algorithm: ", dt_classifier.score(X_test, y_test))
plt.show()

The Decision Tree Classifier had an even lower accuracy score = 0.7352.

The Classification Report and Confusion Matrix of the Decision Tree Classifier are as follows:

Classification Report for Decision Tree Classifier (Image by Author)
Confusion Matrix for Decision Tree Classifier (Image by Author)

4. Random Forest

from sklearn.ensemble import RandomForestClassifier
RF_classifier = RandomForestClassifier(max_depth=2, random_state=0)
RF_classifier.fit(X_train, y_train)
predicted = RF_classifier.predict(X_test)_, axes = plt.subplots(2, 4)
images_and_labels = list(zip(digits.images, digits.target))
for ax, (image, label) in zip(axes[0, :], images_and_labels[:4]):
ax.set_axis_off()
ax.imshow(image, cmap=plt.cm.gray_r, interpolation='nearest')
ax.set_title('Training: %i' % label)

images_and_predictions = list(zip(digits.images[n_samples // 2:], predicted))
for ax, (image, prediction) in zip(axes[1, :], images_and_predictions[:4]):
ax.set_axis_off()
ax.imshow(image, cmap=plt.cm.gray_r, interpolation='nearest')
ax.set_title('Prediction: %i' % prediction)

print("nClassification report for classifier %s:n%sn" % (RF_classifier, metrics.classification_report(y_test, predicted)))
disp = metrics.plot_confusion_matrix(RF_classifier, X_test, y_test)
disp.figure_.suptitle("Confusion Matrix")
print("nConfusion matrix:n%s" % disp.confusion_matrix)
print("nAccuracy of the Algorithm: ", RF_classifier.score(X_test, y_test))
plt.show()

The accuracy score of the Random Forest Classifier is 0.7530 — nearly that of the Decision Tree Classifier.

The Classification Report and Confusion Matrix are as follows:

Classification Report for Random Forest Classifier (Image by Author)
Confusion Matrix for Random Forest Classifier (Image by Author)

So far, the SVM classifier gives the best accuracy score. But as we know, it is not the only metric to evaluate how well the algorithm works.

Let us try a few more classifiers from sklearn.

5. K Nearest Neighbours (KNN)

from sklearn.neighbors import KNeighborsClassifier
KNN_classifier = KNeighborsClassifier(n_neighbors=5, metric='euclidean')
KNN_classifier.fit(X_train, y_train)
predicted = KNN_classifier.predict(X_test)_, axes = plt.subplots(2, 4)
images_and_labels = list(zip(digits.images, digits.target))
for ax, (image, label) in zip(axes[0, :], images_and_labels[:4]):
ax.set_axis_off()
ax.imshow(image, cmap=plt.cm.gray_r, interpolation='nearest')
ax.set_title('Training: %i' % label)

images_and_predictions = list(zip(digits.images[n_samples // 2:], predicted))
for ax, (image, prediction) in zip(axes[1, :], images_and_predictions[:4]):
ax.set_axis_off()
ax.imshow(image, cmap=plt.cm.gray_r, interpolation='nearest')
ax.set_title('Prediction: %i' % prediction)

print("nClassification report for classifier %s:n%sn" % (KNN_classifier, metrics.classification_report(y_test, predicted)))
disp = metrics.plot_confusion_matrix(KNN_classifier, X_test, y_test)
disp.figure_.suptitle("Confusion Matrix")
print("nConfusion matrix:n%s" % disp.confusion_matrix)
print("nAccuracy of the Algorithm: ", KNN_classifier.score(X_test, y_test))
plt.show()

So the accuracy comes out to be 0.9555! Good thing we didn’t stop at Random Forest!

The Classification Report and Confusion Matrix for a KNN based Classifier is as follows:

Classification Report for KNN Classifier (Image by Author)
Confusion Matrix for KNN Classifier (Image by Author)

6. Stochastic Gradient Descent

from sklearn.linear_model import SGDClassifier
sgd_classifier = SGDClassifier(loss="hinge", penalty="l2", max_iter=5)
sgd_classifier.fit(X_train, y_train)
predicted = sgd_classifier.predict(X_test)_, axes = plt.subplots(2, 4)
images_and_labels = list(zip(digits.images, digits.target))
for ax, (image, label) in zip(axes[0, :], images_and_labels[:4]):
ax.set_axis_off()
ax.imshow(image, cmap=plt.cm.gray_r, interpolation='nearest')
ax.set_title('Training: %i' % label)

images_and_predictions = list(zip(digits.images[n_samples // 2:], predicted))
for ax, (image, prediction) in zip(axes[1, :], images_and_predictions[:4]):
ax.set_axis_off()
ax.imshow(image, cmap=plt.cm.gray_r, interpolation='nearest')
ax.set_title('Prediction: %i' % prediction)

print("nClassification report for classifier %s:n%sn" % (sgd_classifier, metrics.classification_report(y_test, predicted)))
disp = metrics.plot_confusion_matrix(sgd_classifier, X_test, y_test)
disp.figure_.suptitle("Confusion Matrix")
print("nConfusion matrix:n%s" % disp.confusion_matrix)
print("nAccuracy of the Algorithm: ", sgd_classifier.score(X_test, y_test))
plt.show()

The Stochastic Gradient Descent Algorithm gives us an accuracy score of 0.8932.

Below are the Classification Report and the Confusion Matrix:

Classification Report for SGD Classifier (Image by Author)
Confusion Matrix for SGD Classifier (Image by Author)

In terms of accuracy score, the SVM classifier was the most accurate, whereas Decision Trees were the least!

Classifiers arranged in Descending Accuracy Score (Image by Author)

Now let us come to the F1- score. Again, the SVM has the highest range of F1-Score:

Classifiers arranged in Descending F1-Score (Image by Author)

Hence, we conclude that both in terms of accuracy score and F1-score, the SVM classifier performed the best. That is why you will often see it used in image recognition problems as well!

Here is an article I wrote in which I used SVM (along with PCA) to build a facial recognition model.

Let me know what other algorithms you could have used for your classifier! I also tried Ada Boost Classifier but it gave me such a low accuracy (0.2558) that I didn’t bother including in the article! 😅