December 5, 2020

# The Best Machine Learning Algorithm for Handwritten Digits Recognition | by Mahnoor Javed | Nov, 2020 ## Implementing Machine Learning Classification Algorithms to Recognize Handwritten Digits

Handwritten Digit Recognition is an interesting machine learning problem in which we have to identify the handwritten digits through various classification algorithms. There are a number of ways and algorithms to recognize handwritten digits, including Deep Learning/CNN, SVM, Gaussian Naive Bayes, KNN, Decision Trees, Random Forests, etc.

In this article, we will deploy a variety of machine learning algorithms from the Sklearn’s library on our dataset to classify the digits into their categories.

Let us first look at the dataset:

Let us import the dataset as digits:

`from sklearn.datasets import load_digitsdigits = load_digits()`

The DESCR provides a description of the dataset. The dataset contains images of hand-written digits: 10 classes where each class refers to a digit (0, 1, 2, 3, 4, 5, 6, 7, 8, 9).

Let us visualize the first image of the handwritten digits stored in images, and plot it using matplotlib.

`import matplotlib.pyplot as plt plt.gray() plt.matshow(digits.images) plt.show()`

The output will be as follows — it shows a handwritten ‘0’

And it can be verified by opening up the images NumPy array in the variable explorer, and resizing it so it can be easier to visualize:

Our dataset also consists of a NumPy array ‘target’ which stores the labels of all the images. The ‘target_names’ NumPy array stores the label values (the actual number which is the same in this case).

If our images stores the following 8×8 image (which is the number 5):

The target array will store its label in the 35th index place — target:

And the target_names array will be the names of the labels/classes, which in the case of digit recognition, are the digit themselves!

`from sklearn import metricsfrom sklearn.metrics import accuracy_scorefrom sklearn.model_selection import train_test_split`
`n_samples = len(digits.images)data = digits.images.reshape((n_samples, -1))`
`X_train, X_test, y_train, y_test = train_test_split(data, digits.target, test_size=0.5, shuffle=False)`

Our variable explorer will show the newly declared arrays:

The Classification Report will give us the precision, recall, f1-score, support, and accuracy, whereas the Confusion Matrix will show us the number of True Positives, False Positives, and False Negatives for each Classifier.

We will use the following classifiers from Sklearn:

1. Support Vector Machine
2. Gaussian Naive Bayes
3. Decision Trees
4. Random Forest
5. K Nearest Neighbours

## 1. Support Vector Machines (SVM)

`from sklearn import svmsvm_classifier = svm.SVC(gamma=0.001)svm_classifier.fit(X_train, y_train)predicted = svm_classifier.predict(X_test)_, axes = plt.subplots(2, 4)images_and_labels = list(zip(digits.images, digits.target))for ax, (image, label) in zip(axes[0, :], images_and_labels[:4]):ax.set_axis_off()ax.imshow(image, cmap=plt.cm.gray_r, interpolation='nearest')ax.set_title('Training: %i' % label)images_and_predictions = list(zip(digits.images[n_samples // 2:], predicted))for ax, (image, prediction) in zip(axes[1, :], images_and_predictions[:4]):ax.set_axis_off()ax.imshow(image, cmap=plt.cm.gray_r, interpolation='nearest')ax.set_title('Prediction: %i' % prediction)print("nClassification report for classifier %s:n%sn" % (svm_classifier, metrics.classification_report(y_test, predicted)))disp = metrics.plot_confusion_matrix(svm_classifier, X_test, y_test)disp.figure_.suptitle("Confusion Matrix")print("nConfusion matrix:n%s" % disp.confusion_matrix)print("nAccuracy of the Algorithm: ", svm_classifier.score(X_test, y_test))plt.show()`

The SVM Classifier gives an Accuracy of 0.9688!

The Classification Report it generates is as follows:

The Confusion Matrix is as follows:

## 2. Gaussian Naive Bayes

`from sklearn.naive_bayes import GaussianNBGNB_classifier = GaussianNB()GNB_classifier.fit(X_train, y_train)predicted = GNB_classifier.predict(X_test)_, axes = plt.subplots(2, 4)images_and_labels = list(zip(digits.images, digits.target))for ax, (image, label) in zip(axes[0, :], images_and_labels[:4]):ax.set_axis_off()ax.imshow(image, cmap=plt.cm.gray_r, interpolation='nearest')ax.set_title('Training: %i' % label)images_and_predictions = list(zip(digits.images[n_samples // 2:], predicted))for ax, (image, prediction) in zip(axes[1, :], images_and_predictions[:4]):ax.set_axis_off()ax.imshow(image, cmap=plt.cm.gray_r, interpolation='nearest')ax.set_title('Prediction: %i' % prediction)print("nClassification report for classifier %s:n%sn" % (GNB_classifier, metrics.classification_report(y_test, predicted)))disp = metrics.plot_confusion_matrix(GNB_classifier, X_test, y_test)disp.figure_.suptitle("Confusion Matrix")print("nConfusion matrix:n%s" % disp.confusion_matrix)print("nAccuracy of the Algorithm: ", GNB_classifier.score(X_test, y_test))plt.show()`

The Classification Report for the Naive Bayes Classifier and the Confusion Matrix are as follows:

The accuracy of the Gaussian Naive Bayes Classifier was found out to be 0.8075. Gaussian Naive Bayes Classifier is less accurate than the SVM Classifier (0.9688) when it comes to recognizing handwritten digits.

## 3. Decision Trees

`from sklearn import treedt_classifier = tree.DecisionTreeClassifier()dt_classifier.fit(X_train, y_train)predicted = dt_classifier.predict(X_test)_, axes = plt.subplots(2, 4)images_and_labels = list(zip(digits.images, digits.target))for ax, (image, label) in zip(axes[0, :], images_and_labels[:4]):ax.set_axis_off()ax.imshow(image, cmap=plt.cm.gray_r, interpolation='nearest')ax.set_title('Training: %i' % label)images_and_predictions = list(zip(digits.images[n_samples // 2:], predicted))for ax, (image, prediction) in zip(axes[1, :], images_and_predictions[:4]):ax.set_axis_off()ax.imshow(image, cmap=plt.cm.gray_r, interpolation='nearest')ax.set_title('Prediction: %i' % prediction)print("nClassification report for classifier %s:n%sn" % (dt_classifier, metrics.classification_report(y_test, predicted)))disp = metrics.plot_confusion_matrix(dt_classifier, X_test, y_test)disp.figure_.suptitle("Confusion Matrix")print("nConfusion matrix:n%s" % disp.confusion_matrix)print("nAccuracy of the Algorithm: ", dt_classifier.score(X_test, y_test))plt.show()`

The Decision Tree Classifier had an even lower accuracy score = 0.7352.

The Classification Report and Confusion Matrix of the Decision Tree Classifier are as follows:

## 4. Random Forest

`from sklearn.ensemble import RandomForestClassifierRF_classifier = RandomForestClassifier(max_depth=2, random_state=0)RF_classifier.fit(X_train, y_train)predicted = RF_classifier.predict(X_test)_, axes = plt.subplots(2, 4)images_and_labels = list(zip(digits.images, digits.target))for ax, (image, label) in zip(axes[0, :], images_and_labels[:4]):ax.set_axis_off()ax.imshow(image, cmap=plt.cm.gray_r, interpolation='nearest')ax.set_title('Training: %i' % label)images_and_predictions = list(zip(digits.images[n_samples // 2:], predicted))for ax, (image, prediction) in zip(axes[1, :], images_and_predictions[:4]):ax.set_axis_off()ax.imshow(image, cmap=plt.cm.gray_r, interpolation='nearest')ax.set_title('Prediction: %i' % prediction)print("nClassification report for classifier %s:n%sn" % (RF_classifier, metrics.classification_report(y_test, predicted)))disp = metrics.plot_confusion_matrix(RF_classifier, X_test, y_test)disp.figure_.suptitle("Confusion Matrix")print("nConfusion matrix:n%s" % disp.confusion_matrix)print("nAccuracy of the Algorithm: ", RF_classifier.score(X_test, y_test))plt.show()`

The accuracy score of the Random Forest Classifier is 0.7530 — nearly that of the Decision Tree Classifier.

The Classification Report and Confusion Matrix are as follows:

So far, the SVM classifier gives the best accuracy score. But as we know, it is not the only metric to evaluate how well the algorithm works.

Let us try a few more classifiers from sklearn.

## 5. K Nearest Neighbours (KNN)

`from sklearn.neighbors import KNeighborsClassifierKNN_classifier = KNeighborsClassifier(n_neighbors=5, metric='euclidean')KNN_classifier.fit(X_train, y_train)predicted = KNN_classifier.predict(X_test)_, axes = plt.subplots(2, 4)images_and_labels = list(zip(digits.images, digits.target))for ax, (image, label) in zip(axes[0, :], images_and_labels[:4]):ax.set_axis_off()ax.imshow(image, cmap=plt.cm.gray_r, interpolation='nearest')ax.set_title('Training: %i' % label)images_and_predictions = list(zip(digits.images[n_samples // 2:], predicted))for ax, (image, prediction) in zip(axes[1, :], images_and_predictions[:4]):ax.set_axis_off()ax.imshow(image, cmap=plt.cm.gray_r, interpolation='nearest')ax.set_title('Prediction: %i' % prediction)print("nClassification report for classifier %s:n%sn" % (KNN_classifier, metrics.classification_report(y_test, predicted)))disp = metrics.plot_confusion_matrix(KNN_classifier, X_test, y_test)disp.figure_.suptitle("Confusion Matrix")print("nConfusion matrix:n%s" % disp.confusion_matrix)print("nAccuracy of the Algorithm: ", KNN_classifier.score(X_test, y_test))plt.show()`

So the accuracy comes out to be 0.9555! Good thing we didn’t stop at Random Forest!

The Classification Report and Confusion Matrix for a KNN based Classifier is as follows:

`from sklearn.linear_model import SGDClassifiersgd_classifier = SGDClassifier(loss="hinge", penalty="l2", max_iter=5)sgd_classifier.fit(X_train, y_train)predicted = sgd_classifier.predict(X_test)_, axes = plt.subplots(2, 4)images_and_labels = list(zip(digits.images, digits.target))for ax, (image, label) in zip(axes[0, :], images_and_labels[:4]):ax.set_axis_off()ax.imshow(image, cmap=plt.cm.gray_r, interpolation='nearest')ax.set_title('Training: %i' % label)images_and_predictions = list(zip(digits.images[n_samples // 2:], predicted))for ax, (image, prediction) in zip(axes[1, :], images_and_predictions[:4]):ax.set_axis_off()ax.imshow(image, cmap=plt.cm.gray_r, interpolation='nearest')ax.set_title('Prediction: %i' % prediction)print("nClassification report for classifier %s:n%sn" % (sgd_classifier, metrics.classification_report(y_test, predicted)))disp = metrics.plot_confusion_matrix(sgd_classifier, X_test, y_test)disp.figure_.suptitle("Confusion Matrix")print("nConfusion matrix:n%s" % disp.confusion_matrix)print("nAccuracy of the Algorithm: ", sgd_classifier.score(X_test, y_test))plt.show()`

The Stochastic Gradient Descent Algorithm gives us an accuracy score of 0.8932.

Below are the Classification Report and the Confusion Matrix:

In terms of accuracy score, the SVM classifier was the most accurate, whereas Decision Trees were the least!

Now let us come to the F1- score. Again, the SVM has the highest range of F1-Score:

Hence, we conclude that both in terms of accuracy score and F1-score, the SVM classifier performed the best. That is why you will often see it used in image recognition problems as well!

Here is an article I wrote in which I used SVM (along with PCA) to build a facial recognition model.

Let me know what other algorithms you could have used for your classifier! I also tried Ada Boost Classifier but it gave me such a low accuracy (0.2558) that I didn’t bother including in the article! 😅

#### You may have missed 5 min read

#### How to Choose the Best Nearest Neighbors Algorithm | by Marie Stephen Leo | Towards AI | Dec, 2020 5 min read

#### How Transfer Learning Can Make Machine Learning More Efficient – The New Stack 13 min read

#### What Is Big Data and How Does It Work? | by Malika Harkati | Cloudit-eg | Dec, 2020 5 min read