Non-parametric Machine Learning Models

We learned about parameteeric machine learning algorithms in previous post and in this post, we're going to learn about their counterpart, Nnon-parameteric machine learning algorithms

Non-parameteric Models

Unlike parametric techniques, do not make any strong assumption about the form of the mapping function $\hat{f}$. Most of the advnaced machine learning algorithms fall into this category. These algorithms make a few or no assumptions abuot the relationship between the features and outcome.

One example of the non-parameteric machine learning algorithms are decision trees. Let's think of a binary classification task where we have $n$ labeled data points with $m$ features. Decision tree is a binary tree where the root node contains all $n$ data. As we go down the tree, the tree nodes split into two branches according to a rule that is figured out by the algorithm. To find the rule, at each split (node) and for every feature $x_i$, the model tries to find a value $\beta_i$ that best divides the entire data into classes 1 and 2. Best division in a sense that the data points with $x_i\leq\beta_i$ fall into one class and the data points with $x_i>\beta_i$ fall into the other class with the least number of misclassified data points. Once all the features are tested at one node, the feature with the least number of misclassified classes sets the rule (if $x_i>\beta_i$ the data point belongs to class I, otherwise class II) on that node.

The algorithm did not make any prior assumption about the data or the rules (model parameters). It simply divided the observations according to the patterns that demonstrated similar output variables (Classes 1 and 2).

Other machine learning algorithms that use non-parameteric algorithms include neural networks, support vector machines, k-nearest neighbors, and naive Bayes.