Skip to main content

Supervised Model Evaluation

Premium

Evaluating machine learning models is crucial for understanding their performance and effectiveness. Different tasks, such as classification and regression, require specific metrics to assess how well a model performs. In classification, we often look at metrics like accuracy, precision, and recall, while in regression, metrics such as Mean Squared Error (MSE) and R-squared are commonly used. This lesson will delve into these evaluation metrics, helping you understand their applications and implications.

Overview

TaskMetricDescriptionCaveats/Notes
ClassificationAccuracyRatio of correctly predicted instances to total instances.Not suitable for imbalanced data.
PrecisionRatio of true positive predictions to total predicted positives.Useful in cases where false positives are costly.
RecallRatio of true positive predictions to total actual positives.Important in scenarios with high false negatives.
F1 ScoreHarmonic mean of precision and recall.Balances precision and recall.
Macro F1F1 score averaged across classes.Treats all classes equally.
Micro F1F1 score calculated globally.More influenced by larger classes.
AUC-ROCArea under the ROC curve, which plots true positive rate vs. false positive rate.Good for comparing models.
TaskMetricDescriptionCaveats/Notes
RegressionMSEMean of the squared differences between predicted and actual values.Sensitive to outliers.
MAEMean of the absolute differences between predicted and actual values.More robust to outliers.
R-squaredProportion of variance in the dependent variable explained by the model.Values range from 0 to 1.

Classification metrics

The confusion matrix is a powerful tool for visualizing the performance of a classification model. It tabulates the counts of true positive (TP), true negative (TN), false positive (FP), and false negative (FN) predictions, providing a detailed breakdown of the model's performance. From the confusion matrix, various classification metrics can be computed.

Accuracy

Accuracy measures the proportion of correctly classified instances out of total instances. It is calculated as the ratio of the sum of true positive and true negative predictions to the total number of predictions. Mathematically, accuracy is expressed as: Accuracy=TP+TNTP+TN+FP+FN\text{Accuracy} = \frac{\text{TP} + \text{TN}}{\text{TP} + \text{TN} + \text{FP} + \text{FN}} . Note that we should not use this metric with imbalanced data

Precision

Precision measures the proportion of true positive predictions out of all positive predictions made by the model. It quantifies the model's ability to avoid false positive predictions and is calculated as the ratio of true positives to the sum of true positives and false positives. Mathematically, precision is calculated as: Precision=TPTP+FP\text{Precision} = \frac{\text{TP}}{\text{TP} + \text{FP}}

Recall

Recall, also known as sensitivity, measures the proportion of true positive predictions out of all actual positive instances in the dataset. It quantifies the model's ability to capture all positive instances and is calculated as the ratio of true positives to the sum of true positives and false negatives. Mathematically, recall is calculated as: Recall=TPTP+FN\text{Recall} = \frac{\text{TP}}{\text{TP} + \text{FN}}

F1 score

The F1 score is the harmonic mean of precision and recall, providing a balanced measure of a classifier's performance. It combines both precision and recall into a single metric, offering a comprehensive assessment of the model's effectiveness in both minimizing false positives and capturing true positives. Mathematically, the F1 score is calculated as: F1=2×Precision×RecallPrecision+RecallF1 = 2 \times \frac{\text{Precision} \times \text{Recall}}{\text{Precision} + \text{Recall}}

AUC-ROC

The area under the receiver operating characteristic curve (AUC-ROC) is a widely used evaluation metric for binary classification models. The ROC curve plots the true positive rate (sensitivity) against the false positive rate (1 - specificity) at various classification thresholds. The AUC-ROC metric quantifies the overall discriminative power of the model across all possible threshold settings, with higher values indicating better performance. An AUC-ROC score of 1 signifies perfect discrimination, while a score of 0.5 suggests random guessing. AUC-ROC provides a comprehensive assessment of the model's ability to distinguish between positive and negative instances, making it a valuable tool for evaluating classification models in various domains.

Regression metrics

Evaluating regression models requires specific metrics to understand their accuracy and effectiveness. These metrics help in quantifying the difference between predicted and actual values, offering insights into the model's performance. The most commonly used regression metrics include Mean Squared Error (MSE), Mean Absolute Error (MAE), and R-squared (R²). Each of these metrics provides a different perspective on the model's predictive power and error characteristics.

Mean squared error (MSE)

Mean squared error (MSE) is a commonly used metric for evaluating the performance of regression models. It measures the average of the squares of the errors, where the error is the difference between the predicted value and the actual value. Mathematically, MSE is expressed as:

MSE=1m∑i=1m(ytrue(i)−ypred(i))2\text{MSE} = \frac{1}{m}\sum_{i=1}^m (y_{\text{true}}^{(i)} - y_{\text{pred}}^{(i)})^2 ymean=1m∑i=1mytrue(i)y_{\text{mean}} = \frac{1}{m}\sum_{i=1}^m y_{\text{true}}^{(i)} Total variance=1m∑i=1m(ymean−ytrue(i))2\text{Total variance} = \frac{1}{m}\sum_{i=1}^m (y_{\text{mean}} - y_{\text{true}}^{(i)})^2

where ytrue(i)y_{\text{true}}^{(i)} is the actual value, ypred(i)y_{\text{pred}}^{(i)} is the predicted value and nn is the number of data points.

MSE is sensitive to outliers because the errors are squared, which can significantly impact the overall metric.

Mean absolute error (MAE)

Mean absolute error (MAE) is another metric used to evaluate regression models. It measures the average magnitude of the errors in a set of predictions, without considering their direction. It is the mean of the absolute differences between the predicted values and actual values. Mathematically, MAE is calculated as:

MAE=1n∑i=1n∣yi−y^i∣\text{MAE} = \frac{1}{n} \sum_{i=1}^{n} |y_i - \hat{y}_i|

Unlike MSE, MAE is more robust to outliers, as it does not square the errors.

R-squared

A key metric for evaluating regression models e.g. linear regression on the supplied data is the R2R^2 metric, also known as coefficient of determination. R2R^2 metric is the ratio of explained variance (by the model) over the total variance in the data, and it is calculated as:

R2=1−MSETotal varianceR^2 = 1 - \frac{\text{MSE}}{\text{Total variance}}

The R2R^2 value ranges from 0 to 1, where a value closer to 1 indicates that a larger proportion of the variance in the dependent variable is predictable from the independent variables.

The diagram below presents the statistical output from a regression analysis. Data scientists are generally expected to have a comprehensive understanding of the various fields and metrics reported in such regression results. However, for machine learning engineers, the focus is typically on understanding a subset of these metrics, including the R-squared value (which indicates the model's goodness of fit), the coefficients (which represent the estimated relationship between each predictor variable and the outcome), the standard errors (which measure the uncertainty or variability in the coefficient estimates), the p-values (which assess the statistical significance of each coefficient), and the confidence intervals (which provide a range of plausible values for each coefficient).

Statistical output from regression analysis