Cross-Validation Evaluating the same model on different subsets of data In this article, I’ll discuss parameter tuning, which involves selecting the optimal parameter. Typically, we start by splitting our entire dataset into three parts: training, validation, and testing. We utilize the validation dataset to determine the best parameter for the formula g(xi), essentially finding theContinue reading “ML Zoomcamp 2023 – Evaluation metrics for classification– Part 7”
Category Archives: Evaluation Metrics
ML Zoomcamp 2023 – Evaluation metrics for classification– Part 6
ROC AUC – Area under the ROC curve Useful metric One way to quantify how close we are to the ideal point is by measuring the area under the ROC curve (AUC). AUC equals 0.5 for a random baseline and 1.0 for an ideal curve. Therefore, our model’s AUC should fall between 0.5 and 1.0.Continue reading “ML Zoomcamp 2023 – Evaluation metrics for classification– Part 6”
ML Zoomcamp 2023 – Evaluation metrics for classification– Part 5
ROC Curve (Receiver Operating Characteristics) ROC (Receiver Operating Characteristic) curves are a valuable tool for evaluating binary classification models, especially in scenarios where you want to assess the trade-off between false positives and true positives at different decision thresholds. The ROC curve visually represents the performance of a model by plotting the TPR (True PositiveContinue reading “ML Zoomcamp 2023 – Evaluation metrics for classification– Part 5”
ML Zoomcamp 2023 – Evaluation metrics for classification– Part 4
Precision & Recall Precision and Recall are essential metrics for evaluating binary classification models. Precision measures the fraction of positive predictions that were correct. In other words, it quantifies how accurately the model predicts customers who are likely to churn. Precision = True Positives / (# Positive Predictions) = True Positives / (True Positives +Continue reading “ML Zoomcamp 2023 – Evaluation metrics for classification– Part 4”
ML Zoomcamp 2023 – Evaluation metrics for classification– Part 3
Confusion table / matrix Different types of errors and correct decisions In this section, we’ll discuss the confusion matrix, a vital tool for evaluating the performance of binary classification models. The confusion matrix allows us to examine the various errors and correct decisions made by our model. As we’ve previously discussed, class imbalance can significantlyContinue reading “ML Zoomcamp 2023 – Evaluation metrics for classification– Part 3”
ML Zoomcamp 2023 – Evaluation metrics for classification– Part 2
Accuracy and Dummy Model In the last article, we calculated that our model achieved an accuracy of 80% on the validation data. Now, let’s determine whether this is a good value or not. Accuracy measures the fraction of correct predictions made by the model. In our evaluation, we checked each customer in the validation datasetContinue reading “ML Zoomcamp 2023 – Evaluation metrics for classification– Part 2”
ML Zoomcamp 2023 – Evaluation metrics for classification– Part 1
Overview Today’s post recaps all the important lines of code that are crucial for the rest of this chapter. This includes the necessary imports, data preparation, data splitting for training, validation, and testing, separating the target variable ‘churn’, training the logistic regression model, and finally, validating the model on the validation data and outputting theContinue reading “ML Zoomcamp 2023 – Evaluation metrics for classification– Part 1”
knowMLedge.com 