Classification Basics, Logistic Regression, Softmax Regression, Multi-Class Classification, Regularization and Model Evaluation
Md. Atikuzzaman
Lecturer, Department of Computer Science & Engineering
atik@cse.green.edu.bd
Regression predicts a continuous number. Classification predicts a category.
Examples: spam/not spam, pass/fail, disease/no disease, grade A/B/C/D/F.
FeaturesScoreProbabilityClass
Spam or not spam.
Normal or abnormal case.
Fraud or legitimate transaction.
At-risk or not-at-risk student.
| # | Study hours | Attendance | Previous mark | Label |
|---|---|---|---|---|
| 1 | 1.5 | 48 | 42 | Fail |
| 2 | 2.0 | 55 | 46 | Fail |
| 3 | 3.5 | 64 | 58 | Pass |
| 4 | 4.0 | 70 | 61 | Pass |
| 5 | 5.5 | 82 | 70 | Pass |
| 6 | 2.8 | 60 | 50 | Fail |
Two classes.
y ∈ {0,1}
More than two exclusive classes.
y ∈ {1,...,K}
Multiple labels at the same time.
y ∈ {0,1}K
Suppose the input is only weekly study hours.
Output: a continuous value such as GPA.
For classification, we can still begin with the same linear score.
But now the target is not a continuous GPA. The target is a class:
The problem: the linear score \(z\) can be any real number, so it is not yet a probability.
Then it adds one extra step:
The sigmoid converts any value of \(z\) into a number between 0 and 1.
Main idea: the linear part is still there. Logistic regression adds sigmoid because classification needs a probability and a decision.
Logistic regression starts exactly like a linear model by computing z = wᵀx + b.
The sigmoid then turns that unrestricted score into a valid probability between 0 and 1.
The logit becomes a probability. The threshold converts probability into a final class.
It strongly penalizes confident wrong predictions.
Predict probabilities, calculate loss, update parameters, and repeat.
Softmax converts multiple class scores into a probability distribution.
For classes A, B, C, true class B becomes [0, 1, 0].
Train one binary classifier per class.
Simple, but probabilities may not sum to 1.
One model compares all classes together.
Common for mutually exclusive classes.
Regularization discourages overly large weights and helps reduce overfitting.
Gradient-based training works better when features have comparable scales.
Accuracy can be misleading when one class dominates.
Use precision, recall, F1, ROC-AUC, class weights or resampling.
Accuracy alone hides mistake types. False negatives and false positives have different real-world costs.
(TP+TN)/Total
TP/(TP+FP)
TP/(TP+FN)
2PR/(P+R)
ROC curve compares true positive rate and false positive rate across many thresholds.
import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score, confusion_matrix
X = np.array([[1.5,48], [2.0,55], [2.8,60], [3.5,64],
[4.0,70], [5.5,82], [6.0,85], [7.0,90]])
y = np.array([0, 0, 0, 1, 1, 1, 1, 1])
model = LogisticRegression()
model.fit(X, y)
y_pred = model.predict(X)
print(confusion_matrix(y, y_pred))
print("Accuracy:", accuracy_score(y, y_pred))
print("Precision:", precision_score(y, y_pred))
print("Recall:", recall_score(y, y_pred))
print("F1:", f1_score(y, y_pred))
print(model.predict_proba([[3, 66]]))
Misleading in imbalanced datasets.
0.5 may not match real risk.
Optimization can become unstable.
Test data used during training.
A model predicts whether a student is at risk of failing. False negatives mean missing students who need support.
Which metric matters most: accuracy, precision, recall or F1?
Should the threshold be lower than 0.5?
Classification predicts categories.
Logistic regression uses sigmoid for binary classification.
Softmax regression handles mutually exclusive multi-class classification.
Cross-entropy penalizes confident wrong predictions.
Evaluation should include confusion matrix, precision, recall, F1 and threshold analysis.
Next: Decision Trees and Ensemble Learning