Concepts, Gradient Descent, Ordinary Least Squares, Regularization and Model Evaluation
Md. Atikuzzaman
Lecturer, Department of Computer Science & Engineering
atik@cse.green.edu.bd
A teacher wants to predict a student’s final mark using weekly study hours.
We start with a few observed students, fit a line, then use that line to predict new students.
DataModelErrorLearningEvaluation
It introduces the full ML pipeline without too much mathematical overhead.
Each coefficient shows how the prediction changes when a feature changes.
Loss functions, optimization, overfitting and regularization all become visible.
n = number of training examples
d = number of input features
x = input/features
y = target/output
ŷ = model prediction
| # | Study hours x | Final mark y | Split |
|---|---|---|---|
| 1 | 1 | 42 | Training |
| 2 | 2 | 50 | Training |
| 3 | 3 | 55 | Training |
| 4 | 4 | 60 | Training |
| 5 | 5 | 70 | Training |
| 6 | 6 | 75 | Training |
| 7 | 7 | 82 | Test |
| 8 | 8 | 88 | Test |
Learn a function that maps study hours to final marks.
The last two examples are kept for testing the model after training.
Simple linear regression
\[ \hat{y}=wx+b \]Multiple linear regression
\[ \hat{y}=\theta_0+\theta_1x_1+\cdots+\theta_dx_d \] \[ \hat{y}=\boldsymbol{\theta}^{T}\tilde{\mathbf{x}},\quad \tilde{\mathbf{x}}=[1,x_1,\ldots,x_d]^T \]w or θj controls the effect of a feature.
b or θ0 is the intercept/bias term.
The model is linear in its parameters, not necessarily only a straight line in raw features.
The factor \(\frac{1}{2}\) makes derivatives cleaner.
A residual is the vertical distance between the actual data point and the model’s predicted line.
Training tries to make the residuals small on average.
The best line balances all points, not just one point.
OLS chooses the parameters that minimize the sum of squared residuals.
\[ \min_{\boldsymbol{\theta}} \; ||\mathbf{X}\boldsymbol{\theta}-\mathbf{y}||_2^2 \] \[ \boldsymbol{\theta}^{*}=(\mathbf{X}^{T}\mathbf{X})^{-1}\mathbf{X}^{T}\mathbf{y} \]The closed-form solution requires \(\mathbf{X}^{T}\mathbf{X}\) to be invertible.
If features are highly correlated or columns are redundant, use pseudo-inverse or regularization.
In practice, libraries solve this using numerically stable linear algebra rather than manually computing the inverse.
α is the learning rate.
The gradient points toward increasing loss, so we move in the opposite direction.
Start with a poor line, measure error, update the line, and repeat.
Gradient descent is especially useful when datasets are large or models are more complex.
For \(\hat{y}=wx+b\):
\[ \frac{\partial J}{\partial w}=\frac{1}{n}\sum_{i=1}^{n}(\hat{y}^{(i)}-y^{(i)})x^{(i)} \] \[ \frac{\partial J}{\partial b}=\frac{1}{n}\sum_{i=1}^{n}(\hat{y}^{(i)}-y^{(i)}) \]Update rules:
\[ w := w-\alpha\frac{\partial J}{\partial w},\quad b := b-\alpha\frac{\partial J}{\partial b} \]Repeat until the loss stops decreasing or a fixed number of epochs is reached.
Try a large learning rate. If the loss jumps, the step size is too aggressive.
Current fitted line during optimization
How the loss changes across epochs
A good learning rate makes the loss curve decrease smoothly. If the curve oscillates or grows, the step size is too large.
Uses the full dataset for each update.
Stable, but slow for very large data.
Uses one example for each update.
Fast but noisy.
Uses a small batch of examples.
Most common in modern ML training.
Standardization makes features comparable in scale.
Gradient descent may move slowly when one feature is in taka, another in square feet, and another in number of rooms.
Scaling can make training faster and more stable.
Features may include:
The model remains linear in the parameters \(\theta\).
This model can fit a curve, but it is still linear in \(\theta_0,\theta_1,\theta_2\).
Higher-degree polynomial features can fit training data very well but may generalize poorly.
This leads to overfitting.
The model is too simple and performs poorly on both training and test data.
High bias.
The model captures the dominant pattern while still generalizing to unseen data.
Good bias–variance tradeoff.
The model follows the noise in the training set and usually performs poorly on new data.
High variance.
General idea:
\[ \text{Training loss} + \lambda \times \text{Penalty} \]The penalty discourages overly large parameter values.
Regularization helps reduce overfitting and improve performance on new data.
λ controls the strength of the penalty.
Ridge shrinks coefficients toward zero but usually does not make them exactly zero.
Useful when features are correlated.
Lasso
\[ J_{lasso}=\frac{1}{n}||\mathbf{X}\boldsymbol{\theta}-\mathbf{y}||_2^2+\lambda||\boldsymbol{\theta}||_1 \]Elastic Net
\[ J=\text{MSE}+\lambda_1||\boldsymbol{\theta}||_1+\lambda_2||\boldsymbol{\theta}||_2^2 \]Lasso can set some coefficients exactly to zero, so it can perform feature selection.
Elastic Net combines L1 and L2 penalties.
MAE is easy to interpret.
MSE/RMSE punish large errors more strongly.
R² shows explained variance, but can be misleading alone.
Used to learn parameters.
Used to tune choices such as degree, λ and learning rate.
Used once at the end to estimate real performance.
Never tune the model repeatedly on the test set, otherwise the test result becomes optimistic.
import numpy as np
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score
X = np.array([[1], [2], [3], [4], [5], [6]])
y = np.array([42, 50, 55, 60, 70, 75])
model = LinearRegression()
model.fit(X, y)
X_test = np.array([[7], [8]])
y_test = np.array([82, 88])
y_pred = model.predict(X_test)
print("w:", model.coef_[0])
print("b:", model.intercept_)
print("MAE:", mean_absolute_error(y_test, y_pred))
print("RMSE:", mean_squared_error(y_test, y_pred) ** 0.5)
print("R2:", r2_score(y_test, y_pred))
Using test information during training or preprocessing.
Gradient descent becomes slow or unstable.
A high R² does not guarantee useful predictions.
Errors may reveal patterns the model missed.
Use the dataset table and fit a line manually by trying different values of \(w\) and \(b\).
Then compare your answer with OLS or gradient descent.
Which line looks good visually?
Which line has the lowest MSE?
Can a line fit every point perfectly?
Improvement may mean better features, scaling, regularization, more data or a different model.
Linear regression predicts a continuous value using a weighted sum of features.
OLS gives a closed-form solution when the matrix conditions are suitable.
Gradient descent learns parameters through repeated small updates.
Regularization controls model complexity and helps reduce overfitting.
Model evaluation requires test data, appropriate error metrics and residual analysis.
Next: Logistic Regression and Classification