Basic Neural Network
Example 1
Consider a simple neural network with one weight \( w \) and no bias. Let the input \( x \) and the actual output \( y \) be given. The prediction \( \hat{y} \) is:
The MSE loss function for a single data point is:
Step-by-Step Calculation
Prediction: Compute the predicted output \( \hat{y} \):
\[ \hat{y} = w \cdot x \]Loss Calculation: Compute the MSE loss:
\[ \text{MSE} = (y - w \cdot x)^2 \]Let’s take a numerical example to illustrate this process.
Numerical Example:
Suppose \( x = 2 \), \( y = 5 \), and the current weight \( w = 1 \).
Prediction:
\[ \hat{y} = 1 \cdot 2 = 2 \]Loss Calculation:
\[ \text{MSE} = (5 - 2)^2 = 9 \]Gradient Calculation: Compute the partial derivative of the loss with respect to the weight \( w \):
\[ \frac{\partial \text{MSE}}{\partial w} = \frac{\partial}{\partial w} (y - w \cdot x)^2 \]Using the chain rule:
\[ \frac{\partial \text{MSE}}{\partial w} = 2 (y - w \cdot x) \cdot (-x) \]Substitute the values from our example:
\[ \frac{\partial \text{MSE}}{\partial w} = 2 (5 - 1 \cdot 2) \cdot (-2) \]\[ \frac{\partial \text{MSE}}{\partial w} = 2 (5 - 2) \cdot (-2) \]\[ \frac{\partial \text{MSE}}{\partial w} = 2 \cdot 3 \cdot (-2) \]\[ \frac{\partial \text{MSE}}{\partial w} = -12 \]Update the Weight: Use the gradient to update the weight \( w \) using gradient descent:
Choose a learning rate \( \eta \), typically a small positive number.
Update rule: \( w_{\text{new}} = w_{\text{old}} - \eta \cdot \frac{\partial \text{MSE}}{\partial w} \)
For example:
Suppose the learning rate \( \eta = 0.1 \).
Weight Update:
\[ w_{\text{new}} = 1 - 0.1 \cdot (-12) \]\[ w_{\text{new}} = 1 + 1.2 \]\[ w_{\text{new}} = 2.2 \]
So, after one iteration of gradient descent with MSE loss, the updated weight \( w \) is \( 2.2 \).
Iterative Process
This process repeats for each training sample during training:
Compute \( \hat{y} \).
Calculate \( \text{MSE} \).
Compute \( \frac{\partial \text{MSE}}{\partial w} \).
Update \( w \).
Through multiple iterations (epochs), the network learns to minimize the MSE loss across all training samples, improving its ability to predict \( y \) from \( x \).
Conclusion
The gradient of the loss function (in this case, MSE) with respect to the weights of the network tells us how to adjust the weights to reduce the error between predicted and actual outputs during training. This iterative process is fundamental in training neural networks using gradient-based optimization algorithms like gradient descent.
import torch
import torch.nn as nn
import torch.optim as optim
# Define the input and output data (tensor form)
x_data = torch.tensor([2.0]) # Input
y_data = torch.tensor([5.0]) # Actual output
# Define the model: Linear regression model with one weight parameter
class LinearRegression(nn.Module):
def __init__(self):
super(LinearRegression, self).__init__()
self.w = nn.Parameter(torch.tensor([1.0])) # Weight parameter
def forward(self, x):
return self.w * x
# Instantiate the model
model = LinearRegression()
# Define the Mean Squared Error (MSE) loss function
criterion = nn.MSELoss()
# Define the optimizer (Gradient Descent optimizer with learning rate 0.1)
optimizer = optim.SGD(model.parameters(), lr=0.1)
# Training loop
epochs = 5 # Number of training epochs
for epoch in range(epochs):
# Forward pass
y_pred = model(x_data)
# Compute loss
loss = criterion(y_pred, y_data)
# Backward pass and optimize
optimizer.zero_grad() # Clear gradients
loss.backward() # Compute gradients
optimizer.step() # Update weights
# Print progress
print(f'Epoch [{epoch + 1}/{epochs}], Loss: {loss.item():.4f}, Updated weight: {model.w.item():.4f}, Y: {y_data.item():.4f}, Y_pred: {y_pred.item():.4f}')
# Print gradients
print(f'Gradients: {model.w.grad.item():.4f}')
# After training, you can access the updated weight
print(f'Final weight: {model.w.item():.4f}')
Epoch [1/5], Loss: 9.0000, Updated weight: 2.2000, Y: 5.0000, Y_pred: 2.0000
Gradients: -12.0000
Epoch [2/5], Loss: 0.3600, Updated weight: 2.4400, Y: 5.0000, Y_pred: 4.4000
Gradients: -2.4000
Epoch [3/5], Loss: 0.0144, Updated weight: 2.4880, Y: 5.0000, Y_pred: 4.8800
Gradients: -0.4800
Epoch [4/5], Loss: 0.0006, Updated weight: 2.4976, Y: 5.0000, Y_pred: 4.9760
Gradients: -0.0960
Epoch [5/5], Loss: 0.0000, Updated weight: 2.4995, Y: 5.0000, Y_pred: 4.9952
Gradients: -0.0192
Final weight: 2.4995
Linear Regression
The example above is a simple linear regression model with one weight and no bias. In practice, linear regression models can have multiple weights (coefficients) and an additional bias term. The process of training a linear regression model involves minimizing a loss function (e.g., MSE) by adjusting the weights and bias through gradient descent.
import matplotlib.pyplot as plt
import numpy
n_points = 1000
x = torch.randn((n_points, 1))
y = -5 * x + 2 + torch.randn(n_points,1)
plt.scatter(x, y, s=10, c='black', alpha=0.5)
plt.xlabel('x')
plt.ylabel('-5x + 2')
plt.plot(x, -5 * x + 2, color='orange',)
plt.show()
# Define the weight and bias parameters
w = torch.tensor([1.0], requires_grad=True)
b = torch.tensor([0.0], requires_grad=True)
# h = lambda x: w * x + b # Hypothesis function
def h(x):
return w * x + b
mse = lambda y, yhat: torch.mean((y - yhat) ** 2) # Mean Squared Error (MSE) loss function
w , b
(tensor([1.], requires_grad=True), tensor([0.], requires_grad=True))
indexes = numpy.random.choice(n_points, 10)
indexes, x[indexes]
(array([ 59, 741, 403, 519, 564, 278, 487, 996, 791, 958]),
tensor([[-0.1687],
[ 1.2634],
[-0.1861],
[ 0.6616],
[-0.0601],
[ 0.5264],
[-0.4343],
[-0.2446],
[-0.6709],
[ 1.5085]]))
# stochastic gradient descent
epochs = 10
batch = 20
lr = 0.05
# For tracking progress
losses = []
weights = []
biases = []
# stochastic gradient descent with explicit math steps
epochs = 10
batch = 20
lr = 0.05
# For tracking progress
losses = []
weights = []
biases = []
for epoch in range(epochs):
# Sample a random batch
indexes = numpy.random.choice(n_points, batch)
x_batch = x[indexes]
y_batch = y[indexes]
# Forward pass - compute predictions manually
# Linear function: yhat = wx + b
yhat = h(x_batch)
# Compute MSE loss explicitly: (1/n) * Σ(y - yhat)²
loss = mse(y_batch, yhat)
# Save the current state
losses.append(loss.item())
weights.append(w.item())
biases.append(b.item())
# Compute gradients (backward pass)
loss.backward()
# Print gradient information
# For MSE, ∂Loss/∂w = (-2/n) * Σ(y - yhat) * x
# For MSE, ∂Loss/∂b = (-2/n) * Σ(y - yhat)
print(f"Epoch {epoch} gradients:")
print(f" dLoss/dw: {w.grad.item():.4f}")
print(f" dLoss/db: {b.grad.item():.4f}")
# Manual gradient descent update
with torch.no_grad():
w -= w.grad * lr
b -= b.grad * lr
# Reset gradients for next iteration
w.grad.zero_()
b.grad.zero_()
# Print progress every 2 epochs or at the end
if epoch % 2 == 0 or epoch == epochs-1:
print(f'Epoch {epoch}: Loss = {loss.item():.4f}, Weight = {w.item():.4f}, Bias = {b.item():.4f}')
# Compare final parameters with target parameters
print(f"Final parameters: Weight = {w.item():.4f}, Bias = {b.item():.4f}")
print(f"Target parameters: Weight = -5, Bias = 2")
Epoch 0 gradients:
dLoss/dw: 13.3680
dLoss/db: -1.2836
Epoch 0: Loss = 41.4883, Weight = 0.3316, Bias = 0.0642
Epoch 1 gradients:
dLoss/dw: 18.1593
dLoss/db: -6.4670
Epoch 2 gradients:
dLoss/dw: 10.6063
dLoss/db: -0.4896
Epoch 2: Loss = 24.8467, Weight = -1.1067, Bias = 0.4120
Epoch 3 gradients:
dLoss/dw: 9.9294
dLoss/db: -5.0492
Epoch 4 gradients:
dLoss/dw: 12.3271
dLoss/db: -0.9192
Epoch 4: Loss = 22.4522, Weight = -2.2195, Bias = 0.7104
Epoch 5 gradients:
dLoss/dw: 4.0482
dLoss/db: 0.0120
Epoch 6 gradients:
dLoss/dw: 5.2819
dLoss/db: -3.3917
Epoch 6: Loss = 10.9676, Weight = -2.6860, Bias = 0.8794
Epoch 7 gradients:
dLoss/dw: 6.2168
dLoss/db: -3.1713
Epoch 8 gradients:
dLoss/dw: 2.0559
dLoss/db: -0.8704
Epoch 8: Loss = 3.2740, Weight = -3.0996, Bias = 1.0815
Epoch 9 gradients:
dLoss/dw: 3.1468
dLoss/db: -2.2981
Epoch 9: Loss = 4.8486, Weight = -3.2570, Bias = 1.1964
Final parameters: Weight = -3.2570, Bias = 1.1964
Target parameters: Weight = -5, Bias = 2
w , b
(tensor([-3.2570], requires_grad=True), tensor([1.1964], requires_grad=True))