Backpropagation

Backpropagation algorithm involves several steps. Here is a high-level overview of the process

Steps

1. Forward Pass

• Input: Start by feeding the input data through the neural network layer by layer.
• Weighted Sum: Each layer performs a weighted sum of the inputs from the previous layer, which is calculated as the dot product of the input vector and the weight matrix.
• Activation Function: Apply an activation function to the weighted sum to introduce non-linearity and produce the output of the layer.

2. Loss Calculation

• Compare Predictions with True Labels: Calculate the discrepancy between the predicted outputs obtained from the forward pass and the true labels associated with the input data.
• Loss Function: Use a suitable loss function to quantify the error between the predictions and true labels. The choice of the loss function depends on the task at hand (e.g., mean squared error for regression, cross-entropy loss for classification).

3. Backward Pass (Gradient Calculation)

• Gradient of Loss with Respect to Parameters: Calculate the gradients of the loss function with respect to the parameters of the neural network. This step involves applying the chain rule of calculus to propagate the gradients backward through the network.
• Partial Derivatives: Compute the partial derivatives of the loss function with respect to each parameter (weights and biases) in the network. These derivatives indicate the sensitivity of the loss to changes in the parameters.

4. Gradient Descent

• Update Parameters: Adjust the parameters of the neural network based on the gradients computed in the previous step. The most common optimization algorithm used is gradient descent.
• Learning Rate: Determine the learning rate, which controls the step size taken in the direction of the gradients during parameter updates. It is a hyperparameter that affects the convergence and stability of the training process.
• Parameter Update Rule: Update each parameter by subtracting the product of the learning rate and its corresponding gradient. This update rule moves the parameters towards the direction that reduces the loss function.

5. Iterative Process

• Repeat Steps 1 to 4: Perform the forward pass, loss calculation, backward pass, and gradient descent steps iteratively for a specified number of epochs or until a convergence criterion is met.
• Batch Processing: The training data is typically divided into smaller batches for efficiency. The gradients are calculated and the parameters are updated based on the average of the gradients computed over the batch, reducing the computational cost.

Loss Function

A loss function, also known as an objective function or cost function, quantifies the discrepancy between the predicted outputs of a machine learning model and the true outputs. It provides a measure of how well the model is performing on a given task. The choice of a loss function depends on the problem type (regression, classification, etc.) and the desired behavior of the model.

Commonly used loss functions include

• Mean Squared Error (MSE): MSE is typically used for regression problems. It computes the average squared difference between the predicted and true values. The goal is to minimize this value, aiming for the predicted values to be as close as possible to the true values.

• Binary Cross-Entropy: Binary cross-entropy is often used for binary classification tasks. It measures the dissimilarity between the predicted probability distribution and the true binary labels. The objective is to minimize this value, encouraging the model to assign higher probabilities to the correct class.

• Categorical Cross-Entropy: Categorical cross-entropy is employed for multi-class classification problems. It quantifies the dissimilarity between the predicted class probabilities and the true one-hot encoded labels. The aim is to minimize this value, encouraging the model to assign high probabilities to the correct class and low probabilities to the others.

Gradient Descent

Gradient descent is an optimization algorithm used to minimize a loss function by iteratively updating the model parameters. It relies on the gradient, which represents the rate of change of the loss function with respect to the parameters. The goal is to find the set of parameters that correspond to the minimum of the loss function.

Here's an outline of how gradient descent works:

• Initialize Parameters: Start by initializing the parameters of the model randomly or with predefined values.

• Forward Pass: Perform a forward pass through the network to obtain predictions for a batch of training examples.

• **Loss Calculation: ** Compute the loss function based on the predicted outputs and the true labels.

• Backward Pass (Gradient Calculation): Calculate the gradients of the loss function with respect to the parameters using backpropagation. This involves computing the partial derivatives of the loss with respect to each parameter in the network.

• Parameter Updates: Update the parameters by taking a small step in the opposite direction of the gradients, multiplied by a learning rate. The learning rate controls the size of the step taken in each iteration.

• Repeat: Repeat steps 2 to 5 for a specified number of iterations or until convergence, where the loss function reaches a minimum or no longer decreases significantly.

By repeatedly performing the forward and backward passes, the neural network gradually learns to improve its predictions by adjusting the weights and biases based on the calculated gradients. This iterative process allows the network to learn from the training data and optimize its performance.