Notes about backpropagation

Notes about backpropagation

The basic concept of backpropagation seems rather simple: We want to find the derivative of the loss function with respect to a certain weight w_i. This shows us how much that weight affects the loss or error of the network. The value of the derivative at the weight w_i gives us the gradient d_i. We then use that gradient to update each weight w_iwith the following formula:

It's noteworthy that we want to move to the opposite direction of the gradient. Imagine the curve of the loss function. At any given point one can draw a tangent line, which shows the direction of the loss function at that point. The slope of the tangent line is the value of the function's derivative at that piont, which also describes change in the function. Since we're looking to minimize the function, we should move to the opposite direction of the gradient.

For example consider the following figure:

The value of the derivative at point w_i is going to be a negative number, since the curve is descending there. At that point, the local minimum we want to reach is to our right, and so we want to take a step to the right i.e. the positive direction. If the derivative were to be positive, and the loss function rising, we want to go to the left to minimize the loss function.

So we need backpropagation to define d_i for each weight w_i, which means deriving a bunch. That's all fine and dandy and surprisingly intuitive, but how is it implemented?

The answer is rather mathematical. Basically we use the chain rule to divide the derivative into pieces we know.

You can read the calculations here.

So the next question is, how is this done with batches and matrixes. First of all, when training in batches, the gradient used in upating the weights is the sum of all the gradients in a batch. This means that once whe have the errors and neuron outputs in matrices, we can just use matrix multiplication to get the sum of their products.

Let's see what happens in the code in backpropagation:

        errors[-1] = self.loss_derivative * self.saved_activation_functions[-1].derivative(self.saved_inputs[-1])

This is the error base case. self.loss_derivative returns a vector of the size (batch size, output_dimension). It's the derivative of the loss function with regards to the output of the network. loss_derivative is calculated at the same time as loss with

        self.loss_derivative = self.loss.derivative(predictions, labels)

Next we iterate over other layers.

for i in range(no_layers - 2, -1, -1):
    errors[i] = np.dot(np.asarray(self.saved_weights[i]), errors[i+1]) \
                * self.saved_activation_functions[i].derivative(self.saved_inputs[i])

This is a bit hard to formulate because there are no actual neurons in the code, just weights combining two layers. self.saved_weights[i] returns the weights on layer i in the shape (output dimension, input dimension).errors[i+1] returns the error of the next layer. self.saved_inputs[i] means the inputs of the neurons of the layer, e.g. weights of layer i multiplied by the input of that layer (which is the output of the previous layer). We iterate "backwards", starting from the layer before the last, ending in the first. In the end, errors[i] contains the error of the layer i.

The code used to have a bug

which was actually caused by a mistake in forward propagation. In addition, I was mixing these two sites as sources, and ended up multiplying the error of the output layer with the derivative of its activation function twice (in the base case and in the loop)(// edit this ended up being a bug as well, I shouldn't have removed the multiplication :D corrected on week 6). I also transposed the input of the whole network before passing it to the layers, which resulted in shape problems in backpropagation (the inputs were saved in the transposed state even though in backpropagation they weren't supposed to be like that). So the problem was that the previous forward pass did give similar outputs as pytorch in my tests, but the saved inputs behaved differently in backpropagation, as the inputs were transposed under the hood.

I got the neural network working by slowly going through this tutorial with code (my backpropagation implementation is not identical, but gives similiar results with pytorch, and the tutorial gave me an idea about the correct shapes of the matrices). I also altered my forward pass to be the same as pytorch's and did some tests

TL;DR: For a long time, things were the correct shape and size, but didn't work as intended. Oh, linear algebra.