AI ANN Neural Networks - Zero to Hero Andrej Karpathy
The spelled-out intro to neural networks and backpropagation: building micrograd
- Step by step instructions for building micrograd
- "micrograd is all you need to train a artificial neural network, everything else is efficiency"
- The backpropagation part is only ~100 lines of Python
- What is a derivative measuring?
- If you increase some number ($a$) by some small number ($h$) with what sensitivity does the slope respond?
- Does the function go up or down? By how much?
- Eventually you get a lot of connected neurons and a loss function
- The loss measures the accuracy of the neural net
- We backpropagate with respect to the accuracy, trying to increase it
- Tensors are n-dimensional arrays of scalars
- The tuning of the loss function is a subtle art
- Too low of a learning rate --> too long to converge
- Too high of a learning rate --> unstable, loss could explode
- It's common to forget to zero out the grads prior to performing backpropagation