AI ANN Neural Networks - Zero to Hero Andrej Karpathy

# The spelled-out intro to neural networks and backpropagation: building micrograd

## Resources

## Notes

- Step by step instructions for building micrograd
- "micrograd is all you need to train a artificial neural network, everything else is efficiency"
- Like Nand to Tetris but for neural networks?

- The backpropagation part is only ~100 lines of Python
- What is a derivative measuring?
- If you increase some number ($a$) by some small number ($h$) with what sensitivity does the slope respond?
- Does the function go up or down? By how much?

- Eventually you get a lot of connected neurons and a loss function
- The loss measures the accuracy of the neural net
- We backpropagate with respect to the accuracy, trying to increase it

- Tensors are n-dimensional arrays of scalars
- The tuning of the loss function is a subtle art
- Too low of a learning rate --> too long to converge
- Too high of a learning rate --> unstable, loss could explode

- It's common to forget to zero out the grads prior to performing backpropagation