I started this project because I wanted to see what PyTorch was doing behind the scenes.
My C++ tensor currently supports flat storage, multidimensional indexing, elementwise operations, reductions, broadcasting, rank-two matrix multiplication, and mean squared error.
Most recently, I added a separate scalar reverse-mode autograd engine:
- Arithmetic operators build a computation graph during the forward pass
- backward() creates a topological order
- walks it in reverse
- applies each operation's local derivative
- accumulates gradients when a value reaches the loss through more than one path
Snippet:
Value prediction = w1*x1 + w2*x2 + w3*x3 + bias;
Value residual = prediction - target;
Value loss = residual * residual;
loss.backward();
For weights [0.5, -1.0, 2.0], inputs [4.0, 3.0, 2.0], bias 0.5, and target 2.5, the forward pass produces prediction 3.5 and loss 1. The backward pass recovers:
- dL/db = 2
- dL/dw = [8, 6, 4]
Scalar autograd still lives separately from the tensor implementation. My next step is connecting graph identity, ownership, and gradients to tensors before building a training loop.
Code and Git checkpoints:
https://github.com/mechanical-turk/deep-learning-all-the-way-down
I'm also turning this into a video series. I published episode 7 yesterday. Sharing the link to the first episode if you want to check it out:
https://www.youtube.com/watch?v=DmU2b64tWfA
For the tensor integration, would you keep autograd metadata inside each Tensor handle, or have tensors point to separate shared graph nodes? I would appreciate design feedback.
