Autograd From Scratch

Stack: Python, pytest, mypy, ruff, black, GitHub Actions, semantic-release · Repo: Axeloooo/autograd-from-scratch

Problem

This project builds backpropagation in plain Python from one idea: a scalar Value node that records the operation that produced it and a closure that pushes gradients back to its inputs. A small Neuron / Layer / MLP library sits on top and trains on tiny datasets.

The runtime has no dependencies (dependencies = []), and engine, network and graph export total about 260 lines.

Architecture diagram

  nn.py:   Neuron --> Layer --> MLP
                                 |
                  forward pass builds the graph
                                 v
  +----------------------------------------------------+
  | engine.py   Value                                  |
  |   data, grad, label, _prev, _op, _backward         |
  |   add / mul / pow / exp / log / tanh / relu        |
  |   each op = new Value + closure doing  grad += ... |
  +----------------------------------------------------+
        |                                  |
        | backward(): iterative            | trace() / to_dot()
        | topological sort, then           | graph.py
        | reversed _backward calls         v
        v                            Graphviz DOT text
  MLP.parameters()  (.grad filled in)
        |
        v
  p.data -= lr * p.grad      <- update loop lives in the README and tests

Value stores its data, gradient, child nodes, the operation and a backward closure. backward() sorts the graph topologically, seeds the root gradient with 1.0 and runs the closures in reverse. graph.py walks the same graph to emit DOT text. The training loop is deliberately not library code.

Key decisions

• Accumulate gradients with +=. A value used more than once must sum its contributions; test_value_used_twice_accumulates covers it.
• Iterative topological sort. backward() uses an explicit stack so a very deep graph cannot hit Python's recursion limit. A test builds a 5,000-node chain.
• Few primitive derivatives. Subtraction and division are composed from * -1.0 and ** -1, so only add, mul, pow, exp, log, tanh and relu have hand-written backward rules.
• DOT as plain text. to_dot returns Graphviz source and escapes record-label characters, so the graphviz package is not needed.
• Defensive details. Neuron and MLP accept an injectable random.Random, and zip(..., strict=True) catches input-size mismatches.

Challenges

• Deep graphs. backward() was recursive until v1.2.0, which made it iterative alongside graph drawing; the 5,000-node test pins the fix.
• Trusting hand-written derivatives. Tests compare analytic gradients with finite differences across nine parametrized cases (relu at +0.9 and -0.9 among them) and check a hand-worked expression with loss -8.0. relu's derivative at exactly 0 is 0, and no test covers that point.

Results

• A working scalar autograd engine, an MLP that learns a four-sample dataset (the test requires the loss to fall below 10% of its start after 50 steps), and DOT export of any computation graph.
• Three test modules (engine, network, graph). CI runs ruff, black, mypy, prettier and pytest on Python 3.10 and 3.13; mypy is strict and the package ships py.typed.
• Releases v1.0.0 to v1.2.0: Conventional Commits are enforced by commitlint and pre-commit (with gitleaks), release.yaml runs semantic-release on main and merges main back into devel.

Links

• Repository