A Tiny Neural Net

A neural network is a stack of linear maps with bends (nonlinearities) between them. Without the bend, the whole stack is still one linear map and cannot learn XOR-shaped problems. With it, you get a messy function that can fit your data.

This page is a forward pass you can print. It is not a course on language-model internals. Billions of parameters are still Wx + b and a bend, repeated. You already know the training loop: loss, slope, step. Depth adds capacity — and overfitting, cost, and latency.

One neuron

A neuron: z = w · x + b, then a = f(z). For modern nets, f is often ReLU: max(0, z). Logistic sigmoid squashes to (0, 1) for chances. You already trained that as a linear classifier.

Dot means multiply matching slots of two lists and add. Bias is an extra knob added after the weighted sum. A linear map from a list of length 2 to a list of length 2 is two dots: one per output slot. We store that map as a list of rows (each row is the weights for one output).

One hidden layer

Input x (length 2) → hidden (length 2) → output (length 1):

1. h_raw = W1 x + b1
2. h = relu(h_raw)
3. y = W2 h + b2

W1 is two hidden neurons, each with two weights. This is the same shape as a small feed-forward block: matrix times list, then a bend. We will not unpack attention or token stacks here. If you can write matvec, you can read “linear projection” in a paper without panic.

Why the bend: linear then linear is still linear. W2 (W1 x) is some other matrix times x. ReLU is the cheapest bend that lets regions turn off (output zero) so different parts of space can use different linear pieces.

When ReLU outputs zero, the slope through that unit is zero for that example. If it stays off, it is “dead.” For agents, the analog is a tool the policy never calls: it cannot learn from a path it never takes. A little temperature on the router, or a forced tool on eval, keeps tools alive.

Run to execute this in your browser. Nothing is sent to a server.

What printed: four forward passes. (0, 0), (1, 0), and (0, 1) end quiet (y at or below 0): at least one ReLU is off, so the output never clears the last bias. (1, 1) fires: both hidden slots go positive (1.0 - 0.5 = 0.5), they add, minus 0.5, y is positive. That is a nonlinear decision — both inputs must be on. A single linear neuron cannot draw that “both must be on” region as cleanly.

Weights were hand-set, not learned. Training would run gradient descent on W1, b1, W2, b2 with a loss. The forward picture would not change.

Capacity, depth, and what you should not do yet

Width is how many hidden units. Depth is how many layers. More of either is more knobs. More knobs fit more patterns and more accidents. Regularize. Split. Early-stop. A tiny net around an LLM is often enough: injection detector, chunk ranker, abstain head.

You do not need to train a giant net to use one. Calling a hosted model is inference of a giant net. Your job is still the ML loop: features, split, metric, baseline. The giant function does not excuse a 12-trace eval.

Skip transformer internals here. If you need tokens and attention, that is a later track. This track’s neural net is: lists, dots, a bend, a loss, a downhill step.

What the tiny net is for, and what it is not

The live box is XOR-shaped on purpose: a pattern a straight cut misses. Agent routing sometimes looks like that: “call sql if the user wants a count and we already searched,” or “escalate if VIP and refund.” You can hard-code the AND as a feature (vip_and_refund = vip * refund) and stay linear. You can also let a hidden layer learn a bend. The feature is usually cheaper and easier to log. Use the net when the ANDs and ORs are many and unnamed.

Training this net would mean a loss on y vs the output, slopes through ReLU (0 if the unit was off, 1 if it was on), and a step on every weight. If a unit is off for every train row, its incoming weights get zero slope — dead. A small random start and a sensible scale of inputs keep more units alive. Print the hidden list on a few rows after a few steps. All zeros is a bug report.

A 2-layer net on 40 traces will memorize. Regularize, or do not use the net. The LLM already has more than enough capacity. The tiny net’s job is cheap, inspectable, local: injection-ish scores, abstain heads, rerank on a handful of features. If you need language understanding, call the frozen LLM. If you need a millisecond router, a linear map or this tiny net is the shape.

Do not stack more layers because a diagram had more layers. Depth without a bend is wasted. Depth with a bend and no extra data is overfitting. Stay on this side of the line until you have labels, a split, and a dummy that lost.

Common mistakes

• Stacking linear layers with no bend and expecting XOR.
• Celebrating train accuracy on four points.
• A huge hidden layer on 40 traces.
• Dead ReLUs (all zeros) and no print of hidden lists.
• Assuming a deep net replaces a keyword baseline.

How agents use this

Small nets still earn rent around the LLM: a tiny classifier for prompt injection, a ranker for chunks, a “should we abstain?” head. Running a 2-layer net on CPU is cheaper than another 8k-token call. Split data, pick a loss, report precision/recall, watch validation.

If you can write matvec, you can log a linear router and a tiny net the same way: print the hidden list when you debug. “All zeros” is a dead path. “This unit fires only on refund” is a feature you could have written by hand — and maybe should have.

Tip:If you can write matvec, you can read a paper’s “linear projection” without panic. It is a matrix times a vector.