← Lesson
Why Transformers
Joeven
Run
Reset
Python loads on first run
def rnn_steps(n): return n def transformer_pairs(n): return n * n def parallel_train_positions(n): return n print("n | rnn steps | attn pairs | train positions in parallel") for n in (8, 64, 512): print(n, rnn_steps(n), transformer_pairs(n), parallel_train_positions(n)) print("RNNs scale with depth of time; attention scales with pairs; training still parallel") print("causal triangle is about n*(n+1)/2 pairs, still quadratic") for n in (8, 64, 512): print("causal pairs", n, n * (n + 1) // 2)
Run to execute this in your browser. Nothing is sent to a server.