Padding and Masks

Training and some local stacks batch several sequences. They must have the same length, so shorter rows get pad tokens.

Attention must not treat pad as content. A mask sets those scores to a huge negative number before softmax, the same trick as hiding the future. Forget the mask and the model “attends to pad” and gets dumber in a way that looks like a random seed bug.

Left pad vs right pad matters for decoders. Causal models usually want the real tokens packed so the last position is the last real token (the one you read logits from). Pad the unused side, then mask it. If you left-pad a decoder and then read logits at the last index, you may be reading a pad position. That is a silent empty generation.

You rarely write the mask by hand on a hosted call. You do hit pad bugs when you build a local batcher or a fine-tune collator. This lesson is that collator.

A wrong picture

A wrong picture is: “pad is empty, so mixing it in is like mixing zeros, harmless.” Pad is a real id with a real embedding row. Even a zero row, if unmasked, dilutes softmax mass. A trained pad row can be worse than zeros. Mask it.

Another wrong picture is: “I can type <pad> in the prompt to save space.” That is English (or a lookalike), not the pad id. Special tokens lesson: do not confuse the string with the id.

A third wrong picture is: “one EOS for the whole batch.” Variable-length generation needs EOS per row. A naive loop that waits for the longest answer bills you for pad decode on the finished rows unless the server is clever. Finished rows should stop; siblings may continue.

Pad for shape, mask for mix

Batch two sequences:

• [4, 5, 6]
• [7, 8]

Max length 3. Right pad the second: [7, 8, 0] if pad id is 0. The tensor is now 2 by 3. Attention on row 1 must block column 2 (the pad). Causal must also block the future. Combined: pad OR future → blocked.

Legal Y marks sit only on real tokens, and never on the future. That combined mask is what production kernels implement.

Left pad would yield [0, 7, 8] for the short row. Then the last index is a real token, which can be convenient for “read logits at position -1.” You must still mask the leading pad. Mixing left-pad and “read last index” without thinking is how people read pad logits.

Causal plus pad, in words

For query index i and key index j on one row:

• If the key is pad → illegal
• If the query is pad → do not care; you will not use those logits
• If causal and j > i → illegal
• Else legal

Softmax on illegal scores (huge negative) gives ~0 weight. The mix ignores pad. Residuals still add whatever the layer outputs; if the layer ignored pad, you are safe.

Never generate pad as a content token. EOS ends a row. Pad is for shape.

Right-pad a batch and print the mask

Lists of integer ids. Y allowed, . blocked. Pad id 0.

Run to execute this in your browser. Nothing is sent to a server.

The shorter row has pad on the right. Legal Y marks sit only on real tokens, and never on the future. The last query on the short row is pad: all dots. Do not read logits there.

The long row is a causal triangle of Y. That is the decoder picture with no pad.

If loss is weirdly good and generations are garbage, print a batch and look for pad ids inside the “content.” If labels were not masked, the model is rewarded for predicting pad. That collator bug is famous.

Collator bugs that look like model bugs

• Labels include pad, so loss is easy (predict 0)
• Attention unmasked, so pad dilutes every row
• Left-pad + logits at last index on a short row that was actually right-padded
• EOS missing, so the row never stops and pad decode continues
• Different max length at train vs run, so position tables disagree

Print the batch. Print the mask. Print which index you read logits from. That is the debug ritual. The transformer block is innocent until those prints are clean.

Fine-tunes die here more often than people admit. A LoRA cannot fix a collator that trains on pad.

Common mistakes

Right-padding a decoder batch and then always reading logits at index max_len - 1. Short rows are pad there. You sample pad or junk. Read the last real token index, or left-pad carefully and still mask.

Masking the future but forgetting pad. Pad sits in the past, so causal allows it. Softmax then mixes pad into every real token. Combined mask: future OR pad.

Training with pad in the label tensor. Loss on “predict pad” is easy. The reported loss looks great. Generation is garbage. Mask labels too.

Using the English string <pad> as if it were free space in a chat prompt. It is content tiles, or a lookalike of a special, not a shorter window.

Batching two different tasks with different EOS habits and sharing one stop. One row finishes; the other eats pad decode. Per-row EOS.

How agents use this

You rarely write the mask by hand on a hosted chat call. You do hit pad bugs when you build a local batcher or a fine-tune collator. If loss is weirdly good and generations are garbage, print a batch and look for pad ids inside the “content.” If you left-pad a decoder and then read logits at the last index, you may be reading a pad position.

When you pad tool traces to a fixed length for a tiny local model, mask them. When you pack a window (context-window lesson), you are choosing content, not pad. Do not pad a prompt with pad ids “to fill 4k.” That is not a packing strategy. That is noise.

• Pad: shape only.
• Mask: pad out of softmax.
• Last index: must be a real token if you read logits there.
• EOS: per row, not per batch.
• Collator: print ids before you blame the tune.

Watch out:Special token pad in the prompt text as English is not the same as the pad id. Do not type it to “save space.”