Exercises / transformers

Whitespace tokenize

tokenize(text) lowercases and splits on spaces. Detokenize with join. Round-trip a short sentence and print token count.

Run to execute this in your browser. Nothing is sent to a server.