Tokenizer playground
@peterhdd

See how GPT, LLaMA, and other models chop your prompt into tokens.

Pick a model, type a prompt, and visualize the token boundaries, IDs, and vocabulary size.

Model

Choose a tokenizer

Idle

Vocabulary size

—

Tokens produced

—

Special tokens

—

Visualization

Token stream

Run a prompt to see color-coded tokens appear here.

#TokenIDType
No tokens yet.