Source Code to Tokens
← Back to Lexing - Tokenization
The lexer reads the raw character stream and groups characters into tokens based on patterns (regular expressions). Each token has a type (keyword, identifier, number, string, operator) and a value, forming the input for the parser.