CcCompilers · Lesson 2 of 7

Lexing — Text to Tokens

The lexer reads raw characters and groups them into tokens: words, numbers, operators. It's the simplest stage — simple enough that you can write a real one in 40 lines, and you're about to.

Text
Input:   let price = 3 * (cost + 12);

Tokens:
  KEYWORD(let)  IDENT(price)  EQUALS
  NUMBER(3)  STAR  LPAREN  IDENT(cost)
  PLUS  NUMBER(12)  RPAREN  SEMICOLON

Whitespace and comments: consumed, discarded.
The lexer knows NOTHING about grammar — '3 + + let ('
lexes fine. Structure is the parser's job.
Python
# A real lexer for arithmetic — this is the whole idea:
def lex(src):
    tokens, i = [], 0
    while i < len(src):
        c = src[i]
        if c.isspace():
            i += 1
        elif c.isdigit():
            start = i
            while i < len(src) and src[i].isdigit():
                i += 1
            tokens.append(("NUMBER", int(src[start:i])))
        elif c.isalpha():
            start = i
            while i < len(src) and src[i].isalnum():
                i += 1
            tokens.append(("IDENT", src[start:i]))
        elif c in "+-*/()=;":
            tokens.append(("OP", c))
            i += 1
        else:
            raise SyntaxError(f"unexpected {c!r} at {i}")
    return tokens

print(lex("price = 3 * (cost + 12);"))
# [('IDENT','price'), ('OP','='), ('NUMBER',3), ('OP','*'), ...]

Real lexers add string literals (with escape sequences), multi-character operators (== vs =, maximal munch: always take the longest match), comments, and position tracking so error messages can say line 12, column 8. The core loop never changes: look at the current character, decide the token type, consume characters until it ends.

✦ Tip
'Unexpected token' errors come from this stage's output: the parser received a legal token in an illegal place. 'Unexpected character' or 'invalid token' means the lexer itself choked — usually a stray symbol or an unterminated string.