Lesson 2 of 7

Lexing — Text to Tokens

The lexer reads raw characters and groups them into tokens such as words, numbers and operators. It's the simplest stage, and you'll write a working one in about 40 lines.

Text

Input:   let price = 3 * (cost + 12);

Tokens:
  KEYWORD(let)  IDENT(price)  EQUALS
  NUMBER(3)  STAR  LPAREN  IDENT(cost)
  PLUS  NUMBER(12)  RPAREN  SEMICOLON

Whitespace and comments: consumed, discarded.
The lexer knows NOTHING about grammar — '3 + + let ('
lexes fine. Structure is the parser's job.

# A real lexer for arithmetic — this is the whole idea:
def lex(src):
    tokens, i = [], 0
    while i < len(src):
        c = src[i]
        if c.isspace():
            i += 1
        elif c.isdigit():
            start = i
            while i < len(src) and src[i].isdigit():
                i += 1
            tokens.append(("NUMBER", int(src[start:i])))
        elif c.isalpha():
            start = i
            while i < len(src) and src[i].isalnum():
                i += 1
            tokens.append(("IDENT", src[start:i]))
        elif c in "+-*/()=;":
            tokens.append(("OP", c))
            i += 1
        else:
            raise SyntaxError(f"unexpected {c!r} at {i}")
    return tokens

print(lex("price = 3 * (cost + 12);"))
# [('IDENT','price'), ('OP','='), ('NUMBER',3), ('OP','*'), ...]

Real lexers add literals (with escape sequences), multi-character operators (== vs =, maximal munch: always take the longest match), , and position tracking so messages can say line 12, column 8. The core never changes: look at the current character, decide the token type, consume characters until it ends.

Finished reading?Mark it done to track your progress.

Tip: use the ← and → keys to move between lessons.