ACompressed recap
🧾
The lexer cuts the character stream into lexemes, matches each against a pattern, and returns a token ⟨name, attribute⟩ to the parser on demand. It strips whitespace and comments, fills the symbol table and counts lines. When counting tokens: comments and blanks give none, a string literal is one, multi-character operators are one, and runs of operator characters split by longest match.
BMerged rule card
1.1.1 Tokens, lexemes, patterns and attributes; how the lexer works with the parser and symbol table
- Token = ⟨token name, attribute⟩; lexeme = matched characters; pattern = rule (regular expression).
- Token classes: keywords, identifiers, constants, string literals, operators, punctuation.
- Attributes: id → symbol table pointer; num → value; relop → which operator.
- Parser calls getNextToken(); lexer strips blanks/comments, fills symbol table, counts lines.
- Each occurrence = one token; main/printf are identifiers.
1.1.2 Counting tokens in C code: whitespace, comments, string literals, ++ and >=
- Whitespace, comments → 0 tokens.
- String literal, character constant → 1 token each, whatever is inside.
- Keyword, identifier, number (3.14, 2.5e3) → 1 each.
- Each punctuation mark and each operator → 1; multi-character operators (++, >=, +=, ->, =) count once.
- Longest match from the left: x+++y = x ++ + y.
- C comments do not nest; quotes inside comments and /* inside quotes are plain text.
CInterleaved GATE-style questions
Q1NAT2 marksToken count
How many tokens are in int x, y; float z = x * 2.5e3; /* scale */ return z >= y;?
Numerical answer: ________
Q2NAT1 markToken count
How many tokens are in a = b ? c : d->e;?
Numerical answer: ________
Q3MCQ1 markTerms
The rule letter (letter | digit)* for identifiers is a:
- (A) lexeme
- (B) token
- (C) pattern
- (D) attribute
Q4MSQ1 markJobs
Which are performed by the lexical analyzer?
- (A) skipping whitespace
- (B) detecting an undeclared variable
- (C) detecting an illegal character such as @
- (D) counting lines for error messages
Q5NAT1 markToken count
How many tokens are in printf("%d %d", a, b);?
Numerical answer: ________
Q6MCQ1 markLongest match
How does the lexer split i+++++j?
- (A) i ++ + ++ j
- (B) i ++ ++ + j
- (C) i + ++ ++ j
- (D) i +++ ++ j
Q7MCQ2 marksAttributes
Two occurrences of rate in one function produce:
- (A) one token, because the name is the same
- (B) two tokens whose attributes point to the same symbol table entry
- (C) two tokens with different symbol table entries
- (D) no tokens; names are stored only in the symbol table
Q8NAT2 marksComments
How many tokens are in a = b /* c = d; */ + e; // f;?
Numerical answer: ________
Answer Vault
Try the questions first. Open when you are ready to check.
Q1 Answer: 172 marks
int · x · , · y · ; · float · z · = · x · * · 2.5e3 · ; · return · z · >= · y · ; = 17.
Re-read: Lesson 1.1.2 — Counting tokens in C code: whitespace, comments, string literals, ++ and >=
Q2 Answer: 101 mark
a · = · b · ? · c · : · d · -> · e · ; = 10 (-> is one token).
Re-read: Lesson 1.1.2 — Counting tokens in C code: whitespace, comments, string literals, ++ and >=
Q3 Answer: (C)1 mark
A pattern describes the set of lexemes of one token name.
- (A) is wrong: A lexeme is one actual character sequence.
- (B) is wrong: A token is the pair the lexer returns.
- (D) is wrong: An attribute is extra information inside a token.
Re-read: Lesson 1.1.1 — Tokens, lexemes, patterns and attributes; how the lexer works with the parser and symbol table
Q4 Answer: (A) (C) (D)1 mark
Undeclared variables are a semantic check.
Re-read: Lesson 1.1.1 — Tokens, lexemes, patterns and attributes; how the lexer works with the parser and symbol table
Q5 Answer: 91 mark
printf · ( · "%d %d" · , · a · , · b · ) · ; = 9.
Re-read: Lesson 1.1.2 — Counting tokens in C code: whitespace, comments, string literals, ++ and >=
Q6 Answer: (B)1 mark
Greedy from the left: ++, ++, then +. (The parser then reports an error, because i++ cannot be incremented again.)
- (A) is wrong: After the first ++, the next two + form another ++.
- (C) is wrong: The first two + already form ++.
- (D) is wrong: C has no +++ operator.
Re-read: Lesson 1.1.2 — Counting tokens in C code: whitespace, comments, string literals, ++ and >=
Q7 Answer: (B)2 marks
Each occurrence is a token; both refer to the one entry for rate.
- (A) is wrong: Every occurrence produces a token.
- (C) is wrong: One name, one entry (in the same declaration region).
- (D) is wrong: The parser needs a token for every occurrence.
Re-read: Lesson 1.1.1 — Tokens, lexemes, patterns and attributes; how the lexer works with the parser and symbol table
Q8 Answer: 62 marks
Without comments: a · = · b · + · e · ; = 6.
Re-read: Lesson 1.1.2 — Counting tokens in C code: whitespace, comments, string literals, ++ and >=