CHAPTER 4 — Scanners

This is the first of two chapters that discuss how scanners are built. A scanner is a program that reads a stream of characters and divides them into a sequence of groups called tokens. A token is a basic symbol of a programming language. For example, in Java the tokens are the keywords, identifiers, separators, literals, and operators. A scanner is the first phase of a compiler. Natural language processing programs also use scanners.

Chapter Topics:

• Tokens and delimiters
• Scanners as finite automata
• Character input from a file

A scanner is usually implemented as a finite-state transducer. This chapter extends the ideas of the previous sections.

The Java class Scanner (beginning with Java 1.5) implements the idea of a scanner, and can be used for many scanning tasks. However, sometimes you need to write a scanner. The ideas discussed in this chapter will be useful.