go to home page   go to next page

created: 01/10/01; revised: 10/14/06, 07/15/07

CHAPTER 4 — Scanners

This is the first of two chapters that discuss how scanners are built. A scanner is a program that reads a stream of characters and divides them into a sequence of groups called tokens. A token is a basic symbol of a programming language. For example, in Java the tokens are the keywords, identifiers, separators, literals, and operators. A scanner is the first phase of a compiler. Natural language processing programs also use scanners.

Chapter Topics:

A scanner is usually implemented as a finite-state transducer. This chapter extends the ideas of the previous sections.

The Java class Scanner (beginning with Java 1.5) implements the idea of a scanner, and can be used for many scanning tasks. However, sometimes you need to write a scanner. The ideas discussed in this chapter will be useful.


Would a spelling checker program use a scanner?