Scanner

A scanner is a program that groups input characters into the tokens of a particular language. A scanner is crucial in any program with data formatted as groups of characters. Of course, this is a common situation. Scanners are useful for dealing with many kinds of data.

A scanner can be implemented as a finite-state transducer. An application program that processes a stream of data would call the scanner each time it is ready for the next token. Tokens are delivered (and then processed) one by one, in order.

A scanner must isolate each token from its neighbors. It must determine the first and last character of each token. Early programming languages were designed before scanners were understood. Determining the tokens was difficult. Modern languages are designed to make scanning easy.

For example, in early versions of FORTRAN

GOTO 5

means to transfer control to statement number 5. However

GOTO 5  = 15

means to assign 15 to the variable GOTO 5. A scanner must scan up to the line end or = to figure out if the first six characters are one token or two.