go to previous page   go to home page   go to next page
int value = 12 ;


Five. The tokens are



A scanner is a program that groups input characters into the tokens of a particular language. A scanner is crucial in any program with data formatted as groups of characters. Of course, this is a common situation. Scanners are useful for dealing with many kinds of data.

Scanner with input stream of characters and output stream of tokens

A scanner can be implemented as a finite-state transducer. An application program that processes a stream of data would call the scanner each time it is ready for the next token. Tokens are delivered (and then processed) one by one, in order.

A scanner must isolate each token from its neighbors. It must determine the first and last character of each token. Early programming languages were designed before scanners were understood. Determining the tokens was difficult. Modern languages are designed to make scanning easy.

For example, in early versions of FORTRAN


means to transfer control to statement number 5. However

GOTO 5  = 15

means to assign 15 to the variable GOTO 5. A scanner must scan up to the line end or = to figure out if the first six characters are one token or two.


How many tokens are in the following Java statement:

myObject.age = 13.6;