Yes — it divides text into individual words (tokens) then checks their spelling.
You could view
a Java program
as a long sequence of individual characters.
But usually you think of a program as
a sequence of tokens,
such as class
,
Hello
, public
, and so on.
Punctuation marks (separators) also have
significance: {
, (
, [
and others.
class Hello { public static void main ( String[] args ) { System.out.println("Hello World!"); } }
In a formal language (such as a programming language), a character or string of characters that has significance as an indivisible symbol is a token of the language.
A token is a
word-like group
that has meaning in a formal language
and can't be divided into smaller meaningful
pieces.
In the example program,
the string class
is a token,
but its individual characters are not.
The identifier Hello
is a token,
as is {
.
However, System.out.println
consists of
five tokens: three identifiers and two dots.
In processing a formal language the first step is to divide the text into tokens. Exactly what counts as a token depends on the language, and must be precisely defined. Natural language, such as English text, is often processed as if it consists of tokens, although it is sometimes unclear what the tokens are. (This is what a spelling checker or text formatter would do).
How many tokens are in the following Java statement:
int value = 12 ;