Tokens

You could view a Java program as a long sequence of individual characters. But usually you think of a program as a sequence of tokens, such as class, Hello, public, and so on. Punctuation marks (separators) also have significance: {, (, [ and others.

class Hello
{
  public static void main ( String[] args )
  {
    System.out.println("Hello World!");
  }
}

In a formal language (such as a programming language), a character or string of characters that has significance as an indivisible symbol is a token of the language.

A token is a word-like group that has meaning in a formal language and can't be divided into smaller meaningful pieces. In the example program, the string class is a token, but its individual characters are not. The identifier Hello is a token, as is {.  However, System.out.println consists of five tokens: three identifiers and two dots.

In processing a formal language the first step is to divide the text into tokens. Exactly what counts as a token depends on the language, and must be precisely defined. Natural language, such as English text, is often processed as if it consists of tokens, although it is sometimes unclear what the tokens are. (This is what a spelling checker or text formatter would do).