{
{   The purpose of this request is to scan the next lexical token in a line.
{ On entry, INDEX indicates where to begin scanning TEXT.  On exit, INDEX
{ indicates where scanning stopped (i.e.  the next character, if any, to be
{ scanned).
{
{   The TEXT_INDEX field of TOKEN is set to the position of the first
{ character of the returned token.  The TEXT_SIZE field of TOKEN is set to the
{ number of characters occupied by the token in the text (i.e.  exit value of
{ INDEX - TOKEN.TEXT_INDEX).  The DESCRIPTOR field of TOKEN is set to a string
{ that describes the token, e.g.  for use in error messages.
{
{   Name tokens are 1 to 31 characters in length and are returned in TOKEN.STR
{ with all lower case letters translated to their upper case counterparts.
{ The following BNF definitions illustrate the syntax of a name:
{
{    <clc$name_token> ::= <alphabetic char> [<alphanumeric char>]...
{    <alphanumeric char> ::= <alphabetic char> | <digit>
{    <alphabetic char> ::= <letter>
{                        | <special alphabetic char>
{                        | <international letter>
{    <letter> ::= <upper case letter> | <lower case letter>
{    <upper case letter> ::= A | B | C | D | E | F | G | H | I | J | K | L | M
{                          | N | O | P | Q | R | S | T | U | V | W | X | Y | Z
{    <lower case letter> ::= a | b | c | d | e | f | g | h | i | j | k | l | m
{                          | n | o | p | q | r | s | t | u | v | w | x | y | z
{    <international letter> ::= <upper case international letter>
{                             | <lower case international letter>
{    <upper case international letter> ::= @ | '[' |  \  | ^ | ']'
{    <lower case international letter> ::= ` |  {  | '|' | ~ |  }
{    <special alphabetic char> ::= # | $ | _
{    <digit> ::= 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9
{
{   A CYBIL name token differs from an SCL name token in that it must start
{ with a letter and may not contain any of the international alphabetic
{ characters except @.  The distinction between a CYBIL name and an SCL name
{ is made only if the scan option CLC$CLASSIFY_NAME_TOKEN is selected.  The
{ following BNF definitions illustrate the syntax of a CYBIL name:
{
{    <clc$cybil_name_token> ::= <letter> [<cybil name char>]...
{    <cybil name char> ::= <letter>
{                        | <special alphabetic char>
{                        | @
{                        | <digit>
{
{   A special CYBIL name token differs from a regular CYBIL name token in that
{ it starts with either a $ or a # and is 2 to 32 characters in length.  The
{ intent of this kind of token is to accomodate set value constructor
{ notation.  A special CYBIL name is recognized as such only if the scan
{ options CLC$CLASSIFY_NAME_TOKEN and CLC$SPECIAL_CYBIL_NAME_IS_TOKEN are
{ selected.  The following BNF definitions illustrate the syntax of a special
{ CYBIL name:
{
{    <clc$special_cybil_name_token> ::= $ <clc$cybil_name_token>
{                                     | # <clc$cybil_name_token>
{
{   A simple name token contains only letters or digits and must start with a
{ letter.  Thus a simple name may be considered to be any of the other kinds
{ of names (except a special CYBIL name).  The distinction between a simple
{ name and an SCL name is made only if the scan option CLC$CLASSIFY_NAME_TOKEN
{ is selected.  The following BNF definitions illustrate the syntax of a
{ simple name.
{
{    <clc$simple_name_token> ::= <letter> [<letter | digit>]...
{    <letter | digit> ::= <letter> | <digit>
{
{   COBOL name tokens are 1 to 30 characters in length and may contain
{ hyphens.  A COBOL name is recognized only if the scan options
{ CLC$CLASSIFY_NAME_TOKEN and CLC$COBOL_NAME_IS_TOKEN are selected.  The
{ following BNF definitions illustrate the syntax of a COBOL name:
{
{    <clc$cobol_name_token> ::=
{          <letter> [[<COBOL name char>]... <letter | digit>]
{        | <digit> [<COBOL name char>]... <letter>
{                 [[<COBOL name char>]... <letter | digit>]
{    <COBOL name char> ::= <letter> | <digit> | <hyphen>
{    <letter | digit> ::= <letter> | <digit>
{    <hyphen> ::= -
{
{   Integer tokens may have radix specifications and must be delimited at both
{ ends.  The default radix is decimal (10).  The following BNF definitions
{ illustrate the syntax of integers:
{
{    <clc$signed_integer_token> ::= <sign> <clc$unsigned_integer_token>
{    <sign> ::= + | -
{    <clc$unsigned_integer_token> ::=  <digit> [<hex digit>]... [(<radix>)]
{    <hex digit> ::= <digit> | A | B | C | D | E | F
{                            | a | b | c | d | e | f
{    <radix> ::= <unsigned decimal>
{    <unsigned decimal> ::= <digit>...
{
{   Real tokens, must have fractional portions and may have an exponent.  The
{ following BNF definitions illustrate the syntax of real numbers:
{
{    <clc$signed_real_token> ::= <sign> <clc$unsigned_real_token>
{    <clc$unsigned_real_token> ::= <mantissa> [<exponent>]
{    <mantissa> ::= <integer part> . <fraction part>
{    <integer part> ::= <unsigned decimal>
{    <fraction part> ::= <unsigned decimal>
{    <exponent> ::= <E|e> [<sign>] <unsigned decimal>
{    <E|e> ::= E | e
{
{   Strings must be enclosed in apostrophes (single quote marks).  In the STR
{ field of TOKEN, the enclosing apostrophes are removed and doubled
{ apostrophes within the original TEXT are replaced by a single apostrophe.
{ The token can hold a string up to 256 characters long.  If the STR_COMPLETE
{ field of TOKEN is FALSE, the string is too large to fit in TOKEN and must be
{ accessed via the TEXT_INDEX and TEXT_SIZE fields.  The following BNF
{ definitions illustrate the syntax of strings:
{
{    <clc$string_token> ::= ' [<string char>]... '
{    <string char> ::= <any ascii character except '> | ''
{
{   Comments are treated as spaces unless the scan option CLC$COMMENT_IS_TOKEN
{ is selected.  The following BNF definitions illustrate the syntax of
{ comments:
{
{    <clc$comment_token> ::= " [<comment char>]... ["]
{    <comment char> ::= <any ascii character except ">
{
{   The following BNF definitions illustrate the representation of the
{ delimiter and operator tokens:
{
{    <clc$semicolon_token> ::= ;
{    <clc$colon_token> ::= :
{    <clc$cybil_assign_token> ::= :=
{    <clc$left_parenthesis_token> ::= (
{    <clc$right_parenthesis_token> ::= )
{    <clc$comma_token> ::= ,
{    <clc$ellipsis_token> ::= .. [.]...
{    <clc$dot_token> ::= .
{    <clc$query_token> ::= ?
{    <clc$greater_than_token> ::= >
{    <clc$greater_equal_token> ::= >=
{    <clc$less_than_token> ::= <
{    <clc$less_equal_token> ::= <=
{    <clc$equal_token> ::= =
{    <clc$not_equal_token> ::= <>
{    <clc$assign_token> ::= =
{    <clc$concatenate_token> ::= //
{    <clc$exponentiate_token> ::= **
{    <clc$multiply_token> ::= *
{    <clc$divide_token> ::= /
{    <clc$add_token> ::= +
{    <clc$subtract_token> ::= -
{
{   The following are considered tokens only if the scan options
{ CLC$CLASSIFY_NAME_TOKEN and CLC$INTERNATIONAL_CHAR_IS_TOKEN are selected.
{
{    <clc$commercial_at_token> ::= @
{    <clc$left_bracket_token> ::= '['
{    <clc$reverse_slant_token> ::= \
{    <clc$right_bracket_token> ::= ']'
{    <clc$circumflex_token> ::= ^
{    <clc$grave_accent_token> ::= `
{    <clc$left_brace_token> ::= {
{    <clc$vertical_bar_token> ::= '|'
{    <clc$right_brace_token> ::= }
{    <clc$tilde_token> ::= ~
{
{   The following are considered tokens only if the scan options
{ CLC$CLASSIFY_NAME_TOKEN and CLC$SPECIAL_CHAR_IS_TOKEN are selected.
{
{    <clc$number_sign_token> ::= #
{    <clc$dollar_sign_token> ::= $
{    <clc$underscore_token> ::= _
{
{   Contiguous spaces are treated collectively as a clc$space_token.  The
{ horizontal tab (HT) character is treated identically to the space character.
{ Also, comments are normally treated as spaces.
{
{   Any character that does not begin a token previously described, is
{ returned as a clc$unknown_token.
{
{
{       CLP$EVALUATE_TOKEN (TEXT, SCAN_OPTIONS, INDEX, SPACES_PRECEDED_TOKEN,
{         TOKEN, STATUS)
{
{ TEXT: (input)  This parameter specifies the text to be scanned.
{
{ SCAN_OPTIONS: (input)  This parameter specifies options for the token
{       scanning process.  These options are:
{
{       clc$ignore_spaces_before_token:  if selected and a clc$space_token
{             would be returned, the space token is bypassed and the next
{             token is returned.  For a clc$signed_integer_token or a
{             clc$signed_real_tokens, this applies to spaces appearing between
{             the sign and the number as well.
{
{       clc$comment_is_token:  if selected this option defeats the normal
{             process of comments being considered as equivalent to spaces.
{
{       clc$classify_name_token:  if selected this option causes names to be
{             classified according to the BNF definitions above.
{
{       clc$cobol_name_is_token:  if selected this option allows a
{             clc$cobol_name_token to be returned.  This option is ignored
{             unless clc$classify_name_token is also selected.
{
{       clc$special_cybil_name_is_token:  if selected this option allows a
{             clc$special_cybil_name_token to be returned.  This option is
{             ignored unless clc$classify_name_token is also selected.
{
{       clc$international_char_is_token:  if selected allows the international
{             characters, normally considered to be part of a name, to be
{             returned as tokens (see above).  This option is ignored unless
{             clc$classify_name_token is also selected.
{
{       clc$special_char_is_token:  if selected allows the special characters,
{             normally considered to be part of a name, to be returned as
{             tokens (see above).  This option is ignored unless
{             clc$classify_name_token is also selected.
{
{ INDEX: (input, output)  This parameter specifies the next character within
{       TEXT to be scanned.
{
{ SPACES_PRECEDED_TOKEN: (output)  This parameter specifies whether spaces
{       were found before the returned token and is only meaningful if the
{       clc$ignore_spaces_before_token option is selected.
{
{ TOKEN: (output)  This parameter specifes the returned token.
{
{ STATUS: (output) This parameter specifies the request status.
{       CONDITIONS:
{
