module Token: sig end
This module defines the Camlp4 lexer type to be used in extensible
grammars (see module Grammar). It also provides some useful functions
to create lexers (this module should be renamed Glexer one day).
typepattern =string * string
tok_match below.exception Error of string
Lexer type
|
typelocation =int * int
typelocation_function =int -> location
type'alexer_func =char Stream.t -> 'a Stream.t * location_function
type 'a glexer = {
|
tok_func : |
|
tok_using : |
|
tok_removing : |
|
tok_match : |
|
tok_text : |
|
mutable tok_comm : |
tok_func is the main lexer function. See lexer_func
type above. This function may be created from a char stream parser
or for an ocamllex function using the functions below.tok_using is a function telling the lexer that the grammar
uses this token (pattern). The lexer can check that its constructor
is correct, and interpret some kind of tokens as keywords (to record
them in its tables). Called by EXTEND statements.tok_removing is a function telling the lexer that the
grammar does not uses the given token (pattern) any more. If the
lexer has a notion of "keywords", it can release it from its tables.
Called by DELETE_RULE statements.tok_match is a function taking a pattern and returning
a function matching a token against the pattern. Warning: for
efficency, write it as a function returning functions according
to the values of the pattern, not a function with two parameters.tok_text returns the name of some token pattern,
used in error messages.tok_comm if not None asks the lexer to record the
locations of the comments.val lexer_text : pattern -> stringtok_text function for lexersval default_match : pattern -> string * string -> stringtok_match function for lexers, appling to token type
(string * string)
Lexers from char stream parsers or ocamllex function
|
The functions below create lexer functions either from a char stream
parser or for an ocamllex function. With the returned function f,
the simplest Token.lexer can be written:
{ Token.tok_func = f;
Token.tok_using = (fun _ -> ());
Token.tok_removing = (fun _ -> ());
Token.tok_match = Token.default_match;
Token.tok_text = Token.lexer_text }
Note that a better tok_using function should check the used tokens
and raise Token.Error for incorrect ones. The other functions
tok_removing, tok_match and tok_text may have other implementations
as well.val lexer_func_of_parser : (char Stream.t -> 'a * location) -> 'a lexer_funcval lexer_func_of_ocamllex : (Lexing.lexbuf -> 'a) -> 'a lexer_funcocamllexval make_stream_and_location : (unit -> 'a * location) -> 'a Stream.t * location_function
Useful functions
|
val eval_char : string -> charval eval_string : string -> stringFailure if
bad backslash sequence found; Token.eval_char (Char.escaped c)
returns c and Token.eval_string (String.escaped s) returns s