Creates a Lexer for the source code base[begoffset..endoffset+1]. The last character, baseendoffset, must be null (0) or EOF (0x1A).
used for error messages
source code, must be terminated by a null (0) or EOF (0x1A) character
starting offset into base[]
the last offset to read into base[]
handle documentation comments
comments become TOK.comment's
See Implementation
Creates a Lexer for the source code base[begoffset..endoffset+1]. The last character, baseendoffset, must be null (0) or EOF (0x1A).