lex
Creates a lazy token source without consulting an operator table.
Lexical failures are reported when the malformed construct is requested.
Throws
when requested input is lexically invalid or exceeds the configured retained-token limit
abstract fun lex(source: TextChunkSource, sourceId: String? = null, options: LexerOptions = LexerOptions()): LexedSource
Creates a lazy token source backed by synchronous text chunks.
Throws
when requested text is invalid, reading fails, or the retained-token limit is exceeded