TokenStore

An ID-addressable token view.

Unlike List, the first retained token need not have ID zero. Iteration is in lexical order, while get always uses the absolute Token.id.

Properties

Absolute ID of the first retained token.

abstract val lastTokenId: Int

Absolute ID of the last token, forcing production through EOF for lazy stores.

Number of tokens currently retained without forcing additional lexing.

abstract operator fun get(tokenId: Int): Token

Returns the token with absolute tokenId, forcing forward production when supported.

abstract operator fun iterator(): Iterator<Token>

Returns retained tokens in lexical order, forcing production through EOF for lazy stores.