parser-impl/it.unibo.tuprolog.parser.tokens/Token

Token

data class Token(val id: Int, val kind: TokenKind, val channel: TokenChannel, val span: SourceSpan, val payload: TokenPayload? = null) : Representable

A lossless lexical token.

Raw spelling is obtained from span through the owning it.unibo.tuprolog.parser.sources.LexedSource. toRepresentation instead returns normalized content and is not a substitute for source extraction.

Constructors

constructor(id: Int, kind: TokenKind, channel: TokenChannel, span: SourceSpan, payload: TokenPayload? = null)

Properties

val channel: TokenChannel

whether this token participates in grammar parsing

val id: Int

absolute, monotonically increasing token identifier

val kind: TokenKind

operator-independent lexical category

val payload: TokenPayload?

decoded or variable token data, when applicable

val span: SourceSpan

end-exclusive range in the complete source

Functions

toRepresentation

open override fun toRepresentation(): String

Returns the token's fixed spelling or normalized payload representation.