Enum TokenType

#[repr(u32)]
pub enum TokenType {
    Comment = 0,
    String = 1,
    Keyword = 2,
    Operator = 3,
    Number = 4,
    Function = 5,
    Decorator = 6,
    Type = 7,
    Namespace = 8,
    Bool = 9,
    Punctuation = 10,
    Escape = 11,
    Link = 12,
    Raw = 13,
    Label = 14,
    Ref = 15,
    Heading = 16,
    ListMarker = 17,
    ListTerm = 18,
    Delimiter = 19,
    Interpolated = 20,
    Error = 21,
    Text = 22,
    None = 23,
}

Very similar to typst_ide::Tag, but with convenience traits, and extensible because we want to further customize highlighting

Variants

Comment = 0

A comment token.

String = 1

A string token.

Keyword = 2

A keyword token.

Operator = 3

An operator token.

Number = 4

A number token.

Function = 5

A function token.

Decorator = 6

A decorator token.

Type = 7

A type token.

Namespace = 8

A namespace token.

Bool = 9

A boolean token.

Punctuation = 10

A punctuation token.

Escape = 11

An escape token.

Link = 12

A link token.

Raw = 13

A raw token.

Label = 14

A label token.

Ref = 15

A markup reference token.

Heading = 16

A heading token.

ListMarker = 17

A list marker token.

ListTerm = 18

A list term token.

Delimiter = 19

A delimiter token.

Interpolated = 20

An interpolated token.

Error = 21

An error token.

Text = 22

Any text in markup without a more specific token type, possible styled.

We perform styling (like bold and italics) via modifiers. That means everything that should receive styling needs to be a token so we can apply a modifier to it. This token type is mostly for that, since text should usually not be specially styled.

None = 23

A token that is not recognized by the lexer

Trait Implementations

impl Clone for TokenType

fn clone(&self) -> TokenType

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Default for TokenType

fn default() -> TokenType

Returns the “default value” for a type.

impl From<TokenType> for SemanticTokenType

fn from(token_type: TokenType) -> Self

Converts to this type from the input type.

impl IntoEnumIterator for TokenType

type Iterator = TokenTypeIter

fn iter() -> TokenTypeIter

impl PartialEq for TokenType

fn eq(&self, other: &TokenType) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl Copy for TokenType

impl Eq for TokenType

impl StructuralPartialEq for TokenType