Class Lexer

Separates arbitrary source text into a stream of SyntaxToken.

Inheritance
System.Object
Lexer
Namespace: DotLang.CodeAnalysis.Syntax
Assembly: DotLang.dll
Syntax
public sealed class Lexer

Constructors

| Improve this Doc View Source

Lexer(SourceText)

Declaration
public Lexer(SourceText sourceText)
Parameters
Type Name Description
SourceText sourceText
| Improve this Doc View Source

Lexer(String)

Declaration
public Lexer(string sourceText)
Parameters
Type Name Description
System.String sourceText

Methods

| Improve this Doc View Source

PeekToken()

Peeks at the next token in the source text but does not consume it. Invoke ReadToken() to consume the token.

Declaration
public SyntaxToken PeekToken()
Returns
Type Description
SyntaxToken
| Improve this Doc View Source

ReadToken()

Consumes the next token in the source text.

Declaration
public SyntaxToken ReadToken()
Returns
Type Description
SyntaxToken
| Improve this Doc View Source

ReadTokens(Boolean)

Reads all tokens in the source text, yielding as they are ready, until EndOfFileToken is seen.

Declaration
public IEnumerable<SyntaxToken> ReadTokens(bool promoteTrivia = false)
Parameters
Type Name Description
System.Boolean promoteTrivia

If true, leading trivia is detached from non-trivia tokens and yielded in order as if the triva was not trivia. For example:

ID  →  `=`  →  ID  →  EOF
↑       ↑      ↑
SPACE   SPACE  SPACE

Becomes:

SPACE → ID → SPACE → `=` → SPACE → ID → EOF
Returns
Type Description
System.Collections.Generic.IEnumerable<SyntaxToken>