Tokenizer

public protocol Tokenizer

A string tokenizer which tokenizes a string based on final productions of a context free grammar.

  • Tokenizes the given word and returns a sequence of possible tokens for each unit of the string

    For a grammar

    A -> a | A B
B -> a | B b

    and a string “ab”

    The tokenizer generates the tokenization

    [[a], [b]]

    Throws

    A syntax error if the word could not be tokenized according to rules of the recognized language

    Declaration

    Swift

    func tokenize(_ word: String) throws -> [[(terminal: Terminal, range: Range<String.Index>)]]

    Parameters

    word

    Word which should be tokenized

    Return Value

    Tokenization of the word