A single token extracted by a tokenizer. It is simply the lexeme itself, decorated with a 'group' attribute to identify the type of the lexeme.
the type of the lexeme that was extracted.
the instruction associated with this token (:none, :region_open, or :region_close)
Create a new Token representing the given text, and belonging to the given group.
# File lib/syntax/common.rb, line 19 def initialize( text, group, instruction = :none ) super text @group = group @instruction = instruction end