A default tokenizer for handling syntaxes that are not explicitly handled elsewhere. It simply yields the given text as a single token.
Yield the given text as a single token.
# File lib/syntax.rb, line 10 def tokenize( text ) yield Token.new( text, :normal ) end