Public Types | Public Member Functions
zorba::Tokenizer::Callback Class Reference

A Callback is called once per token. More...

#include <zorba/tokenizer.h>

List of all members.

Public Types

typedef Tokenizer::size_type size_type

Public Member Functions

virtual void operator() (char const *utf8_s, size_type utf8_len, size_type token_no, size_type sent_no, size_type para_no, void *payload=0)=0
 This member-function is called once per token.
virtual ~Callback ()

Detailed Description

A Callback is called once per token.

This is only internally by Zorba. You do not need to derive from this class. The only thing you need to do is call the callback's operator() once for each token you parse in tokenize().

Definition at line 73 of file tokenizer.h.


Member Typedef Documentation

Definition at line 75 of file tokenizer.h.


Constructor & Destructor Documentation


Member Function Documentation

virtual void zorba::Tokenizer::Callback::operator() ( char const *  utf8_s,
size_type  utf8_len,
size_type  token_no,
size_type  sent_no,
size_type  para_no,
void *  payload = 0 
) [pure virtual]

This member-function is called once per token.

Parameters:
utf8_sThe UTF-8 token string. It is not null-terminated.
utf8_lenThe number of bytes in the token string.
token_noThe token number. Token numbers start at 0.
sent_noThe sentence number. Sentence numbers start at 1.
para_noThe paragraph number. Paragraph numbers start at 1.
payloadOptional user-defined data.

The documentation for this class was generated from the following file:
blog comments powered by Disqus