A Callback is called once per token. More...
#include <zorba/tokenizer.h>
Public Types | |
typedef Tokenizer::size_type | size_type |
Public Member Functions | |
virtual void | operator() (char const *utf8_s, size_type utf8_len, size_type token_no, size_type sent_no, size_type para_no, void *payload=0)=0 |
This member-function is called once per token. | |
virtual | ~Callback () |
A Callback is called once per token.
This is only internally by Zorba. You do not need to derive from this class. The only thing you need to do is call the callback's operator()
once for each token you parse in tokenize()
.
Definition at line 73 of file tokenizer.h.
Definition at line 75 of file tokenizer.h.
virtual zorba::Tokenizer::Callback::~Callback | ( | ) | [virtual] |
virtual void zorba::Tokenizer::Callback::operator() | ( | char const * | utf8_s, |
size_type | utf8_len, | ||
size_type | token_no, | ||
size_type | sent_no, | ||
size_type | para_no, | ||
void * | payload = 0 |
||
) | [pure virtual] |
This member-function is called once per token.
utf8_s | The UTF-8 token string. It is not null-terminated. |
utf8_len | The number of bytes in the token string. |
token_no | The token number. Token numbers start at 0. |
sent_no | The sentence number. Sentence numbers start at 1. |
para_no | The paragraph number. Paragraph numbers start at 1. |
payload | Optional user-defined data. |