Classes | Namespaces | Macros
tokenizer.h File Reference
#include <vector>
#include <zorba/config.h>
#include <zorba/locale.h>
#include <zorba/internal/unique_ptr.h>
#include <zorba/internal/ztd.h>
Include dependency graph for tokenizer.h:

Go to the source code of this file.

Classes

class  zorba::Tokenizer::Callback
 A Callback is called once per token. More...
 
struct  zorba::Tokenizer::Properties
 Various properties of this Tokenizer. More...
 
struct  zorba::Tokenizer::State
 A State contains inter-Tokenizer state, currently the current token, sentence, and paragraph numbers. More...
 
class  zorba::Tokenizer
 A Tokenizer breaks a string into a stream of word tokens. More...
 
class  zorba::TokenizerProvider
 A TokenizerProvider provides a Tokenizer for a given language. More...
 

Namespaces

namespace  zorba
 Copyright 2006-2011 The FLWOR Foundation.
 

Macros

#define ZORBA_TOKENIZER_API_H
 

Macro Definition Documentation

#define ZORBA_TOKENIZER_API_H

Definition at line 19 of file tokenizer.h.

blog comments powered by Disqus