Lucene++ - a full-featured, c++ search engine
API Documentation
ReaderPtr input
The text source for this Tokenizer.
Definition: Tokenizer.h:43
Tokenizer(const AttributeSourcePtr &source)
Construct a token stream processing the given input using the given AttributeSource.
#define LUCENE_CLASS(Name)
Definition: LuceneObject.h:24
virtual void reset(const ReaderPtr &input)
Reset the tokenizer to a new reader. Typically, an analyzer (in its reusableTokenStream method) will ...
virtual void reset()
Resets this stream to the beginning. This is an optional operation, so subclasses may or may not impl...
Tokenizer()
Construct a tokenizer with null input.
boost::shared_ptr< Reader > ReaderPtr
Definition: LuceneTypes.h:547
A TokenStream enumerates the sequence of tokens, either from Fields of a Document or from query text.
Definition: TokenStream.h:44
boost::shared_ptr< CharStream > CharStreamPtr
Definition: LuceneTypes.h:27
Definition: AbstractAllTermDocs.h:12
CharStreamPtr charStream
Definition: Tokenizer.h:48
boost::shared_ptr< AttributeSource > AttributeSourcePtr
Definition: LuceneTypes.h:520
A Tokenizer is a TokenStream whose input is a Reader.
Definition: Tokenizer.h:20
virtual void close()
By default, closes the input Reader.
Tokenizer(const ReaderPtr &input)
Construct a token stream processing the given input.
Tokenizer(const AttributeFactoryPtr &factory)
Construct a tokenizer with null input using the given AttributeFactory.
Tokenizer(const AttributeSourcePtr &source, const ReaderPtr &input)
Construct a token stream processing the given input using the given AttributeSource.
Tokenizer(const AttributeFactoryPtr &factory, const ReaderPtr &input)
Construct a token stream processing the given input using the given AttributeFactory.
boost::shared_ptr< AttributeFactory > AttributeFactoryPtr
Definition: LuceneTypes.h:519
virtual int32_t correctOffset(int32_t currentOff)
Return the corrected offset. If input is a CharStream subclass this method calls CharStream#correctOf...
clucene.sourceforge.net