Package org.apache.lucene.analysis
Class WhitespaceTokenizer
- java.lang.Object
-
- org.apache.lucene.util.AttributeSource
-
- org.apache.lucene.analysis.TokenStream
-
- org.apache.lucene.analysis.Tokenizer
-
- org.apache.lucene.analysis.CharTokenizer
-
- org.apache.lucene.analysis.WhitespaceTokenizer
-
- All Implemented Interfaces:
Closeable
,AutoCloseable
public final class WhitespaceTokenizer extends CharTokenizer
A WhitespaceTokenizer is a tokenizer that divides text at whitespace. Adjacent sequences of non-Whitespace characters form tokens.You must specify the required
Version
compatibility when creatingWhitespaceTokenizer
:- As of 3.1,
CharTokenizer
uses an int based API to normalize and detect token characters. SeeCharTokenizer.isTokenChar(int)
andCharTokenizer.normalize(int)
for details.
-
-
Nested Class Summary
-
Nested classes/interfaces inherited from class org.apache.lucene.util.AttributeSource
AttributeSource.AttributeFactory, AttributeSource.State
-
-
Constructor Summary
Constructors Constructor Description WhitespaceTokenizer(Reader in)
Deprecated.useWhitespaceTokenizer(Version, Reader)
instead.WhitespaceTokenizer(AttributeSource.AttributeFactory factory, Reader in)
Deprecated.WhitespaceTokenizer(AttributeSource source, Reader in)
Deprecated.WhitespaceTokenizer(Version matchVersion, Reader in)
Construct a new WhitespaceTokenizer.WhitespaceTokenizer(Version matchVersion, AttributeSource.AttributeFactory factory, Reader in)
Construct a new WhitespaceTokenizer using a givenAttributeSource.AttributeFactory
.WhitespaceTokenizer(Version matchVersion, AttributeSource source, Reader in)
Construct a new WhitespaceTokenizer using a givenAttributeSource
.
-
Method Summary
All Methods Instance Methods Concrete Methods Modifier and Type Method Description protected boolean
isTokenChar(int c)
Collects only characters which do not satisfyCharacter.isWhitespace(int)
.-
Methods inherited from class org.apache.lucene.analysis.CharTokenizer
end, incrementToken, isTokenChar, normalize, normalize, reset
-
Methods inherited from class org.apache.lucene.analysis.Tokenizer
close, correctOffset
-
Methods inherited from class org.apache.lucene.analysis.TokenStream
reset
-
Methods inherited from class org.apache.lucene.util.AttributeSource
addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, copyTo, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, reflectAsString, reflectWith, restoreState, toString
-
-
-
-
Constructor Detail
-
WhitespaceTokenizer
public WhitespaceTokenizer(Version matchVersion, Reader in)
Construct a new WhitespaceTokenizer. * @param matchVersion Lucene version to match See {@link above}- Parameters:
in
- the input to split up into tokens
-
WhitespaceTokenizer
public WhitespaceTokenizer(Version matchVersion, AttributeSource source, Reader in)
Construct a new WhitespaceTokenizer using a givenAttributeSource
.
-
WhitespaceTokenizer
public WhitespaceTokenizer(Version matchVersion, AttributeSource.AttributeFactory factory, Reader in)
Construct a new WhitespaceTokenizer using a givenAttributeSource.AttributeFactory
.
-
WhitespaceTokenizer
@Deprecated public WhitespaceTokenizer(Reader in)
Deprecated.useWhitespaceTokenizer(Version, Reader)
instead. This will be removed in Lucene 4.0.Construct a new WhitespaceTokenizer.
-
WhitespaceTokenizer
@Deprecated public WhitespaceTokenizer(AttributeSource source, Reader in)
Deprecated.useWhitespaceTokenizer(Version, AttributeSource, Reader)
instead. This will be removed in Lucene 4.0.Construct a new WhitespaceTokenizer using a givenAttributeSource
.
-
WhitespaceTokenizer
@Deprecated public WhitespaceTokenizer(AttributeSource.AttributeFactory factory, Reader in)
Deprecated.useWhitespaceTokenizer(Version, AttributeSource.AttributeFactory, Reader)
instead. This will be removed in Lucene 4.0.Construct a new WhitespaceTokenizer using a givenAttributeSource.AttributeFactory
.
-
-
Method Detail
-
isTokenChar
protected boolean isTokenChar(int c)
Collects only characters which do not satisfyCharacter.isWhitespace(int)
.- Overrides:
isTokenChar
in classCharTokenizer
-
-