Package org.apache.lucene.analysis
Class LowerCaseTokenizer
- java.lang.Object
-
- org.apache.lucene.util.AttributeSource
-
- org.apache.lucene.analysis.TokenStream
-
- org.apache.lucene.analysis.Tokenizer
-
- org.apache.lucene.analysis.CharTokenizer
-
- org.apache.lucene.analysis.LetterTokenizer
-
- org.apache.lucene.analysis.LowerCaseTokenizer
-
- All Implemented Interfaces:
Closeable
,AutoCloseable
public final class LowerCaseTokenizer extends LetterTokenizer
LowerCaseTokenizer performs the function of LetterTokenizer and LowerCaseFilter together. It divides text at non-letters and converts them to lower case. While it is functionally equivalent to the combination of LetterTokenizer and LowerCaseFilter, there is a performance advantage to doing the two tasks at once, hence this (redundant) implementation.Note: this does a decent job for most European languages, but does a terrible job for some Asian languages, where words are not separated by spaces.
You must specify the required
Version
compatibility when creatingLowerCaseTokenizer
:- As of 3.1,
CharTokenizer
uses an int based API to normalize and detect token characters. SeeCharTokenizer.isTokenChar(int)
andCharTokenizer.normalize(int)
for details.
-
-
Nested Class Summary
-
Nested classes/interfaces inherited from class org.apache.lucene.util.AttributeSource
AttributeSource.AttributeFactory, AttributeSource.State
-
-
Constructor Summary
Constructors Constructor Description LowerCaseTokenizer(Reader in)
Deprecated.useLowerCaseTokenizer(Version, Reader)
instead.LowerCaseTokenizer(AttributeSource.AttributeFactory factory, Reader in)
Deprecated.LowerCaseTokenizer(AttributeSource source, Reader in)
Deprecated.useLowerCaseTokenizer(Version, AttributeSource, Reader)
instead.LowerCaseTokenizer(Version matchVersion, Reader in)
Construct a new LowerCaseTokenizer.LowerCaseTokenizer(Version matchVersion, AttributeSource.AttributeFactory factory, Reader in)
Construct a new LowerCaseTokenizer using a givenAttributeSource.AttributeFactory
.LowerCaseTokenizer(Version matchVersion, AttributeSource source, Reader in)
Construct a new LowerCaseTokenizer using a givenAttributeSource
.
-
Method Summary
All Methods Instance Methods Concrete Methods Modifier and Type Method Description protected int
normalize(int c)
Converts char to lower caseCharacter.toLowerCase(int)
.-
Methods inherited from class org.apache.lucene.analysis.LetterTokenizer
isTokenChar
-
Methods inherited from class org.apache.lucene.analysis.CharTokenizer
end, incrementToken, isTokenChar, normalize, reset
-
Methods inherited from class org.apache.lucene.analysis.Tokenizer
close, correctOffset
-
Methods inherited from class org.apache.lucene.analysis.TokenStream
reset
-
Methods inherited from class org.apache.lucene.util.AttributeSource
addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, copyTo, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, reflectAsString, reflectWith, restoreState, toString
-
-
-
-
Constructor Detail
-
LowerCaseTokenizer
public LowerCaseTokenizer(Version matchVersion, Reader in)
Construct a new LowerCaseTokenizer.- Parameters:
matchVersion
- Lucene version to match See {@link above}in
- the input to split up into tokens
-
LowerCaseTokenizer
public LowerCaseTokenizer(Version matchVersion, AttributeSource source, Reader in)
Construct a new LowerCaseTokenizer using a givenAttributeSource
.
-
LowerCaseTokenizer
public LowerCaseTokenizer(Version matchVersion, AttributeSource.AttributeFactory factory, Reader in)
Construct a new LowerCaseTokenizer using a givenAttributeSource.AttributeFactory
.
-
LowerCaseTokenizer
@Deprecated public LowerCaseTokenizer(Reader in)
Deprecated.useLowerCaseTokenizer(Version, Reader)
instead. This will be removed in Lucene 4.0.Construct a new LowerCaseTokenizer.
-
LowerCaseTokenizer
@Deprecated public LowerCaseTokenizer(AttributeSource source, Reader in)
Deprecated.useLowerCaseTokenizer(Version, AttributeSource, Reader)
instead. This will be removed in Lucene 4.0.Construct a new LowerCaseTokenizer using a givenAttributeSource
.
-
LowerCaseTokenizer
@Deprecated public LowerCaseTokenizer(AttributeSource.AttributeFactory factory, Reader in)
Deprecated.useLowerCaseTokenizer(Version, AttributeSource.AttributeFactory, Reader)
instead. This will be removed in Lucene 4.0.Construct a new LowerCaseTokenizer using a givenAttributeSource.AttributeFactory
.
-
-
Method Detail
-
normalize
protected int normalize(int c)
Converts char to lower caseCharacter.toLowerCase(int)
.- Overrides:
normalize
in classCharTokenizer
-
-