Uses of Class
org.apache.lucene.analysis.Token
-
Packages that use Token Package Description org.apache.lucene.analysis API and code to convert text into indexable/searchable tokens.org.apache.lucene.analysis.miscellaneous Miscellaneous TokenStreamsorg.apache.lucene.analysis.shingle Word n-gram filtersorg.apache.lucene.search.highlight The highlight package contains classes to provide "keyword in context" features typically used to highlight search terms in the text of results pages. -
-
Uses of Token in org.apache.lucene.analysis
Methods in org.apache.lucene.analysis that return Token Modifier and Type Method Description Token
Token. clone(char[] newTermBuffer, int newTermOffset, int newTermLength, int newStartOffset, int newEndOffset)
Makes a clone, but replaces the term buffer & start/end offset in the process.Token
Token. reinit(char[] newTermBuffer, int newTermOffset, int newTermLength, int newStartOffset, int newEndOffset)
Shorthand for callingclear()
,CharTermAttributeImpl.copyBuffer(char[], int, int)
,setStartOffset(int)
,setEndOffset(int)
setType(java.lang.String)
on Token.DEFAULT_TYPEToken
Token. reinit(char[] newTermBuffer, int newTermOffset, int newTermLength, int newStartOffset, int newEndOffset, String newType)
Shorthand for callingclear()
,CharTermAttributeImpl.copyBuffer(char[], int, int)
,setStartOffset(int)
,setEndOffset(int)
,setType(java.lang.String)
Token
Token. reinit(String newTerm, int newStartOffset, int newEndOffset)
Shorthand for callingclear()
,CharTermAttributeImpl.append(CharSequence)
,setStartOffset(int)
,setEndOffset(int)
setType(java.lang.String)
on Token.DEFAULT_TYPEToken
Token. reinit(String newTerm, int newTermOffset, int newTermLength, int newStartOffset, int newEndOffset)
Shorthand for callingclear()
,CharTermAttributeImpl.append(CharSequence, int, int)
,setStartOffset(int)
,setEndOffset(int)
setType(java.lang.String)
on Token.DEFAULT_TYPEToken
Token. reinit(String newTerm, int newTermOffset, int newTermLength, int newStartOffset, int newEndOffset, String newType)
Token
Token. reinit(String newTerm, int newStartOffset, int newEndOffset, String newType)
Shorthand for callingclear()
,CharTermAttributeImpl.append(CharSequence)
,setStartOffset(int)
,setEndOffset(int)
setType(java.lang.String)
Methods in org.apache.lucene.analysis with parameters of type Token Modifier and Type Method Description void
Token. reinit(Token prototype)
Copy the prototype token's fields into this one.void
Token. reinit(Token prototype, char[] newTermBuffer, int offset, int length)
Copy the prototype token's fields into this one, with a different term.void
Token. reinit(Token prototype, String newTerm)
Copy the prototype token's fields into this one, with a different term.Constructors in org.apache.lucene.analysis with parameters of type Token Constructor Description CannedTokenStream(Token... tokens)
-
Uses of Token in org.apache.lucene.analysis.miscellaneous
Methods in org.apache.lucene.analysis.miscellaneous that return Token Modifier and Type Method Description Token
SingleTokenTokenStream. getToken()
Token
PrefixAndSuffixAwareTokenFilter. updateInputToken(Token inputToken, Token lastPrefixToken)
Token
PrefixAndSuffixAwareTokenFilter. updateSuffixToken(Token suffixToken, Token lastInputToken)
Token
PrefixAwareTokenFilter. updateSuffixToken(Token suffixToken, Token lastPrefixToken)
The default implementation adds last prefix token end offset to the suffix token start and end offsets.Methods in org.apache.lucene.analysis.miscellaneous with parameters of type Token Modifier and Type Method Description void
SingleTokenTokenStream. setToken(Token token)
Token
PrefixAndSuffixAwareTokenFilter. updateInputToken(Token inputToken, Token lastPrefixToken)
Token
PrefixAndSuffixAwareTokenFilter. updateSuffixToken(Token suffixToken, Token lastInputToken)
Token
PrefixAwareTokenFilter. updateSuffixToken(Token suffixToken, Token lastPrefixToken)
The default implementation adds last prefix token end offset to the suffix token start and end offsets.Constructors in org.apache.lucene.analysis.miscellaneous with parameters of type Token Constructor Description SingleTokenTokenStream(Token token)
-
Uses of Token in org.apache.lucene.analysis.shingle
Methods in org.apache.lucene.analysis.shingle that return types with arguments of type Token Modifier and Type Method Description List<Token>
ShingleMatrixFilter.Matrix.Column.Row. getTokens()
Methods in org.apache.lucene.analysis.shingle with parameters of type Token Modifier and Type Method Description float
ShingleMatrixFilter. calculateShingleWeight(Token shingleToken, List<Token> shingle, int currentPermutationStartOffset, List<ShingleMatrixFilter.Matrix.Column.Row> currentPermutationRows, List<Token> currentPermuationTokens)
Deprecated.Evaluates the new shingle token weight.ShingleMatrixFilter.TokenPositioner
ShingleMatrixFilter.OneDimensionalNonWeightedTokenSettingsCodec. getTokenPositioner(Token token)
ShingleMatrixFilter.TokenPositioner
ShingleMatrixFilter.SimpleThreeDimensionalTokenSettingsCodec. getTokenPositioner(Token token)
abstract ShingleMatrixFilter.TokenPositioner
ShingleMatrixFilter.TokenSettingsCodec. getTokenPositioner(Token token)
Retrieves information on how aToken
is to be inserted to aShingleMatrixFilter.Matrix
.ShingleMatrixFilter.TokenPositioner
ShingleMatrixFilter.TwoDimensionalNonWeightedSynonymTokenSettingsCodec. getTokenPositioner(Token token)
float
ShingleMatrixFilter.OneDimensionalNonWeightedTokenSettingsCodec. getWeight(Token token)
float
ShingleMatrixFilter.SimpleThreeDimensionalTokenSettingsCodec. getWeight(Token token)
Returns a 32 bit float from the payload, or 1f it null.abstract float
ShingleMatrixFilter.TokenSettingsCodec. getWeight(Token token)
Have this method return 1f in order to 'disable' weights.float
ShingleMatrixFilter.TwoDimensionalNonWeightedSynonymTokenSettingsCodec. getWeight(Token token)
void
ShingleMatrixFilter.OneDimensionalNonWeightedTokenSettingsCodec. setTokenPositioner(Token token, ShingleMatrixFilter.TokenPositioner tokenPositioner)
void
ShingleMatrixFilter.SimpleThreeDimensionalTokenSettingsCodec. setTokenPositioner(Token token, ShingleMatrixFilter.TokenPositioner tokenPositioner)
Sets the TokenPositioner as token flags int value.abstract void
ShingleMatrixFilter.TokenSettingsCodec. setTokenPositioner(Token token, ShingleMatrixFilter.TokenPositioner tokenPositioner)
Sets information on how aToken
is to be inserted to aShingleMatrixFilter.Matrix
.void
ShingleMatrixFilter.TwoDimensionalNonWeightedSynonymTokenSettingsCodec. setTokenPositioner(Token token, ShingleMatrixFilter.TokenPositioner tokenPositioner)
void
ShingleMatrixFilter.OneDimensionalNonWeightedTokenSettingsCodec. setWeight(Token token, float weight)
void
ShingleMatrixFilter.SimpleThreeDimensionalTokenSettingsCodec. setWeight(Token token, float weight)
Stores a 32 bit float in the payload, or set it to null if 1f;abstract void
ShingleMatrixFilter.TokenSettingsCodec. setWeight(Token token, float weight)
Have this method do nothing in order to 'disable' weights.void
ShingleMatrixFilter.TwoDimensionalNonWeightedSynonymTokenSettingsCodec. setWeight(Token token, float weight)
void
ShingleMatrixFilter. updateToken(Token token, List<Token> shingle, int currentPermutationStartOffset, List<ShingleMatrixFilter.Matrix.Column.Row> currentPermutationRows, List<Token> currentPermuationTokens)
Deprecated.Final touch of a shingle token before it is passed on to the consumer from methodShingleMatrixFilter.incrementToken()
.Method parameters in org.apache.lucene.analysis.shingle with type arguments of type Token Modifier and Type Method Description float
ShingleMatrixFilter. calculateShingleWeight(Token shingleToken, List<Token> shingle, int currentPermutationStartOffset, List<ShingleMatrixFilter.Matrix.Column.Row> currentPermutationRows, List<Token> currentPermuationTokens)
Deprecated.Evaluates the new shingle token weight.void
ShingleMatrixFilter.Matrix.Column.Row. setTokens(List<Token> tokens)
void
ShingleMatrixFilter. updateToken(Token token, List<Token> shingle, int currentPermutationStartOffset, List<ShingleMatrixFilter.Matrix.Column.Row> currentPermutationRows, List<Token> currentPermuationTokens)
Deprecated.Final touch of a shingle token before it is passed on to the consumer from methodShingleMatrixFilter.incrementToken()
.Constructors in org.apache.lucene.analysis.shingle with parameters of type Token Constructor Description Column(Token token)
-
Uses of Token in org.apache.lucene.search.highlight
Methods in org.apache.lucene.search.highlight that return Token Modifier and Type Method Description Token
TokenGroup. getToken(int index)
-