Package org.carrot2.language
Class TokenTypeUtils
- java.lang.Object
-
- org.carrot2.language.TokenTypeUtils
-
-
Method Summary
All Methods Static Methods Concrete Methods Modifier and Type Method Description static boolean
isCommon(int flag)
Returntrue
ifTokenizer.TF_COMMON_WORD
is set.static boolean
isDocumentSeparator(int type)
Returnstrue
if the given type hasTokenizer.TF_SEPARATOR_DOCUMENT
set.static boolean
isFieldSeparator(int type)
Returnstrue
if the given type hasTokenizer.TF_SEPARATOR_FIELD
set.static boolean
isInQuery(int flag)
Returntrue
ifTokenizer.TF_QUERY_WORD
is set.static boolean
isTerminator(int type)
Returnstrue
if the given type hasTokenizer.TF_TERMINATOR
set.static int
maskType(int rawType)
Mask the given raw token type and leave just the token type bits.
-
-
-
Method Detail
-
maskType
public static int maskType(int rawType)
Mask the given raw token type and leave just the token type bits.
-
isDocumentSeparator
public static boolean isDocumentSeparator(int type)
Returnstrue
if the given type hasTokenizer.TF_SEPARATOR_DOCUMENT
set.
-
isFieldSeparator
public static boolean isFieldSeparator(int type)
Returnstrue
if the given type hasTokenizer.TF_SEPARATOR_FIELD
set.
-
isTerminator
public static boolean isTerminator(int type)
Returnstrue
if the given type hasTokenizer.TF_TERMINATOR
set.
-
isCommon
public static boolean isCommon(int flag)
Returntrue
ifTokenizer.TF_COMMON_WORD
is set.
-
isInQuery
public static boolean isInQuery(int flag)
Returntrue
ifTokenizer.TF_QUERY_WORD
is set.
-
-