org.carrot2.language.TokenTypeUtils

public final class TokenTypeUtils
extends Object

Utility methods for working with Tokenizer attributes.

Method Summary

Modifier and Type	Method	Description
`static boolean`	`isCommon(int flag)`	Return `true` if `Tokenizer.TF_COMMON_WORD` is set.
`static boolean`	`isDocumentSeparator(int type)`	Returns `true` if the given type has `Tokenizer.TF_SEPARATOR_DOCUMENT` set.
`static boolean`	`isFieldSeparator(int type)`	Returns `true` if the given type has `Tokenizer.TF_SEPARATOR_FIELD` set.
`static boolean`	`isInQuery(int flag)`	Return `true` if `Tokenizer.TF_QUERY_WORD` is set.
`static boolean`	`isTerminator(int type)`	Returns `true` if the given type has `Tokenizer.TF_TERMINATOR` set.
`static int`	`maskType(int rawType)`	Mask the given raw token type and leave just the token type bits.

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Class TokenTypeUtils