TokenTypeUtils (Carrot2, Public Java API v4.0.0 (org.carrot2:core))

java.lang.Object
- org.carrot2.language.TokenTypeUtils

```
public final class TokenTypeUtils
extends Object
```
Utility methods for working with Tokenizer attributes.

Method Summary

All Methods Static Methods Concrete Methods
Modifier and Type	Method	Description
`static boolean`	`isCommon(int flag)`	Return `true` if `Tokenizer.TF_COMMON_WORD` is set.
`static boolean`	`isDocumentSeparator(int type)`	Returns `true` if the given type has `Tokenizer.TF_SEPARATOR_DOCUMENT` set.
`static boolean`	`isFieldSeparator(int type)`	Returns `true` if the given type has `Tokenizer.TF_SEPARATOR_FIELD` set.
`static boolean`	`isInQuery(int flag)`	Return `true` if `Tokenizer.TF_QUERY_WORD` is set.
`static boolean`	`isTerminator(int type)`	Returns `true` if the given type has `Tokenizer.TF_TERMINATOR` set.
`static int`	`maskType(int rawType)`	Mask the given raw token type and leave just the token type bits.

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

- Method Detail
  - maskType
```
public static int maskType(int rawType)
```
    Mask the given raw token type and leave just the token type bits.
  - isDocumentSeparator
```
public static boolean isDocumentSeparator(int type)
```
    Returns true if the given type has Tokenizer.TF_SEPARATOR_DOCUMENT set.
  - isFieldSeparator
```
public static boolean isFieldSeparator(int type)
```
    Returns true if the given type has Tokenizer.TF_SEPARATOR_FIELD set.
  - isTerminator
```
public static boolean isTerminator(int type)
```
    Returns true if the given type has Tokenizer.TF_TERMINATOR set.
  - isCommon
```
public static boolean isCommon(int flag)
```
    Return true if Tokenizer.TF_COMMON_WORD is set.
  - isInQuery
```
public static boolean isInQuery(int flag)
```
    Return true if Tokenizer.TF_QUERY_WORD is set.