Package org.carrot2.text.preprocessing

Class PreprocessingContext

java.lang.Object

org.carrot2.text.preprocessing.PreprocessingContext

All Implemented Interfaces:

Closeable, AutoCloseable

public final class PreprocessingContext
extends Object
implements Closeable

Document preprocessing context provides low-level (usually integer-coded) data structures useful for further processing.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static class	PreprocessingContext.AllFields	Information about all fields processed for the input documents.
class	PreprocessingContext.AllLabels	Information about words and phrases that might be good cluster label candidates.
class	PreprocessingContext.AllPhrases	Information about all frequently appearing sequences of words found in the input documents.
class	PreprocessingContext.AllStems	Information about all unique stems found in the input documents.
class	PreprocessingContext.AllTokens	Information about all tokens of the input documents.
class	PreprocessingContext.AllWords	Information about all unique words found in the input documents.

Field Summary

Fields

Modifier and Type	Field	Description
PreprocessingContext.AllFields	allFields	Information about all fields processed for the input documents.
PreprocessingContext.AllLabels	allLabels	Information about words and phrases that might be good cluster label candidates.
PreprocessingContext.AllPhrases	allPhrases	Information about all frequently appearing sequences of words found in the input documents.
PreprocessingContext.AllStems	allStems	Information about all unique stems found in the input documents.
PreprocessingContext.AllTokens	allTokens	Information about all tokens of the input documents.
PreprocessingContext.AllWords	allWords	Information about all unique words found in the input documents.
int	documentCount	Count of documents processed by the tokenizer.
LanguageComponents	languageComponents	Language model to be used

Constructor Summary

Constructors

Constructor	Description
PreprocessingContext(LanguageComponents languageComponents)	Creates a preprocessing context for the provided documents and with the provided languageModel.

Method Summary

Modifier and Type	Method	Description
void	close()	This method should be invoked after all preprocessing contributors have been executed to release temporary data structures.
String	format(LabelFormatter formatter, int featureIndex)	Applies label formatter to a given word or phrase (depending on the feature index provided).
boolean	hasLabels()	Returns true if this context contains any label candidates.
boolean	hasWords()	Returns true if this context contains any words.
char[]	intern(MutableCharArray chs)	Return a unique char buffer representing a given character sequence.
static int[]	toFieldIndexes(byte b)	Convert the selected bits in a byte to an array of indexes.
String	toString()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

languageComponents

public final LanguageComponents languageComponents

Language model to be used

documentCount

public int documentCount

Count of documents processed by the tokenizer.

allTokens

public final PreprocessingContext.AllTokens allTokens

Information about all tokens of the input documents.

allFields

public final PreprocessingContext.AllFields allFields

Information about all fields processed for the input documents.

allWords

public final PreprocessingContext.AllWords allWords

Information about all unique words found in the input documents.

allStems

public final PreprocessingContext.AllStems allStems

Information about all unique stems found in the input documents.

allPhrases

public PreprocessingContext.AllPhrases allPhrases

Information about all frequently appearing sequences of words found in the input documents.

allLabels

public final PreprocessingContext.AllLabels allLabels

Information about words and phrases that might be good cluster label candidates.

Constructor Detail

PreprocessingContext

public PreprocessingContext(LanguageComponents languageComponents)

Creates a preprocessing context for the provided documents and with the provided languageModel.

Method Detail

hasWords

public boolean hasWords()

Returns true if this context contains any words.

hasLabels

public boolean hasLabels()

Returns true if this context contains any label candidates.

format

public String format(LabelFormatter formatter,
                     int featureIndex)

Applies label formatter to a given word or phrase (depending on the feature index provided).

toString

public String toString()

Overrides:

toString in class Object

toFieldIndexes

public static int[] toFieldIndexes(byte b)

Convert the selected bits in a byte to an array of indexes.

close

public void close()

This method should be invoked after all preprocessing contributors have been executed to release temporary data structures.

Specified by:

close in interface AutoCloseable

Specified by:

close in interface Closeable

intern

public char[] intern(MutableCharArray chs)

Return a unique char buffer representing a given character sequence.