Heavy and lightweight components

Initialization of heavy components (LanguageComponents) may take significant time. Load them once and then reuse for all subsequent clustering calls. Heavy components are thread-safe and can be reused or shared between threads.

Lightweight components, such as clustering algorithm instances, are cheap to instantiate so you can create a throw-away instance on-demand for each clustering call.

Clustering

This section discusses code from the E01_ClusteringBasics.java example. It shows just the key elements required to process a stream of documents in English, without any parameter or language resource tuning.

First, let's load the heavy components: the default resources for the English language. The loaded LanguageComponents instance is thread-safe and should be reused for any subsequent calls to clustering algorithms.

// Our documents are in English so we load appropriate language resources.
// This call can be heavy and an instance of LanguageComponents should be
// created once and reused across different clustering calls.
LanguageComponents languageComponents = LanguageComponents.loader().load().language("English");

Now it's time to create the lightweight component: an instance of a clustering algorithm. Let's pick the Lingo algorithm explicitly:

LingoClusteringAlgorithm algorithm = new LingoClusteringAlgorithm();

Once we have the heavy and lightweight components initialized, we can assemble the input for clustering: a stream of Document instances. Each document must implement a single method that presents its clusterable text fields to the algorithm:

void visitFields(BiConsumer<String, String> fieldConsumer);

In this example we will use hardcoded values from an array available in the ExamplesData class:

static final String[][] DOCUMENTS_DATA_MINING = {
  {
    "http://en.wikipedia.org/wiki/Data_mining",
    "Data mining -" + " Wikipedia, the free " + "encyclopedia",
    "Article about knowledge-discovery in databases (KDD), the practice "
        + "of automatically "
        + "searching large stores of data for patterns."
  },

Because the field visitor interface is a single method, it can be implemented using a closure. We convert the above data array into document instances dynamically using Java streams. Note how we only expose the title and the snippet fields, the URL is omitted because it is not really clusterable text content.

// Create a stream of "documents" for clustering.
// Each such document provides text content fields to a visitor.
Stream<Document> documentStream =
    Arrays.stream(ExamplesData.DOCUMENTS_DATA_MINING)
        .map(
            fields ->
                (fieldVisitor) -> {
                  fieldVisitor.accept("title", fields[1]);
                  fieldVisitor.accept("content", fields[2]);
                });

Everything is now ready to call the clustering algorithm and consume the result. Here we just print the top-level cluster labels and document counts to the console:

// Perform clustering.
List<Cluster<Document>> clusters;
clusters = algorithm.cluster(documentStream, languageComponents);

// Print cluster labels and a document count in each top-level cluster.
for (Cluster<Document> c : clusters) {
  String label = String.join("; ", c.getLabels());
  System.out.println(label + ", documents: " + c.getDocuments().size());
}

When executed, this example should result in this output:

Data Mining Software, documents: 13
Knowledge Discovery, documents: 13
Data Mining Process, documents: 12
Data Mining Applications, documents: 11
Data Mining Techniques, documents: 11