Types of dictionaries

Carrot² supports two types of dictionaries:

Word filters

The set of words, commonly known as stop words, that should be ignored during clustering. Typically, word filters would contain function words, such as of or between.

If a word is filtered out, it will not have influence on the clusters the algorithm creates, but may still appear in cluster labels, such as University of Washington, to aid readability.

Word filters are stored in the JSON syntax for each language separately in a file named lang.word-filters.json, where lang is the name of the language, for example english.word-filters.json. Location of the dictionary files is application-specific.

Label filters

The set of words or phrases that must not be used as cluster labels. You can use label filters to suppress offensive or non-informative labels, such as Official website.

Label filters are stored in the JSON syntax for each language separately in a file named lang.label-filters.json, where lang is the name of the language, for example english.label-filters.json. Location of the dictionary files is application-specific.

Location of dictionary files

Carrot² Document Clustering Server (REST API) loads dictionaries from an explicit non-configurable location provided below. The Java API does not dictate the location of these resources, but comes with reasonable defaults and a way to load them from a custom location.

Syntax

Both the word- and label filter dictionaries use a common syntax based on matchers. An example word or label filter dictionary is shown below.

{
  "exact": [
    "Literal case-sensitive match"
  ],
  "glob": [
    "starts with phrase *",
    "* contains words *"
  ],
  "regexp": [
    "(?).+BadLabel.+",
    "(?)^[0-9]\s*.*"
  ]
}

A matcher entry is a description of how to determine whether a word or label should be filtered out. The exact, glob and regexp properties are optional and can contain an array of string entries for the specific matcher types, described in the following sections. If a word or a label matches any matcher of any type, it will be filtered out.

{
  "exact": [
    "DevOps",
    "Windows 2000"
  ]
}

{
  "glob": [
    "more information",
    "more information *",
    "* about *"
  ]
}

{
  "regexp": [
    "Windows 9[58]",
    "(?)^[0-9]\s*.*"
  ]
}

Per-request dictionaries

In some contexts, it may be desirable to apply additional label or word exclusions for a specific clustering request. One example is allowing the users to interactively tune the exclusions directory while observing the results in real time.

Providing per-request dictionaries is possible both in the Java API, where the ephemeral dictionaries is a better term, and in the REST API. In both cases, the available set of filters and matcher syntax is the same as described in this section. The per-request dictionaries are added on top of the existing dictionary files.

Please note that using per-request dictionaries will incur an extra performance cost associated with parsing the provided dictionary entries. In case of interactive label exclusion tuning, you may want to transfer the newly created dictionary entries to the main dictionary file, so that the exclusions apply for all further clustering requests.