elasticsearch-carrot2

This action lists all available clustering algorithms. The returned identifiers can be used as a parameter to the clustering request.

Request

A request to list the available algorithms is a simple GET or POST request to /_algorithms URL.

Response

The response is a JSON object with an algorithms property which is a non-empty array of algorithm identifiers. The following example shows the algorithms available on this plugin instance. The default algorithm is the first one on the list.

$.get(window.ES_URL + "/_algorithms", function(response) {
    $("#list-of-algorithms").text(
              JSON.stringify(response, null, "  "));
});
            

This action performs a search query, fetches matching hits and clusters them on-the-fly.

The index and type URI segments implicitly bind the search request part of the message to a given index and document type, exactly as in the search request API.

A clustering request is a HTTP REST request, where the full set of parameters is supported via HTTP POST request with a JSON body. A limited subset of clustering functionality is also exposed via the HTTP GET method (and URL parameters).

Request (HTTP POST)

A HTTP POST request should contain a JSON object with the following properties.

search_request

required The search request to fetch documents to be clustered. This section follows exactly what the search DSL specifies, including all optional bells and whistles such as sorting, filtering, query DSL, highlighter, etc.

query_hint

required This attribute specifies query terms used to fetch the matching documents. This hint helps the clustering algorithm avoid trivial clusters that would be created around those frequent query terms. Typically the value of this parameter is identical to what the user typed in the search box. If possible, it should be pruned from any boolean or search-engine specific operators which could affect the clustering process. The query hint is obligatory but may be an empty string.

field_mapping

required Defines how to map actual fields of the documents matching the search_request to logical fields of the documents to be clustered. The value should be an object where keys indicate logical document fields and values are arrays with field source specifications (content of fields defined by these specifications is concatenated). For example this is a valid field mapping specification:

{
  "title":    [_source.subject],
  "content":  [_source.abstract, highlight.main],
  "language": [_source.lang]
}

Any of the following logical document field names are valid:

title

The title of the document.

content

The main body (content) of the document.

language

Optional language name for the title and content of a document. The language must be consistent with language components installed and available in Carrot². The list of supported languages is logged at startup. Note that unsupported languages will default to the main language of the request (language request parameter).

The field source specification defines where the value is taken from: the search hit's fields, stored document's content, or from the highlighter's output. The syntax of field source specification is as follows:

_source.{fieldname}

Defines a source document's field (top-level property of the json document). This will reparse the source document and fetch the appropriate value from there. Please see ES documentation for options to filter just specific fields from source.

highlight.{fieldname}

Defines a search hit's highlighted field. The highlighter output must also be configured properly in the search request (see field mapping example).

fields.{fieldname}

Defines a stored field. ! This syntax is deprecated and requires stored fields. See ES documentation for details and preferably use _source syntax.

algorithm

optional Defines which clustering algorithm should be used for clustering. Names of all built-in clustering algorithms are logged at startup and are also returned from the list algorithms request. The default algorithm is used, if value is empty or not provided.

max_hits

optional If set to a non-negative number, the clustering response will be limited to contain only a maximum of the given search hits. The clustering will still run on a full window of results returned by the original search request. This option may be useful to decrease the size of clustering response; in an extreme case of max_hits equal o 0, only clusters are returned and they can be used as facets (for refining the query, but without the immediate link to the search hits).

Note that clusters may still reference documents not present in the returned (trimmed) hits window.

create_ungrouped

optional If set to true, the documents not assigned to any cluster will form a synthetic group appended at the end of all clusters. This is effectively the "other topics" group created by Carrot² versions prior to 4.0.

language

optional Declares the default language for documents where field mapping for the logical language field is not declared (or empty). The default value of this field is English.

attributes

optional A JSON object with attributes overriding the default algorithm settings (per-query or runtime attributes in Carrot² parlance).

Request (HTTP GET)

A HTTP GET clustering request supports a superset of HTTP URI parameters defined by Elasticsearch's URI search request. All additional parameters correspond to those typically defined in the body of a clustering request sent via HTTP POST. Namely, the following parameters are supported by HTTP GET:

field_mapping_*

required This is a wildcard (a family) of parameters, each of which defines a logical field mapping, similar to field_mapping map described in the HTTP POST request. A field_mapping_title will specify the logical title's mapping, wheareas field_mapping_content will specify the logical document content mapping.

The value of the mapping parameter is a comma-separated list of mapping specifications, as described in the description of the POST request.

algorithm

optional Identical semantics to algorithm attribute described in HTTP POST request.

query_hint

optional Identical semantics to query_hint attribute described in HTTP POST request. For GET requests the query hint is optional; if not present, the q attribute is used as the default.

create_ungrouped

optional If set to true, the documents not assigned to any cluster will form a synthetic group appended at the end of all clusters. This is effectively the "other topics" group created by Carrot² versions prior to 4.0.

language

optional Declares the default language for documents where field mapping for the logical language field is not declared (or empty). The default value of this field is English.

An example HTTP GET clustering request is shown below, with the resulting clusters shown on the right-hand side panel.

var getUrl = window.ES_URL + "/test/test/_search_with_clusters?"
  + "q=data+mining&"
  + "size=100&"
  + "field_mapping_title=_source.title&"
  + "field_mapping_content=_source.content";

// Run HTTP GET via jquery and render cluster labels.
$.get(getUrl,
  function(response) {
    $("#cluster-httpget-result").text(
      dumpClusters([], response.clusters).join("\n"));
});
            

Response

The response format is identical to a plain search request response, with extra properties presented in the schematic output below.

{
  /* Typical search response fields. */
  "hits": { /* ... */ },

  /* Clustering response fields. */
  "clusters": [
    /* Each cluster is defined by the following. */
    {
      "id":    /* identifier */,
      "score": /* numeric score */,
      "label": /* primary cluster label */,
      "other_topics": /* if present, and true, this cluster groups
                         unrelated documents (no related topics) */,
      "phrases": [
        /* cluster label array, will include primary. */
      ],
      "documents": [
        /* This cluster's document ID references.
           May be undefined if this cluster holds sub-clusters only. */
      ],
      "clusters": [
        /* This cluster's subclusters (recursive objects of the same
           structure). May be undefined if this cluster holds documents only. */
      ],
    },
    /* ...more clusters */
  ],
  "info": {
    /* Additional information about the clustering: execution times,
       the algorithm used, etc. */
  }
}

Given the following function that recursively dumps clusters:

window.dumpClusters = function(arr, clusters, indent) {
  indent = indent ? indent : "";
  clusters.forEach(function(cluster) {
    arr.push(
        indent + cluster.label
        + (cluster.documents ? " [" + cluster.documents.length + " documents]"   : "")
        + (cluster.clusters  ? " [" + cluster.clusters.length  + " subclusters]" : ""));
    if (cluster.clusters) {
      dumpClusters(arr, cluster.clusters, indent + "  ");
    }
  });
  return arr;
}

We can dump all cluster labels of a clustering request with the following snippet of javascript:

var request = {
  "search_request": {
    "query": {"match" : { "content": "data mining" }},
    "size": 100
  },

  "max_hits": 0,
  "query_hint": "data mining",
  "field_mapping": {
    "title": ["_source.title"],
    "content": ["_source.content"]
  }
};

$.post(window.ES_URL + "/test/test/_search_with_clusters",
  JSON.stringify(request),
  function(response) {
    $("#cluster-list-result").text(
      dumpClusters([], response.clusters).join("\n"));
});
            

The output will vary depending on the choice of clustering algorithm (and particular documents that made it to the hit list if search is not deterministic). The following example shows the same result as above but uses the STC algorithm. We don't need every search hit here so we will omit them in the response.

var request = {
  "search_request": {
    "query": {"match" : { "content": "data mining" }},
    "size": 100
  },

  "max_hits": 0,
  "query_hint": "data mining",
  "field_mapping": {
    "title": ["_source.title"],
    "content": ["_source.content"]
  },
  "algorithm": "STC"
};

$.post(window.ES_URL + "/test/test/_search_with_clusters",
  JSON.stringify(request), function(response) {
    $("#cluster-list-result2").text(
      dumpClusters([], response.clusters).join("\n"));
});
            

A full response for a clustering request can look as shown below (note the difference in field mapping in this example).

var request = {
  "search_request": {
    "_source": [ "title", "content" ],
    "query": {"match" : { "content": "data mining" }},
    "size": 100
  },

  "max_hits": 0,
  "query_hint": "data mining",
  "field_mapping": {
    "title": ["_source.title"],
    "content": ["_source.content"]
  }
};

$.post(window.ES_URL + "/test/test/_search_with_clusters",
  JSON.stringify(request),
  function(response) {
    $("#simple-request-result").text(
      JSON.stringify(response, false, "  "));
});