clusterer()

Group the spans by density, with HDBSCAN.

clusterer(
    items: list,
    embeddings: str = 'allmpnet',
    min_cluster_size: int = 2,
    min_samples: int = 2,
    umap: dict | None = None,
) -> dict

Source: theoryminer/harmonizer/cluster.py

The function turns each span into a vector and finds the dense regions: spans with many close neighbours. Each dense region is a group. A span in no dense region is noise (group -1); it stays in the result as its own central span. The function loads the embedding model.

Arguments

`items`

The spans to work on.

Values: the relations from causenet() (the function reads “cause” and “effect”), or a plain list of span strings. Each unique span is used once.
Default: none. Give the relations or the spans.

`embeddings`

The model that turns a text into a vector.

Values: “allmpnet” (sentence-transformers/all-mpnet-base-v2), “bge_base” (BAAI/bge-base-en-v1.5), “minilm” (sentence-transformers/all-MiniLM-L6-v2, small and fast), or the name of another sentence-transformers model on Hugging Face.
Default "allmpnet": the model of Paper 3. The default thresholds were calibrated for it.

`min_cluster_size`

The smallest group.

Values: a whole number, 2 or more.
Default 2: a pair of spans can be a group (the value of Paper 3).

`min_samples`

How many close neighbours a span needs to be in a dense region.

Values: a whole number, 1 or more. A higher value gives fewer groups and more noise.
Default 2 (the value of Paper 3).

`umap`

Whether to make the vectors shorter with UMAP before the clustering.

Values: None, or a dict of UMAP settings, for example dict(n_components=5, n_neighbors=15, min_dist=0.0). The metric is cosine and random_state is 42, unless the dict sets them.
Default None: cluster the full vectors.

Returns

{span: {“group”: id, “central”: span}}: for each span, the number of its group (0, 1, 2, …) or -1 for noise, and the central span of that group. A noise span is its own central span.

Raises

ValueError: min_cluster_size below 2 or min_samples below 1 (HDBSCAN checks them).

Example

>>> clusters = clusterer(cards)
>>> clusters["loneliness"], clusters["teacher support"]
({'group': 1, 'central': 'reducing loneliness'}, {'group': -1, 'central': 'teacher support'})

Keyboard shortcuts

theoryminer