standardize_constructs()

Find the closest thesaurus concepts for each span.

standardize_constructs(
    items: list,
    taxonomy: str = 'elsst',
    strategy: str = 'enriched',
    top: int = 5,
    embeddings: str = 'allmpnet',
    leaves_only: bool = False,
    cache: bool = True,
    threshold: float | None = None,
) -> dict

Source: theoryminer/harmonizer/standardize.py

The function turns each span (tidied and lowercased, no word removed) and each thesaurus concept into a vector, and scores each pair with the cosine. It returns the top best concepts of each span, one per concept, best first. The first call makes the thesaurus vectors and saves them (see cache). The function loads the embedding model.

Arguments

items

The spans to work on.

• Values: the relations from causenet() (the function reads “cause” and “effect”), or a plain list of span strings. Each unique span is used once.
• Default: none. Give the relations or the spans.

taxonomy

The thesaurus.

• Values: “elsst”, or the path of a SKOS .rdf or .rdf.gz file with the ELSST layout: rdf:Description blocks of type skos:Concept, with an English skos:prefLabel.
• Default "elsst": ELSST release 5 (3,436 concepts), which ships with the package.

strategy

The text that stands for each thesaurus concept.

• Values: “leaf” (the concept name), “path” (the path from the top of the thesaurus), “anchor” (the name, then the path), “context” (“<name> is related to <parents>”), “bracket” (the name, with the parents in brackets), “enriched” (the name, the other names, the definition and the path).
• Default “enriched”.

top

The number of concepts to return, best first.

• Values: a whole number, 1 or more.
• Default 5.

embeddings

The model that turns a text into a vector.

• Values: “allmpnet”, “bge_base”, “minilm”, or the name of another sentence-transformers model on Hugging Face. A new model makes new thesaurus vectors the first time.
• Default “allmpnet”.

leaves_only

Which thesaurus concepts can be a match.

• Values: True: only the most specific concepts (the leaves). False: all concepts, the broad ones too, for example TRUST and MOTIVATION.
• Default False: all concepts (the default since 0.2.0).

cache

Whether to keep the thesaurus vectors on disk.

• Values: True or False.
• Default True: read the vectors from ~/.cache/theoryminer/taxonomy (or from the folder in THEORYMINER_CACHE_DIR), and make and save them on the first call (about 3 minutes on a CPU). False makes them again in each session.

threshold

The lowest score that a concept needs to be a match. The score is a cosine.

• Values: None, or a number from -1 to 1 (in practice 0.4 to 0.7).
• Default None: keep the top concepts, however low their score.

Returns

{span: [match, …]}: for each span, a list of up to top dicts, best first, with the keys

The list is empty when no concept reaches threshold.

Raises

• ValueError: An unknown taxonomy (not “elsst” and not a file), or an unknown strategy.

Example

>>> matches = standardize_constructs(["loneliness", "social trust"], top=2)
>>> [(m["leaf"], m["score"]) for m in matches["social trust"]]
[('TRUST', 0.7604), ('INTERPERSONAL TRUST', 0.6876)]