theoryminer

theoryminer turns a set of research papers into a causal map.

You give it PDF files. It finds the sentences that claim a cause and an effect, and it extracts each cause → effect pair. Then it brings together the different names that authors use for the same construct. The result is an interactive map. Each arrow leads back to the sentences that claim it, with the page and the DOI of each paper.

The pipeline

PDF ──harvest()──► sentences ──causenet()──► relations (cause span → effect span)
                                                  │
              ┌───────────────────────────────────┼───────────────────────────┐
              ▼                                   ▼                           ▼
standardize_constructs()            pairwise_grouping() / clusterer()      (spans)
span → top-k thesaurus concepts     spans → groups of similar spans
                                                  │
                                  ┌───────────────┴───────────────┐
                                  ▼                               ▼
                        standardize_groups()               label_groups()
                        group → thesaurus concept          group → label from its own words
                                  └───────────────┬───────────────┘
                                                  ▼
                        causal_map() ──► draw_map()  the interactive page
                                     └─► save_map()  CSV tables and GraphML

Two more stages feed the pipeline with papers. extract_references() finds the DOIs that your papers cite. download_papers() downloads the open-access PDFs of those DOIs. Together they run a snowball search: the papers of one round give the references of the next round.

What each step gives you

The rules behind the results

• Nothing is deleted. A step that drops a sentence, or hides an edge, marks it and records the reason. The drop log and the hidden edges are part of the result.
• A span stays as the author wrote it. The only change is a repair of the tokenizer output. That repair removes no word.
• Every result leads back to its source. An edge knows its relations. A relation knows its sentence, its page and its paper.
• A thesaurus is optional. The package includes the ELSST thesaurus. Every step that uses it has a twin that needs no thesaurus.

Status

Extraction, the grouping and naming of constructs, and the causal map work. The evaluation against a theory specification that people made by hand is planned.

Where to start

1. Install the package.
2. Run the quick start on the sample papers.
3. Read the guide for the step that you need.

The tutorial notebook runs every function and every setting on 7 sample papers.

Install

theoryminer needs Python 3.10 or newer.

From PyPI

pip install theoryminer

The install includes UMAP (for clusterer(..., umap={...})) and tmsr-doi-downloader, an extra download source. On Python 3.10, pip leaves out tmsr-doi-downloader. The default download sources work on every version.

The newest code

To get the code on GitHub before the next release:

pip install "theoryminer @ git+https://github.com/rasoulnorouzi/theoryminer.git"

In a notebook

In Google Colab or Jupyter, put this line at the start of the first cell:

%pip install theoryminer

The tutorial notebook does this for you.

The models

The package downloads two models on first use. It keeps them in the Hugging Face cache, so later runs use the local copy.

Without a network, set the environment variable HF_HUB_OFFLINE=1. The models must already be in the cache.

The GPU

You do not need to set anything. causenet() uses device="auto": an NVIDIA GPU when PyTorch sees one, else the GPU of an Apple Silicon Mac, else the CPU.

The caches

When the package cannot write a cache folder, the functions still work. They compute the vectors again in the next session.

Quick start

This page runs the full pipeline on the 7 sample papers in the folder sample_files/ of the repository. The numbers below come from that run on a laptop CPU.

To get the sample papers, clone the repository:

git clone https://github.com/rasoulnorouzi/theoryminer.git
cd theoryminer

The full run

from theoryminer import (harvest, causenet, clusterer, label_groups,
                         causal_map, draw_map, save_map)

data = harvest("sample_files")                  # 1. PDFs -> clean sentences
relations = causenet(data["sentences"])         # 2. sentences -> cause -> effect pairs
groups = clusterer(relations)                   # 3. group the spans that name the same construct
names = label_groups(groups)                    # 4. name each group from its own words
cmap = causal_map(relations, names)             # 5. nodes and edges
draw_map(cmap)                                  # 6. the interactive page
save_map(cmap, "outputs/my_map")                # 7. CSV tables and a GraphML file

What each line gives

1. harvest() reads the 7 PDFs. It keeps 2,531 sentences and drops 1,865 rows: table rows, the reference sections, headings, page furniture. Each dropped row keeps its reason in data["dropped"]. Each sentence has the DOI and the title of its paper.
2. causenet() finds 490 causal sentences and 945 relations. It takes about 2 minutes on a CPU. The first call downloads the model (about 430 MB).
3. clusterer() puts the 1,197 different spans into 242 groups. A span that fits no group stays alone.
4. label_groups() names each group with the words that its members repeat.
5. causal_map() makes 50 nodes and 35 shown edges. It hides 849 weak edges and keeps them in cmap["hidden"], each with its reason. By default an edge needs 2 sentences to show.
6. draw_map() shows the page in a notebook. In a script it writes causal_map.html.
7. save_map() writes nodes.csv, edges.csv, hidden.csv and map.graphml.

Look at one relation

Each relation leads back to its source:

relation = relations[0]
print(relation["cause"], "->", relation["effect"])
print(relation["sentence"])
print(relation["doc_id"], "page", relation["page"], relation["doi"])

Next steps

• Use your own PDFs: harvest("path/to/folder").
• Use the ELSST thesaurus for the names: see Group and name constructs.
• Find and download the papers that your papers cite: see A snowball round.
• See every setting with its default: Settings and defaults.

Read PDFs

harvest() reads one PDF, or every PDF in a folder, and returns clean sentences and a drop log.

from theoryminer import harvest

data = harvest("sample_files")                       # every PDF in the folder
data = harvest("book.pdf", pages="12-540")           # one PDF, some pages only

The result has two lists:

What a sentence holds

The cleaning steps

The steps run in this order. You can change the list with steps=.

To switch off one step, give the list without it:

from theoryminer import DEFAULT_STEPS

steps = [step for step in DEFAULT_STEPS if step != "strip_stats"]
data = harvest("sample_files", steps=steps)

You can also add a function of your own. It takes the list of row dicts and returns it. The tutorial notebook, section 1.3b, shows how.

Why a row is dropped

harvest() never deletes a row. A step marks it as dropped and writes the reason.

Flags

A flag marks a doubtful sentence. The sentence stays in data["sentences"].

To keep the flagged sentences away from the model:

sentences = [s for s in data["sentences"] if not s["flag"]]

Pages

Page 1 is the first page, as in a PDF viewer.

harvest("book.pdf", pages=12)               # one page
harvest("book.pdf", pages="12-540")         # a range
harvest("book.pdf", pages="3,7,9-12")       # a list of pages and ranges
harvest("book.pdf", skip_pages=[4, 5])      # every page except 4 and 5

For a folder, the same pages apply to every PDF.

The page cache

Text extraction is the slow part. harvest() writes the text of each PDF to <pdf>.pages.json beside the PDF. The next run reads that file. Use cache=False to read every PDF again and write no file.

A bad file

For one PDF, a missing or broken file raises a ValueError. For a folder, a bad file does not stop the run. The drop log gets one row for the file, with page 0 and the reason not_pdf or unreadable_pdf. harvest() reads the folder only, not its subfolders.

See the full list of settings in the reference of harvest().

DOIs and references

Two functions read DOIs from PDFs:

The DOI of each paper

You do not need extract_dois() for the map. harvest() already puts the DOI and the title on each sentence (find_dois=True), and causenet() copies them to each relation.

Use extract_dois() when you want the list on its own:

from theoryminer import extract_dois

rows = extract_dois("sample_files", save="outputs")      # writes outputs/dois.csv

Each row has doc_id, file, doi, source, title, page, status and note.

The DOI comes from the most certain source, in this order:

1. downloads.csv, the log that download_papers() writes into the folder.
2. The file name of a PDF that download_papers() saved, because it is made from the DOI.
3. The subject or keywords field of the PDF metadata.
4. The first DOI on page 1, else on page 2. Page 1 is sometimes a cover page.

The source column tells which rule gave the DOI. On 21 papers with a known DOI, this order found 20.

The title is the largest text on page 1 that has at least 4 words. A shorter large text is usually a journal name, such as “PLOS ONE”. When no text passes, the title comes from the PDF metadata.

The DOIs in the references

from theoryminer import extract_references

refs = extract_references("sample_files", save="outputs")   # writes outputs/references.csv

Each row has doi, n_seeds (how many of your papers cite it) and cited_by (which papers). The most cited DOIs come first.

The function repairs a DOI that the PDF cut at a line end, for example 10.1007/s1096 4-015-0354-5. It leaves out the DOI of the paper itself and the DOIs of its parts (figures, tables). On the 7 sample papers it found 215 DOIs, and 214 of them exist at doi.org.

A reference that prints no DOI cannot be found. The journal of one sample paper prints no DOIs in its references, so that paper gives 0 DOIs.

Give the result to download_papers() to get the PDFs.

Download papers

download_papers() downloads the PDFs of a list of DOIs into a folder. It gives one status row for each DOI.

from theoryminer import extract_references, download_papers

refs = extract_references("seed_papers")
report = download_papers(refs, "round_2", email="you@your-uni.edu")

dois can be the output of extract_references(), a list of DOI strings, or the path of a CSV file with a doi column.

How it finds a PDF

The function asks OpenAlex for the licence of each paper first. With licences=["cc-by"] it downloads only the CC BY papers, and it never contacts the other papers’ sites.

Then it tries the sources in turn and stops at the first real PDF:

A link can give the PDF or a web page. For a page, the function follows its citation_pdf_url tag, the tag that publishers write for Google Scholar. The function keeps a file only when it is a real PDF. It waits 1 second between two requests to the same site, and it obeys robots.txt.

Give email when you can. OpenAlex and Crossref answer faster with it, and Unpaywall needs it.

The publisher plugins

A plugin knows how one publisher builds its PDF links, in the same way as a Zotero translator. It turns a DOI into the direct link. The package has plugins for Springer (also BMC and Palgrave) and Frontiers. Each plugin is a few lines in theoryminer/download/publishers.py.

A plugin is useful only when the publisher gives the PDF to a plain request. Wiley, Taylor & Francis, SAGE, MDPI, Annual Reviews and Emerald give their PDFs only to a browser that passes a bot check, so they have no plugin. Their open papers often come through the other sources, as a copy in a repository or in PubMed Central.

The result

The function also writes downloads.csv into the folder. harvest() reads that file to know the DOI of each PDF. A PDF that is already in the folder is not downloaded again, so you can run the function again after a stop.

What to expect

We tested the sources on 300 DOIs: 100 references of the sample papers and 200 random articles in psychology and the social sciences.

A closed paper has no legal free copy, so no open source can give it. Many publishers (Wiley, Elsevier, SAGE, Taylor & Francis, MDPI) also block programs, even for their open papers. These two cases are most of the misses. The note column tells you the reason for each DOI.

The benchmark is in benchmarks/downloader/. Run it again with python benchmarks/downloader/run_benchmark.py --label my_run --email you@your-uni.edu.

What the downloader does not do

The downloader gets open-access copies only. It does not log in to a publisher site, and it does not use a browser. A paper that is only behind a paywall stays not_found. Get such a paper through your library, and put the PDF into the folder with the others. harvest() reads it, and it finds the DOI in the PDF metadata or on the first two pages.

tmsr-doi-downloader

Versions 0.2.x used the package tmsr-doi-downloader. Since 0.3.0 the default sources replace it: on the 100 references it added one paper and doubled the run time. It is still available as an extra source: sources=SOURCES + ["tmsr-doi-downloader"].

Find causal claims

causenet() finds the cause → effect pairs in the sentences.

from theoryminer import harvest, causenet

data = harvest("sample_files")
relations = causenet(data["sentences"])

The model rasoultilburg/SocioCausaNet reads one sentence at a time. It decides whether the sentence is causal, marks the cause spans and the effect spans, and links each cause to each effect. One sentence can give more than one relation.

A relation

A span is the words of the sentence, as the author wrote them. The function repairs the spaces that the tokenizer changes, and it removes no word. On the sample papers, 97 % of the relations have both spans word for word in their sentence.

You can also give a plain list of strings. Then doc_id is "input" and page is 0.

Settings

Speed

A GPU is much faster. In Google Colab, choose a GPU runtime. The function writes no file, so keep the result in a variable. To keep it for a later session, save it as JSON:

import json

with open("outputs/relations.json", "w", encoding="utf-8") as fh:
    json.dump(relations, fh)

Leave out the flagged sentences

harvest() flags doubtful sentences but keeps them. To keep them away from the model:

sentences = [s for s in data["sentences"] if not s["flag"]]
relations = causenet(sentences)

See every setting in the reference of causenet().

Group and name constructs

Authors use many names for one construct: “loneliness”, “feelings of loneliness”, “perceived isolation”. Before the map, the spans that mean the same thing must become one node. The package has three routes.

Each route gives the names that causal_map() needs.

from theoryminer import clusterer, label_groups, causal_map

groups = clusterer(relations)
names = label_groups(groups)
cmap = causal_map(relations, names)

All the functions read the cause and effect spans of the relations. You can also give them a plain list of span strings. They use the sentence model all-mpnet-base-v2 by default. Every similarity is a cosine.

Make groups

The result of both is a dict: {span: {"group": id, "central": span}}. The central span is the member that is most like the other members.

Use linkage="average", the default since 0.2.0. With "single", one shared neighbour joins two groups, and long chains form. When A is like B and B is like C, A and C go into one group, also when they are far apart.

Warning: pairwise_grouping() computes the distance between every pair of spans. Its memory grows with the square of the number of spans: about 10 GB for 50,000 spans. For a large corpus, use clusterer().

Name the groups without a thesaurus

label_groups() names each group with the words that its members repeat and that the other groups do not use (class-based TF-IDF). It needs no model and no thesaurus.

Name with the ELSST thesaurus

ELSST (the European Language Social Science Thesaurus, release 5) is part of the package. Its licence is CC BY-SA 4.0.

standardize_constructs() gives each span the 5 best concepts. standardize_groups() gives each group the 5 best concepts. A group of spans gets one name from all its members.

The first run computes the vectors of the thesaurus and caches them in ~/.cache/theoryminer/taxonomy. Later runs load them.

Which route to use

• Start with clusterer() and label_groups(). They need no thesaurus, and the names use the words of your papers.
• Use standardize_groups() when you want names from a shared vocabulary, so that you can compare your map with other work.
• Use standardize_constructs() to check how one span maps to the thesaurus.

The causal map

Three functions turn the relations and the names into a map:

from theoryminer import causal_map, draw_map, save_map

cmap = causal_map(relations, names)
draw_map(cmap)
save_map(cmap, "outputs/my_map")

names is the output of label_groups(), standardize_groups() or standardize_constructs(). See Group and name constructs.

Nodes and edges

A node is a construct: a named group, or a thesaurus concept. An edge goes from a cause node to an effect node. It counts the papers, the sentences and the relations that claim it, and it keeps the rel_ids of those relations. So each edge leads back to its sentences.

A node has a role, from its shown edges: cause only, effect only or cause and effect.

Which edges show

causal_map() hides a weak edge, but it deletes nothing. A hidden edge goes to cmap["hidden"] with its reason.

A hidden edge has the drop_reason below_min_papers or below_min_sentences.

Two-way pairs

When papers claim both A → B and B → A, both edges stay. min_share (default 0.67) decides how the pair is read. When one side has at least 2/3 of the papers, that side is dominant and the other side is minority. Otherwise both sides are contested. A minority edge shows as a dashed line.

The page

draw_map() writes one HTML file that works offline. In a notebook it shows in the cell. In a script it writes causal_map.html, or the file that you give with save=.

On the page you can:

• hover over a node or an edge, and click an edge to read all its sentences, with the DOI of each paper (else its title, else its file name);
• search for a construct: the page shows only the matches and their neighbours;
• filter by paper, by role, by two-way pairs, with the min papers and min sentences sliders, and with the self-loop switch;
• drag, zoom and reset;
• save a picture as PNG, JPG or SVG: the whole map, the current view, or one node with its neighbours.

The page receives every edge, also the hidden ones. Its sliders start at the values of causal_map(), so you can show a hidden edge on the page.

Warning: a large map opens slowly. The page computes the layout in the browser. In our test, a map with 1,000 shown nodes took about 78 seconds to open. A map with 3,000 nodes did not open in 10 minutes. Keep the map small with min_papers and min_sentences. Most readers cannot read more than a few hundred nodes anyway.

The files

save_map(cmap, folder) writes:

In the CSV files, a list (members, doc_ids, rel_ids) is one text, joined by |.

A snowball round

A snowball search starts from seed papers and follows their references. One round has four steps:

1. Find the DOIs in the references of the seed papers.
2. Download the PDFs of those DOIs.
3. Read the new papers and draw their map.
4. Find the references of the new papers: the DOIs for the next round.

from theoryminer import (extract_references, download_papers, harvest, causenet,
                         pairwise_grouping, label_groups, causal_map, draw_map)

# 1. The references of the seed paper
refs = extract_references("sample_files/schutz_2025_loneliness_social_support.pdf")

# 2. Their open-access PDFs, CC BY only
report = download_papers(refs, "outputs/round_2", licences=["cc-by"], email="you@your-uni.edu")

# 3. The map of the new papers
data = harvest("outputs/round_2")
relations = causenet(data["sentences"])
names = label_groups(pairwise_grouping(relations))
cmap = causal_map(relations, names)
draw_map(cmap, save="outputs/round_2_map.html")

# 4. The DOIs for round 3
next_refs = extract_references("outputs/round_2", save="outputs/round_3_dois.csv")

What the tutorial run gave

licences=["cc-by"] keeps only the papers whose text may be shown and shared. Leave it out to download every open-access paper.

The DOIs that more papers cite come first in the output of extract_references(). They are a good start for the next round.

Notes

• The download writes downloads.csv into the folder. harvest() reads it, so each sentence knows the DOI of its paper, also when the PDF does not print its DOI.
• A stopped download can go on: a PDF that is already in the folder is not downloaded again.
• See Download papers for the sources and the limits.

Reference

One page for each public function. The text comes from the docstring in the code.

Read PDFs

Download papers

Find causal claims

Group and name constructs

Draw the causal map

harvest()

Read one PDF, or every PDF in a folder, and return the clean sentences and a drop log.

harvest(
    source: str,
    pages: str | int | list[int] | None = None,
    skip_pages: str | int | list[int] | None = None,
    steps: list[str | Callable] | None = None,
    min_words: int = 4,
    max_words: int = 80,
    cache: bool = True,
    find_dois: bool = True,
) -> dict

Source: theoryminer/prepare/harvest.py

The function reads the text of each page, runs the cleaning steps in order, and splits the text into sentences. A step never deletes a row: it marks the row as dropped and gives the reason. A doubtful sentence keeps its row and gets a flag. For one PDF, a bad file raises a ValueError. For a folder, a bad file does not stop the run: the drop log gets one row for the file, with page 0 and the reason “not_pdf” or “unreadable_pdf”. The function looks in the folder only, not in its subfolders. It writes one file per PDF, the page cache (see cache).

Arguments

source

The PDF file, or the folder of PDF files, to read.

• Values: the path of one PDF file, or the path of a folder.
• Default: none. Give a path.

pages

The pages to read. Page 1 is the first page, as in a PDF viewer.

• Values: one page (12), a text (“12-540” or “3,7,9-12”), a list of pages ([3, 5, 9]), or None. For a folder, the same pages apply to every PDF.
• Default None: read every page.

skip_pages

The pages to leave out, after pages has selected its pages.

• Values: the same forms as pages.
• Default None: leave out no page.

steps

The cleaning steps, in the order in which they run.

• Values: a list of step names from STEPS, of your own functions, or of both. A function of your own takes a list of row dicts and returns a list of row dicts. To switch off one step, give DEFAULT_STEPS without that name.
• Default None: run DEFAULT_STEPS, the 10 steps in this order: fix_unicode, strip_markdown, drop_running_heads, drop_junk_lines, join_paragraphs, split_sentences, strip_citations, strip_stats, drop_junk_sentences, filter_length.

min_words

The fewest words that a sentence can have (the step filter_length).

• Values: a whole number, 1 or more.
• Default 4: a sentence with 3 words or fewer is dropped as “too_short”.

max_words

The most words that a sentence can have (the step filter_length).

• Values: a whole number, min_words or more.
• Default 80: a sentence with 81 words or more is dropped as “too_long”.

cache

Whether to keep the text of the pages on disk.

• Values: True or False.
• Default True: write <pdf>.pages.json beside each PDF, and read it on the next run. The next run then skips the slow text extraction. False reads every PDF again and writes no file.

find_dois

Whether to find the DOI and the title of each paper, for each sentence.

• Values: True or False.
• Default True: find them, as extract_dois() does. The DOI comes from the most certain source: the download log (downloads.csv in the folder), the file name of a download_papers() PDF, the PDF metadata, then the first DOI on pages 1 and 2. causenet() copies them to each relation, and draw_map() shows them. False skips this work and gives “” for both.

Returns

A dict with two lists:

• “sentences”: one dict per kept sentence, with the keys

• “dropped”: one dict per dropped row, with the keys doc_id, page, section, drop_reason and text.

Raises

• ValueError: The source does not exist, the folder has no PDF, or the one PDF cannot be read. Also for a wrong setting: a page selection that is not a page, an unknown step name, min_words below 1, or max_words below min_words.

Example

>>> data = harvest("sample_files")
>>> len(data["sentences"]), data["sentences"][0]["sent_id"]
(2531, 'd000p0000s000')

extract_dois()

Find the DOI and the title of each paper in one PDF, or in every PDF in a folder.

extract_dois(
    source: str,
    save: str | None = None,
    first_pages: int = FIRST_PAGES,
    cache: bool = True,
) -> list[dict]

Source: theoryminer/prepare/dois.py

The DOI rule: the first DOI on page 1, else the first DOI on the next pages, up to first_pages. The title rule: the largest font on page 1 with at least MIN_TITLE_WORDS words, else the title in the PDF metadata. A bad file does not stop the run: it gets a row with the status “not_pdf” or “unreadable_pdf”. The function reads the same page cache as harvest(). It writes a CSV file only when you give save.

Arguments

source

The PDF file, or the folder of PDF files, to read.

• Values: the path of one PDF file, or the path of a folder. The function looks in the folder only, not in its subfolders.
• Default: none. Give a path.

save

Where to write the rows as a CSV file.

• Values: None, a path that ends with “.csv”, or a folder (then <folder>/dois.csv). A missing folder is made.
• Default None: write no file.

first_pages

The last page on which to look for the DOI.

• Values: a whole number, 1 or more. A larger value can find a DOI that is not on the first pages, but also the DOI of a cited paper in the reference list.
• Default 2: look on page 1, then on page 2 (page 1 is sometimes a cover page).

cache

Whether to use the page cache <pdf>.pages.json beside each PDF.

• Values: True or False.
• Default True: read the cache when it exists, else read the PDF and write the cache. False reads every PDF again and writes no cache.

Returns

A list of dicts, one per file, in file-name order, with the keys

Raises

• ValueError: The source does not exist, or the folder has no PDF.

Example

>>> rows = extract_dois("sample_files/alrabai_2021_autonomy_supportive_teaching.pdf")
>>> rows[0]["doi"], rows[0]["status"]
('10.3389/fpsyg.2021.728657', 'found')

extract_references()

Find the DOIs that the seed papers cite, in one PDF or in every PDF in a folder.

extract_references(
    source: str,
    save: str | None = None,
    cache: bool = True,
) -> list[dict]

Source: theoryminer/prepare/dois.py

The function reads every page of each seed paper and finds each DOI. It repairs a DOI that the PDF cut at a line end (“10.1007/s1096 4-015-0354-5”). It leaves out the DOI of the seed itself and the DOIs of its parts (figures, tables). A reference that prints no DOI cannot be found. A bad file does not stop the run: the summary names it. It writes a CSV file only when you give save.

Arguments

source

The seed paper, or the folder of seed papers, to read.

• Values: the path of one PDF file, or the path of a folder. The function looks in the folder only, not in its subfolders.
• Default: none. Give a path.

save

Where to write the rows as a CSV file, with the columns doi, n_seeds, cited_by. A download tool can read its “doi” column.

• Values: None, a path that ends with “.csv”, or a folder (then <folder>/references.csv). A missing folder is made.
• Default None: write no file.

cache

Whether to use the page cache <pdf>.pages.json beside each PDF.

• Values: True or False.
• Default True: read the cache when it exists, else read the PDF and write the cache. False reads every PDF again and writes no cache.

Returns

A list of dicts, one per unique reference DOI, with the keys

The DOIs that most seeds cite come first.

Raises

• ValueError: The source does not exist, or the folder has no PDF.

Example

>>> refs = extract_references("sample_files")
>>> refs[0]["doi"], refs[0]["n_seeds"]
('10.1037/0003-066x.55.1.68', 2)

download_papers()

Download the open-access PDFs of the DOIs into a folder. Return one status row per DOI.

download_papers(
    dois: list | str,
    output_dir: str,
    licences: list[str] | None = None,
    email: str | None = None,
    sources: list[str] | None = None,
    respect_robots: bool = True,
    core_api_key: str | None = None,
    serpapi_key: str | None = None,
    save: str | None = None,
) -> list[dict]

Source: theoryminer/download/papers.py

The function asks OpenAlex for the licence of each paper. With licences, it downloads only the papers with one of those licences. Then it tries the sources in turn until one gives a real PDF. A link can give the PDF or a page; for a page, the function follows its citation_pdf_url meta tag once. It waits 1 second between two requests to the same site. A PDF that is already in the folder is not downloaded again, so a stopped run can go on. A failed DOI does not stop the run. The function writes the PDFs and the log downloads.csv into output_dir (the log keeps the rows of earlier runs, and harvest() reads it to know the DOI of each file), and a report file when you give save. It gets open-access copies only: it does not log in to a publisher site.

Arguments

dois

The papers to download.

• Values: the rows of extract_references() (dicts with a “doi” key), a list of DOI strings, or the path of a CSV file with a “doi” column. A text is always read as the path of a CSV file, not as one DOI.
• Default: none. Give the DOIs.

output_dir

The folder for the PDFs.

• Values: the path of a folder. A missing folder is made. The file name is the DOI with “_” for each unsafe character.
• Default: none. Give a folder.

licences

The licences to download.

• Values: None, or a list of OpenAlex licences: “cc-by”, “cc-by-sa”, “cc-by-nc”, “cc-by-nd”, “cc-by-nc-sa”, “cc-by-nc-nd”, “public-domain”, “other-oa”, “publisher-specific-oa”, “none” and “unknown”.
• Default None: download every paper that has an open-access copy, whatever its licence.

email

Your email address. OpenAlex and Crossref answer faster with it (“polite pool”), and Unpaywall needs it.

• Values: None, or a real email address (Unpaywall refuses a made-up address).
• Default None: no email. The function skips Unpaywall, and tmsr-doi-downloader uses the environment variable UNPAYWALL_EMAIL when it is set.

sources

The sources to try, in this order.

• Values: a list of one or more of “openalex” (the PDF links and repository pages of the open-access copies in OpenAlex), “publisher” (the publisher’s direct PDF link, from a per-publisher plugin), “pmc” (the PDF in the PubMed Central open-access data of NCBI), “europe_pmc” (the free PDF links that Europe PMC lists), “unpaywall” (the PDF links of Unpaywall; it needs email), “semantic_scholar” (the open-access PDF link of Semantic Scholar), “crossref” (the PDF links that the publisher gives to Crossref), “preprint” (the open OSF preprint of the paper, from the Crossref relation), “landing_page” (the page that doi.org opens, and its PDF link), “tmsr-doi-downloader” (the package tmsr-doi-downloader; Python 3.11 or newer).
• Default None: the nine open sources above, from “openalex” to “landing_page”, in that order. The default leaves out “tmsr-doi-downloader”. Add it with this setting when the open sources miss a paper.

respect_robots

Whether to obey the robots.txt file of each site.

• Values: True or False. Set False only when you have the right to download from the sites, for example by an agreement of your institution. The wait between requests stays. tmsr-doi-downloader always obeys robots.txt.
• Default True: skip every link that robots.txt does not allow.

core_api_key

A free key from core.ac.uk, for CORE in tmsr-doi-downloader.

• Values: None, or the key as text.
• Default None: use the environment variable CORE_API_KEY when it is set, else skip CORE.

serpapi_key

A key of SerpAPI (a paid service), for Google Scholar in tmsr-doi-downloader.

• Values: None, or the key as text.
• Default None: use the environment variable SERPAPI_KEY when it is set, else skip Google Scholar.

save

Where to write the report rows as a CSV file.

• Values: None, a path that ends with “.csv”, or a folder (then <folder>/downloads.csv).
• Default None: write no report file.

Returns

A list of dicts, one per DOI, with the keys

Raises

• ValueError: when sources is empty or holds a name that is not one of the sources above.

Example

>>> refs = extract_references("sample_files")
>>> report = download_papers(refs, "outputs/round_2", licences=["cc-by"])
>>> report[0]["status"], report[0]["source"]
('downloaded', 'openalex')

causenet()

Find the cause -> effect pairs in the sentences.

causenet(
    sentences: list,
    threshold: float = 0.8,
    mode: str = 'neural',
    decision: str = 'span_only',
    batch_size: int = 64,
    device: str = 'auto',
    avoid_ambiguous: bool = True,
) -> list[dict]

Source: theoryminer/causenet.py

The model rasoultilburg/SocioCausaNet reads one sentence at a time. It decides whether the sentence is causal, marks the cause spans and the effect spans, and links each cause to each effect. The function then repairs each span with tidy_span(), which removes no word, and gives each relation the path back to its sentence. The first call loads the model (about 440 MB, downloaded on first use). The function writes no file. Keep the result in a variable: a large corpus takes long on a CPU.

Arguments

sentences

The sentences to read.

• Values: the rows of harvest()[“sentences”] (the function reads “clean”, “sent_id”, “doc_id” and “page”), or a plain list of strings (then doc_id is “input”, page is 0, and sent_id is “s0000”, “s0001”, …).
• Default: none. Give the sentences.

threshold

The score that a cause -> effect pair needs to become a relation.

• Values: a number from 0 to 1. A lower value gives more relations and more errors.
• Default 0.8: keep a pair with a score of 0.8 or more.

mode

How the model links each cause to each effect.

• Values: “neural”: the model scores every cause -> effect pair, and threshold decides. “auto”: when a sentence has only one cause or only one effect, every pair is kept without a score; else as “neural”.
• Default “neural”.

decision

The rule that makes a sentence causal.

• Values: “span_only”: the sentence has a cause span and an effect span. “cls+span”: also the sentence classifier must say causal (the default before 0.2.0). “cls_only”: only the sentence classifier decides.
• Default “span_only”.

batch_size

The number of sentences in one model call.

• Values: a whole number, 1 or more. On a GPU a larger value is often faster. Lower it after a GPU “out of memory” error.
• Default 64.

device

Where the model runs.

• Values: “auto”, “cpu”, “cuda” (an NVIDIA GPU) or “mps” (the GPU of an Apple Silicon Mac). If the model fails on an Apple GPU, it moves to the CPU and the run goes on.
• Default "auto": an NVIDIA GPU when torch sees one, else the Apple GPU, else the CPU.

avoid_ambiguous

Whether to skip a relation whose cause or effect only points to another sentence: “this”, “it”, “such things” (the rule is harmonizer.spans.is_vague_span).

• Values: True or False.
• Default True: skip these relations. The summary prints their count and examples. False keeps every relation.

Returns

A list of dicts, one per relation, with the keys

Raises

• ValueError: An unknown mode, decision or device, a threshold outside 0 to 1, a batch_size below 1, or a device that torch cannot see (“cuda” without an NVIDIA GPU).

Example

>>> relations = causenet(["Social support increases feelings of security and well-being."])
>>> [(r["cause"], r["effect"]) for r in relations]
[('social support', 'feelings of security and well-being')]

pairwise_grouping()

Group the spans that are similar, pair by pair.

pairwise_grouping(
    items: list,
    embeddings: str = 'allmpnet',
    threshold: float = 0.69,
    linkage: str = 'average',
) -> dict

Source: theoryminer/harmonizer/pairwise.py

The function turns each span into a vector and joins spans whose cosine reaches threshold. linkage decides when a span joins a group that has more members. Each group gets a central span: the member closest to all the others. Every span is in a group, so there is no noise. The function loads the embedding model.

Arguments

items

The spans to work on.

• Values: the relations from causenet() (the function reads “cause” and “effect”), or a plain list of span strings. Each unique span is used once.
• Default: none. Give the relations or the spans.

embeddings

The model that turns a text into a vector.

• Values: “allmpnet” (sentence-transformers/all-mpnet-base-v2), “bge_base” (BAAI/bge-base-en-v1.5), “minilm” (sentence-transformers/all-MiniLM-L6-v2, small and fast), or the name of another sentence-transformers model on Hugging Face.
• Default "allmpnet": the model of Paper 3. The default thresholds were calibrated for it.

threshold

The cosine that a span needs to join a group.

• Values: a number from -1 to 1 (in practice 0.5 to 0.9). A higher value gives smaller, stricter groups.
• Default 0.69: the value of Paper 3, calibrated for “allmpnet” with “single” linkage.

linkage

How a span is compared with a group.

• Values: “average”: the mean cosine to all the members reaches threshold. “complete”: the cosine to every member reaches threshold. “single”: the cosine to one member reaches threshold. This can chain unrelated spans into one large group (Paper 3’s method).
• Default “average” (since 0.2.0).

Returns

{span: {“group”: id, “central”: span}}: for each span, the number of its group (0, 1, 2, …) and the central span of that group.

Raises

• ValueError: An unknown linkage.

Example

>>> pairwise_grouping(["social support", "strong social support", "teacher support"])
{'social support': {'group': 0, 'central': 'social support'}, 'strong social support': {'group': 0, 'central': 'social support'}, 'teacher support': {'group': 1, 'central': 'teacher support'}}

clusterer()

Group the spans by density, with HDBSCAN.

clusterer(
    items: list,
    embeddings: str = 'allmpnet',
    min_cluster_size: int = 2,
    min_samples: int = 2,
    umap: dict | None = None,
) -> dict

Source: theoryminer/harmonizer/cluster.py

The function turns each span into a vector and finds the dense regions: spans with many close neighbours. Each dense region is a group. A span in no dense region is noise (group -1); it stays in the result as its own central span. The function loads the embedding model.

Arguments

items

The spans to work on.

• Values: the relations from causenet() (the function reads “cause” and “effect”), or a plain list of span strings. Each unique span is used once.
• Default: none. Give the relations or the spans.

embeddings

The model that turns a text into a vector.

• Values: “allmpnet” (sentence-transformers/all-mpnet-base-v2), “bge_base” (BAAI/bge-base-en-v1.5), “minilm” (sentence-transformers/all-MiniLM-L6-v2, small and fast), or the name of another sentence-transformers model on Hugging Face.
• Default "allmpnet": the model of Paper 3. The default thresholds were calibrated for it.

min_cluster_size

The smallest group.

• Values: a whole number, 2 or more.
• Default 2: a pair of spans can be a group (the value of Paper 3).

min_samples

How many close neighbours a span needs to be in a dense region.

• Values: a whole number, 1 or more. A higher value gives fewer groups and more noise.
• Default 2 (the value of Paper 3).

umap

Whether to make the vectors shorter with UMAP before the clustering.

• Values: None, or a dict of UMAP settings, for example dict(n_components=5, n_neighbors=15, min_dist=0.0). The metric is cosine and random_state is 42, unless the dict sets them.
• Default None: cluster the full vectors.

Returns

{span: {“group”: id, “central”: span}}: for each span, the number of its group (0, 1, 2, …) or -1 for noise, and the central span of that group. A noise span is its own central span.

Raises

• ValueError: min_cluster_size below 2 or min_samples below 1 (HDBSCAN checks them).

Example

>>> clusters = clusterer(cards)
>>> clusters["loneliness"], clusters["teacher support"]
({'group': 1, 'central': 'reducing loneliness'}, {'group': -1, 'central': 'teacher support'})

label_groups()

Give each group of spans a name. No thesaurus, no model.

label_groups(
    groups: dict,
    by: str = 'keywords',
    top: int = 3,
) -> dict

Source: theoryminer/harmonizer/labels.py

The keywords of a group are the words that its members repeat and that the other groups do not use (class-based TF-IDF):

score(word) = (count in the group / words in the group)
              * log(1 + groups / groups that hold the word)

Function words (“the”, “of”, …) never enter a name. A group with one member takes the member as its name. The spans are not changed. The function reads no file.

Arguments

groups

The groups to name.

• Values: the result of pairwise_grouping() or clusterer(): {span: {“group”: id, “central”: span}}.
• Default: none. Give the groups.

by

How the name is made.

• Values: “keywords”: the top keywords, joined by a space. “medoid”: the central span. “both”: the central span, “ | “, then the keywords.
• Default “keywords” (since 0.2.0).

top

The number of keywords for each group.

• Values: a whole number, 1 or more.
• Default 3.

Returns

{key: group}: one dict per group. The key is the group id, or the span text for a noise span. Each group has the keys

Raises

• ValueError: An unknown by.

Example

>>> labelled = label_groups(clusterer(cards))
>>> labelled[1]["size"], labelled[1]["name"]
(5, 'loneliness reducing alleviate')

standardize_constructs()

Find the closest thesaurus concepts for each span.

standardize_constructs(
    items: list,
    taxonomy: str = 'elsst',
    strategy: str = 'enriched',
    top: int = 5,
    embeddings: str = 'allmpnet',
    leaves_only: bool = False,
    cache: bool = True,
    threshold: float | None = None,
) -> dict

Source: theoryminer/harmonizer/standardize.py

The function turns each span (tidied and lowercased, no word removed) and each thesaurus concept into a vector, and scores each pair with the cosine. It returns the top best concepts of each span, one per concept, best first. The first call makes the thesaurus vectors and saves them (see cache). The function loads the embedding model.

Arguments

items

The spans to work on.

• Values: the relations from causenet() (the function reads “cause” and “effect”), or a plain list of span strings. Each unique span is used once.
• Default: none. Give the relations or the spans.

taxonomy

The thesaurus.

• Values: “elsst”, or the path of a SKOS .rdf or .rdf.gz file with the ELSST layout: rdf:Description blocks of type skos:Concept, with an English skos:prefLabel.
• Default "elsst": ELSST release 5 (3,436 concepts), which ships with the package.

strategy

The text that stands for each thesaurus concept.

• Values: “leaf” (the concept name), “path” (the path from the top of the thesaurus), “anchor” (the name, then the path), “context” (“<name> is related to <parents>”), “bracket” (the name, with the parents in brackets), “enriched” (the name, the other names, the definition and the path).
• Default “enriched”.

top

The number of concepts to return, best first.

• Values: a whole number, 1 or more.
• Default 5.

embeddings

The model that turns a text into a vector.

• Values: “allmpnet”, “bge_base”, “minilm”, or the name of another sentence-transformers model on Hugging Face. A new model makes new thesaurus vectors the first time.
• Default “allmpnet”.

leaves_only

Which thesaurus concepts can be a match.

• Values: True: only the most specific concepts (the leaves). False: all concepts, the broad ones too, for example TRUST and MOTIVATION.
• Default False: all concepts (the default since 0.2.0).

cache

Whether to keep the thesaurus vectors on disk.

• Values: True or False.
• Default True: read the vectors from ~/.cache/theoryminer/taxonomy (or from the folder in THEORYMINER_CACHE_DIR), and make and save them on the first call (about 3 minutes on a CPU). False makes them again in each session.

threshold

The lowest score that a concept needs to be a match. The score is a cosine.

• Values: None, or a number from -1 to 1 (in practice 0.4 to 0.7).
• Default None: keep the top concepts, however low their score.

Returns

{span: [match, …]}: for each span, a list of up to top dicts, best first, with the keys

The list is empty when no concept reaches threshold.

Raises

• ValueError: An unknown taxonomy (not “elsst” and not a file), or an unknown strategy.

Example

>>> matches = standardize_constructs(["loneliness", "social trust"], top=2)
>>> [(m["leaf"], m["score"]) for m in matches["social trust"]]
[('TRUST', 0.7604), ('INTERPERSONAL TRUST', 0.6876)]

standardize_groups()

Find the closest thesaurus concepts for each group of spans.

standardize_groups(
    groups: dict,
    name_by: str = 'mean_sim',
    taxonomy: str = 'elsst',
    strategy: str = 'enriched',
    top: int = 5,
    embeddings: str = 'allmpnet',
    leaves_only: bool = False,
    cache: bool = True,
    threshold: float | None = None,
) -> dict

Source: theoryminer/harmonizer/standardize.py

The function scores each group against each thesaurus concept, with the rule in name_by, and returns the top best concepts of each group. A noise span (group -1) is a group of its own. The first call makes the thesaurus vectors and saves them (see cache). The function loads the embedding model.

Arguments

groups

The groups to name.

• Values: the result of pairwise_grouping() or clusterer(): {span: {“group”: id, “central”: span}}.
• Default: none. Give the groups.

name_by

How a group gets one score for each concept.

• Values: “mean_sim”: score each member, then take the mean. A mixed group scores low. “centroid”: take the mean vector of the members, then score it. It always ranks the concepts as “mean_sim” does, with higher scores. “medoid”: score the central span only.
• Default “mean_sim”.

taxonomy

The thesaurus.

• Values: “elsst”, or the path of a SKOS .rdf or .rdf.gz file with the ELSST layout: rdf:Description blocks of type skos:Concept, with an English skos:prefLabel.
• Default "elsst": ELSST release 5 (3,436 concepts), which ships with the package.

strategy

The text that stands for each thesaurus concept.

• Values: “leaf” (the concept name), “path” (the path from the top of the thesaurus), “anchor” (the name, then the path), “context” (“<name> is related to <parents>”), “bracket” (the name, with the parents in brackets), “enriched” (the name, the other names, the definition and the path).
• Default “enriched”.

top

The number of concepts to return, best first.

• Values: a whole number, 1 or more.
• Default 5.

embeddings

The model that turns a text into a vector.

• Values: “allmpnet”, “bge_base”, “minilm”, or the name of another sentence-transformers model on Hugging Face. A new model makes new thesaurus vectors the first time.
• Default “allmpnet”.

leaves_only

Which thesaurus concepts can be a match.

• Values: True: only the most specific concepts (the leaves). False: all concepts, the broad ones too, for example TRUST and MOTIVATION.
• Default False: all concepts (the default since 0.2.0).

cache

Whether to keep the thesaurus vectors on disk.

• Values: True or False.
• Default True: read the vectors from ~/.cache/theoryminer/taxonomy (or from the folder in THEORYMINER_CACHE_DIR), and make and save them on the first call (about 3 minutes on a CPU). False makes them again in each session.

threshold

The lowest score that a concept needs to be a match. The score is a cosine.

• Values: None, or a number from -1 to 1 (in practice 0.4 to 0.7).
• Default None: keep the top concepts, however low their score.

Returns

{key: group}: one dict per group. The key is the group id, or the span text for a noise span. Each group has the keys

Raises

• ValueError: An unknown name_by, an unknown taxonomy, or an unknown strategy.

Example

>>> named = standardize_groups(clusterer(cards), top=1)
>>> named[1]["size"], named[1]["matches"][0]["leaf"]
(5, 'LONELINESS')

causal_map()

Build the causal map from the relations and the names of their spans.

causal_map(
    relations: list[dict],
    names: dict,
    min_papers: int = 1,
    min_sentences: int = 2,
    min_share: float = MIN_SHARE,
) -> dict

Source: theoryminer/causal_map.py

Each node is a construct: a named group, or a thesaurus concept. Each edge is “cause node -> effect node”, with the number of papers, sentences and relations that claim it. Self-loops and two-way pairs get flags. A filter hides a weak edge, but deletes nothing: a hidden edge goes to “hidden” with its drop_reason. The function reads no file and loads no model.

Arguments

relations

The relations from causenet().

• Values: a list of relation dicts. Each relation needs rel_id, doc_id, cause and effect; sent_id is used when it is there.
• Default: none. Give the relations.

names

The names of the spans, made from the same relations.

• Values: the result of label_groups() (a node is a group, named by its name), standardize_groups() (a node is the best concept of a group; groups with the same best concept share a node), or standardize_constructs() (a node is the best concept of a span). A group or span without a concept keeps its own node, under its own text.
• Default: none. Give the names.

min_papers

Show an edge only when at least this many papers claim it.

• Values: a whole number, 1 or more.
• Default 1: show the edge whatever the number of papers. This also suits one long book.

min_sentences

Show an edge only when at least this many different sentences claim it.

• Values: a whole number, 1 or more.
• Default 2: hide the claims that only one sentence makes. Use 1 to show every edge.

min_share

The share of the papers that one side of a two-way pair (A -> B and B -> A) needs to be the main direction.

• Values: a number above 0.5, up to 1.
• Default 0.67: one side needs 2/3 of the papers. Then it is “dominant” and the other side is “minority”. Else both sides are “contested”.

Returns

A dict with the keys

Raises

• ValueError: The relations are empty; names is empty or has an unknown shape; a span of a relation is not in names; or a setting is out of its range.

Example

>>> relations = [
...     {"rel_id": "r1", "sent_id": "s1", "doc_id": "p1", "cause": "job insecurity", "effect": "stress"},
...     {"rel_id": "r2", "sent_id": "s2", "doc_id": "p2", "cause": "fear of job loss", "effect": "anxiety"}]
>>> names = {0: {"central": "job insecurity", "members": ["job insecurity", "fear of job loss"],
...              "name": "job insecurity"},
...          1: {"central": "stress", "members": ["stress", "anxiety"], "name": "stress"}}
>>> cmap = causal_map(relations, names)
causal map: 2 nodes | 1 edges shown, 0 hidden | self-loops: 0 edges, 0 of 2 relations (0.0%) | two-way pairs: 0 (0 contested)
>>> edge = cmap["edges"][0]
>>> edge["source"], edge["target"], edge["n_papers"], edge["rel_ids"]
('g:0', 'g:1', 2, ['r1', 'r2'])

draw_map()

Draw the causal map as an interactive page.

draw_map(
    cmap: dict,
    papers: list[dict] | None = None,
    save: str | None = None,
    title: str = 'Causal map',
    height: int = FRAME_HEIGHT,
) -> None

Source: theoryminer/causal_map.py

In a notebook, the page shows in the cell output. In a plain Python script, the function writes the page to causal_map.html. The page is one HTML file that works offline. On the page you can hover over a node or an edge, click an edge to read all its sentences, search, drag, zoom, change the filters, and save a picture (PNG, JPG or SVG). The page gets every edge, also the hidden edges, and its sliders start at the filter values of causal_map().

Arguments

cmap

The map to draw.

• Values: the result of causal_map().
• Default: none. Give the map.

papers

The source of each paper, shown with each sentence: the DOI as a link, else the paper title, else the file name.

• Values: None, or the rows of extract_dois() (they replace the papers of the map).
• Default None: use cmap[“papers”], the DOI and title that harvest() found for each paper. A map from relations without them shows the doc_id (the file name) of each paper.

save

Where to write the page as an HTML file, in addition to the notebook output.

• Values: None, or a path such as “outputs/map.html”.
• Default None: write no file in a notebook, and causal_map.html in a script.

title

The title at the top of the page.

• Values: any text.
• Default “Causal map”.

height

The height of the map in a notebook cell.

• Values: a whole number of pixels.
• Default 720.

Returns

None. The function shows the page, or writes the file, or both.

Example

>>> draw_map(cmap)
>>> draw_map(cmap, save="outputs/map.html")
saved page: outputs/map.html

save_map()

Write the causal map to a folder: three CSV tables and one GraphML file.

save_map(
    cmap: dict,
    folder: str,
) -> list[str]

Source: theoryminer/causal_map.py

The files: nodes.csv one row per node, also the nodes that no shown edge touches (shown False) edges.csv one row per shown edge hidden.csv one row per hidden edge, with its drop_reason map.graphml every node and edge, for Gephi or Cytoscape desktop In the CSV files, a list (members, doc_ids, rel_ids) is one text, joined by “ | “. The function overwrites old files of the same name.

Arguments

cmap

The map to save.

• Values: the result of causal_map().
• Default: none. Give the map.

folder

The folder for the four files.

• Values: the path of a folder. A missing folder is made.
• Default: none. Give a folder.

Returns

A list of the paths of the four files.

Example

>>> save_map(cmap, "outputs/my_map")
saved map: outputs/my_map (nodes.csv, edges.csv, hidden.csv, map.graphml)

Settings and defaults

Each setting of each function, its default, and where the default comes from. The reference pages give every permitted value.

τ, the score that a span needs for a thesaurus concept, is the threshold of the standardize_* functions. It is off by default. Choose it per model, per strategy, per thesaurus. The evidence behind the defaults is in Thesaurus strategies, section 5.

Data shapes

The functions pass plain Python lists and dicts. This page lists the keys of each shape, so that you can read the results in your own code or in pandas.

A sentence

One item of harvest(...)["sentences"].

A dropped row

One item of harvest(...)["dropped"]: doc_id, page, section, drop_reason and text.

A relation

One item of the list that causenet() returns.

Groups

The result of clusterer() and pairwise_grouping(), keyed by the span:

{"loneliness": {"group": 4, "central": "address loneliness"},
 "levels of loneliness": {"group": 4, "central": "address loneliness"}}

A noise span of clusterer() has the group -1.

Names

The result of label_groups(), keyed by the group id (or by the span, for a noise span):

{4: {"central": "address loneliness", "size": 6,
     "members": ["loneliness", "levels of loneliness", ...],
     "keywords": [("loneliness", 0.91), ...],
     "name": "loneliness levels address"}}

standardize_groups() and standardize_constructs() give each group or span a list of thesaurus matches, best first. See their reference pages.

The causal map

causal_map() returns a dict with six keys:

A node:

The counts use the shown edges only. A filter that hides an edge also lowers the counts of its nodes.

An edge: edge_id, source, target, n_papers, n_sentences, n_relations, rel_ids, doc_ids, direction, reciprocal, self_loop and drop_reason. See The causal map for their meaning.

Standardization strategies, with leaves_only on and off

Companion to TUTORIAL.ipynb section 4. Every text and score below is real: it was printed by strategy_text and standardize_constructs on ELSST R5 with allmpnet.

1. What a strategy is

standardize_constructs matches a span to a taxonomy concept by embedding both and taking the cosine. The span is embedded as it is. The concept has no natural text, so a strategy is the recipe that builds one. Six recipes exist. They use four fields of a taxonomy entry:

Everything is lowercased before embedding.

Each strategy is one answer to one question: how much of the tree should the concept vector carry? leaf carries none. path, anchor, context, bracket carry the ancestors in four phrasings. enriched carries the ancestors and, more important, the synonyms.

2. What leaves_only is

ELSST is a tree. A leaf has no narrower concept under it. A branch has children.

MOTIVATION                    branch (and a top concept: no parent)
├── ASPIRATION                leaf
└── INCENTIVES                leaf

WELL-BEING (HEALTH)           branch (top concept)
├── PSYCHOLOGICAL WELL-BEING  leaf
├── LIFESTYLE AND HEALTH      leaf
├── AGEING                    leaf
└── ENVIRONMENTAL EXPOSURE    leaf

IDENTITY > PERSONAL IDENTITY > SELF-ESTEEM     leaf, two paths (also under IMAGE)

The switch changes which concepts may answer. It does not change any recipe or any score. A leaf’s text is the same under both settings.

How a branch becomes an entry

With leaves_only=False every branch gets an entry of the same shape as a leaf entry: its own label as leaf, the path from a top concept down to itself, its own parents. The six recipes then apply unchanged. Two cases:

A branch with parents (PERSONAL IDENTITY, child of IDENTITY and of IMAGE):

A top concept (no parent; 247 of them in R5, MOTIVATION and WELL-BEING (HEALTH) among them). The path is the label alone, the parents list is empty, and the recipes collapse:

anchor and enriched repeat the label for a top concept because the path is the label. Harmless for the cosine, but not pretty. It is the one place where a recipe would deserve a special case (skip the path when it equals the label).

A leaf with two paths

SELF-ESTEEM sits under IDENTITY and under IMAGE. It gets two entries, one per path, under both settings. The label, synonyms and note are the same in both; the path and parents differ:

At lookup the span is scored against both and the best score is kept for the concept. No path is preferred by depth or by any other rule; the span picks the path that fits it. 244 of the 2,661 leaf entries come from this (97 of those concepts also have synonyms). Under leaf the two vectors are identical and the duplicate falls away in the de-duplication.

3. The same spans under every strategy, on and off

Six SDT spans, top-1 concept and cosine, allmpnet.

leaves_only=True

leaves_only=False

How to read the two tables:

• Rows 3 to 6 are identical on and off. When the right concept is already a leaf, the switch does nothing. The only change in the candidate list is the addition of branches, and none of them beats the leaf for these spans.
• Rows 1 and 2 change under every strategy. intrinsic motivation moves from INCENTIVES (a child, and the wrong child: incentives are external rewards) to MOTIVATION. well-being moves from HAPPINESS or MENTAL HEALTH to WELL-BEING (HEALTH). Both targets are top concepts that leaves-only cannot offer. The scores rise too (0.65 → 0.69, 0.55 → 0.61 under enriched), which says the branch is a closer match, not only a permitted one.
• The strategy decides the score, not the concept. Down a column the winner rarely changes; the cosine does. leaf gives the highest cosines for exact and near-exact labels (academic achievement 1.00, self-confidence 0.79 through the synonym in the label space). enriched gives the lowest, because its long text spreads the vector over synonyms and ancestors. This matters for τ: a threshold tuned for leaf is too strict for enriched.
• need frustration fails everywhere. ANGER, BOREDOM, SATISFACTION at 0.50–0.56. ELSST has no concept for it. No strategy and no switch invents one; this is the standardisation-failure class.
• One warning in the anchor column. well-being → WELL-BEING (SOCIETY) instead of WELL-BEING (HEALTH). The two top concepts have the same name apart from the bracket; the anchor recipe repeats a top concept’s label twice, which shifts its vector enough to flip the tie. A recipe artefact, not a signal.

4. Which strategy, with the switch on or off

From the entry-term test in section 5 (positive and negative pairs, three models, both settings, plus a held-out synonym test):

• The ranking of strategies is the same on and off: leaf ≥ enriched > anchor > context ≈ bracket > path. Turning the switch off moves every F1 by at most 0.01.
• enriched stays the default. On held-out synonyms it is never worse than leaf beyond noise and has the best hit@5; on real spans, which are paraphrases rather than labels, the synonyms are the part that helps.
• path is the weakest under both settings. Ancestors written as a chain dilute the label.
• The switch is a granularity choice, not an accuracy one: off, broad constructs get their own concept and the graph carries nodes at mixed depth; on, every node is a leaf and broad constructs are pushed onto a child.

5. The entry-term test: how the defaults were chosen

A thesaurus lists its own synonyms. ELSST gives them as skos:altLabel. Each synonym with its concept is a positive pair: self-confidence ↔ SELF-ESTEEM. For each positive pair the test adds two negative pairs with the same query: a sibling concept (same nearest parent; hard) and a random concept (easy). A fixed seed draws the negatives once. A setting scores a pair by the cosine between the query and the concept text. The rule is same concept if score ≥ τ. A sweep over τ gives the best F1 and its τ. AUC is the threshold-free view. Top-1 accuracy asks if the gold concept wins the ranking against the whole thesaurus.

ELSST, three scopes:

enriched puts the synonyms inside the concept text. On this test the query sits inside its own target, and the score is inflated (F1 0.84–0.92). enriched_noalt (label, note, path; no synonyms) is the fair version. Best F1, fair rows, scope leaves / all:

• Strategy. The bare label wins. Each phrasing of the tree and the note pull the vector away from the label and cost 0.02–0.05 F1. The spread between strategies is small next to the spread between models.
• Model. bge_base beats allmpnet by about 0.02 F1 and 0.04 top-1 accuracy in every cell. minilm is last. bge_base separates positives from siblings more sharply: τ* is 0.65–0.70, against 0.45–0.55 for allmpnet.
• leaves_only. The branches as candidates cost F1 0.005 and top-1 accuracy 0.03–0.07 on the same queries. The branches bring their own labels and synonyms in return. The switch is a granularity choice, not an accuracy choice.
• τ. τ* depends on the model: about 0.50 for allmpnet with leaf, about 0.70 for bge_base. The old τ = 0.55 was set for all-mpnet-base-v2. Do not reuse it for another model, another strategy or another thesaurus.
• The hard errors are the thesaurus’s own granularity. Siblings accepted at τ*: sporting activities → SPORTS CLUBS (gold SPORTS); distance courses → ONLINE EDUCATION (gold DISTANCE EDUCATION); achievement tests → INTELLIGENCE TESTS (gold EDUCATIONAL TESTS). Positives missed: mores → CUSTOMS AND TRADITIONS (0.45); perks → FRINGE BENEFITS (0.63). No setting resolves these. They are the granularity-mismatch class of the disagreement classifier.

Do the synonyms in enriched help with a synonym the text has never seen? A second test holds one out. For the 651 ELSST concepts with two or more synonyms, one synonym leaves the concept text and becomes the query. Mean F1 over three models and two scopes:

The synonyms generalise. They add 0.01–0.03 F1 over the bare label for bge_base and minilm, and nothing for allmpnet. The note and the path cost about 0.02 alone. The synonyms pay that cost back. enriched stays the default: it is never worse than leaf beyond noise, it gives the best hit@5, and it is the recipe Paper 3 ships.

The test itself was a temporary script and is not part of the package.

6. Where to look in the code

• Recipes: strategy_text in theoryminer/harmonizer/taxonomy.py.
• Entries and paths: _load_elsst and its inner paths_to_root in the same file. All root-to-concept paths are kept, one entry each; nothing is chosen by depth.
• The switch: load_taxonomy(name, leaves_only); the cache key includes it.
• Lookup and de-duplication by concept: _top_matches in standardize.py.

Tutorials

The repository has two notebooks. Both use the 7 sample papers in sample_files/.

In Colab, the first cell installs the package and downloads the sample papers. Choose a GPU runtime for causenet(): Runtime → Change runtime type → T4 GPU.

On your own computer, run the notebooks from tutorials/ or from the repository root. The first cell moves to the repository root.

The page Thesaurus strategies explains the six ways to turn a thesaurus concept into text, and the test behind the defaults.

The sample papers

sample_files/ holds 7 open-access social-science papers with a CC BY licence. The file sample_files/SOURCES.md gives the title, the authors, the DOI and the licence of each paper.

Changelog

Each release of theoryminer has one section. The newest release is first. The version numbers follow semantic versioning: before 1.0.0, a minor release (0.x.0) can change the API.

0.3.0

This release rewrites download_papers() and restructures the first stages into folders.

Added

• A documentation website: https://rasoulnorouzi.github.io/theoryminer/. It has a guide for each step, a reference page for each public function, and every setting with its default. The book is made with mdBook from docs/. The reference pages come from the docstrings (docs/make_api.py), so they always match the code. A push to main publishes the book with GitHub Pages.
• download_papers() is rewritten. It tries many free sources in turn until one gives a real PDF: OpenAlex, per-publisher plugins, PubMed Central, Europe PMC, Unpaywall, Semantic Scholar, Crossref, OSF preprints, and the doi.org landing page. On a 100-DOI benchmark it downloads the correct PDF for 33 papers; the old version (OpenAlex, then tmsr-doi-downloader) downloaded 25. It is also faster. The new sources setting chooses the sources and their order.
• Per-publisher plugins, in the style of Zotero translators (theoryminer/download/publishers.py). A plugin turns a DOI into the direct PDF link of its publisher (Springer with BMC and Palgrave, and Frontiers). To add a journal, add one plugin.
• download_papers() gets open-access copies only. It does not log in to a publisher site and it does not use a browser. A paper that is only behind a paywall stays not_found.
• respect_robots (default True). Set it False only when you have the right to download from the sites, for example by an agreement of your institution.
• The DOI and title of each paper travel with the data. harvest() puts doi and title on every sentence (new setting find_dois, default True). causenet() copies them to every relation. causal_map() collects them in cmap["papers"], and draw_map(cmap) shows them. The call draw_map(cmap, papers=extract_dois(folder)) is no longer needed; papers= still works.
• A better DOI rule, shared by harvest() and extract_dois(). The DOI comes from the most certain source: the download log downloads.csv of the folder, a DOI in the text that gives the file name of a download_papers() PDF, the subject or keywords field of the PDF metadata, then the first DOI on pages 1-2. extract_dois() rows have a new column source. On 21 papers with a known DOI (7 sample papers and 14 downloads) the rule finds 20; the old rule found 18.
• download_papers() always writes its log downloads.csv into the PDF folder. The log keeps the rows of earlier runs. harvest() reads it, so it knows the DOI of every downloaded PDF.

Changed

• The modules of the first stages are in two new folders. The public functions do not change: from theoryminer import harvest, extract_dois, extract_references, download_papers works as before.

  Before	Since this release
  theoryminer/harvest.py	theoryminer/prepare/harvest.py
  theoryminer/dois.py	theoryminer/prepare/dois.py
  theoryminer/download.py	theoryminer/download/papers.py

• download_papers() no longer uses tmsr-doi-downloader in the default. On the benchmark it added one paper per hundred and doubled the run time. tmsr-doi-downloader stays a dependency and an opt-in source: sources=[..., "tmsr-doi-downloader"].

• The docstrings of the public functions give, for each argument, every permitted value and what the default does in words. For example, harvest(steps=None) names the 10 steps of DEFAULT_STEPS. Five functions got an example: causenet(), standardize_constructs(), standardize_groups(), pairwise_grouping() and clusterer(). A new test, tests/test_docstrings.py, checks every argument and every choice value.

• causenet() checks its settings before the model loads. An unknown mode or decision, a threshold outside 0 to 1, or a batch_size below 1 raises a ValueError. mode accepts "neural" and "auto". Before, mode="heuristic" (or a typing error) ran the neural path without a message, and an unknown decision ran "cls+span".

• harvest() checks its settings before it reads a PDF. A page selection that is not a page, an unknown step name, min_words below 1, or max_words below min_words raises a ValueError with the permitted values. Before, an unknown step gave a KeyError.

Fixed

• clusterer() no longer prints a FutureWarning from scikit-learn 1.9 about the copy setting of HDBSCAN. The clusters do not change.
• tutorials/PACKAGE_LINE_BY_LINE.ipynb runs again without an error. Four cells used the APA thesaurus (removed) or the defaults before 0.2.0 (leaves_only=True, linkage="single").

Deprecated

• The module paths theoryminer.harvest and theoryminer.dois still work, as aliases of the new modules. They go away in 0.4.0. Use theoryminer.prepare.harvest and theoryminer.prepare.dois.
• theoryminer.download is now a folder. from theoryminer.download import download_papers still works. Its private helpers (names that start with _) are in theoryminer.download.papers.

0.2.2

Added

• Editor support. The 13 public functions have type hints: the kind of each setting and of the result. The package holds the marker file py.typed (PEP 561), so VS Code (Pylance), PyCharm and type checkers use these hints: they suggest the settings, and they mark a wrong value, for example min_words="four".
• __all__ in theoryminer and in theoryminer.harmonizer lists the public names.

0.2.1

Changed

• pip install theoryminer installs tmsr-doi-downloader on Python 3.11 or newer, so download_papers() has all its sources without an extra. On Python 3.10 it is left out, and download_papers() uses OpenAlex only. The old command pip install "theoryminer[download]" still works.
• CI installs and tests the package on Windows and macOS with Python 3.11, 3.12 and 3.13, and on Linux with Python 3.10 to 3.13.

Fixed

• download_papers() no longer clears a key that you set as an environment variable (UNPAYWALL_EMAIL, CORE_API_KEY, SERPAPI_KEY). A key that you do not pass as an argument keeps its environment value.

0.2.0

Added

• The causal map. causal_map() turns the relations and the names of their groups into nodes and edges. An edge counts papers, sentences and relations, and keeps its rel_ids. Two relations of one sentence on one edge count as one sentence. Self-loops and two-way pairs get flags. The filters min_papers (default 1) and min_sentences (default 2) move the weak edges to hidden, with a reason; nothing is deleted.
• draw_map() shows the map as an interactive page: in a notebook cell, or as causal_map.html from a script. The page works offline. It has hover cards, a panel with every sentence and the DOI or title of its paper, a search that filters the map (the matches and their neighbours), filters by paper, role, two-way pairs, papers and sentences, and a picture export as PNG, JPG or SVG.
• save_map() writes nodes.csv, edges.csv, hidden.csv and map.graphml.
• extract_references() finds the DOIs in the references of seed papers, one row per DOI, with the number of seeds that cite it. It repairs a DOI that the PDF cuts at a line end, and it leaves out the seed’s own DOI and its figure and table DOIs. It can save a CSV file with a doi column. On the sample papers, 214 of its 215 DOIs exist at doi.org.
• download_papers() downloads the open-access PDFs of a list of DOIs. It asks OpenAlex for the licence first, so licences=["cc-by"] downloads only CC BY papers. Then it tries the OpenAlex links and, when it is installed, tmsr-doi-downloader (pip install "theoryminer[download]", Python 3.11 or newer). One status row per DOI; a failed DOI does not stop the run; a second run skips the files it has.
• extract_dois() has a new column, title: the largest font on page 1, else the PDF metadata.
• causenet(avoid_ambiguous=True) skips a relation whose cause or effect only points to another sentence (“this”, “it”, “such things”). The summary prints the count and examples.
• pairwise_grouping(linkage=...): "average", "complete" or "single".
• standardize_constructs() and standardize_groups() have a threshold. It is off by default.
• harvest() drops the licence and copyright text of a journal, with the reason licence_text.

Fixed

• causenet() no longer returns a relation with an empty cause or effect (the model decoded only special tokens). The summary counts them. Before, causal_map() stopped on such a relation.

Changed

These defaults change the results of a run that uses them. Give the old value to get the old results.

• causenet(): decision="span_only" (was "cls+span") and avoid_ambiguous=True (new).
• pairwise_grouping(): linkage="average" (the old method is linkage="single"). Single linkage chains: on the sample papers its largest group held 153 spans; average linkage gives 27.
• standardize_constructs() and standardize_groups(): leaves_only=False (was True). A span can now also match a broad ELSST concept, such as TRUST or MOTIVATION.
• label_groups(): by="keywords" (was "both"). The label is the TF-IDF keywords of the group.
• networkx is a declared dependency (save_map() writes GraphML with it). PyTorch installed it already.

Data in the package

• causal_map.html (the page template), Cytoscape.js 3.30.2 and the fcose layout (all MIT).

0.1.1

Changed

• umap-learn is installed by default. pip install theoryminer is enough for clusterer(..., umap={...}). The old command pip install "theoryminer[umap]" still works.
• The thesaurus vectors are saved in ~/.cache/theoryminer/taxonomy, not inside the installed package. The environment variable THEORYMINER_CACHE_DIR chooses another folder.

Fixed

• A read-only install no longer fails on the first standardize_constructs() call. When the cache folder cannot be written, the function prints one line and goes on without the cache.
• pip uninstall theoryminer no longer leaves a cache folder in the package folder.

0.1.0 (first release on PyPI)

Functions

• harvest() reads one PDF, or every PDF in a folder, and returns clean sentences and a drop log. A bad file in a folder does not stop the run. Each sent_id starts with the document number.
• extract_dois() finds the DOI of each paper: the first DOI on page 1, else on page 2. It returns one row per file and can save a CSV file.
• causenet() finds cause-effect pairs with the SocioCausaNet model. device="auto" uses an NVIDIA GPU, else an Apple GPU, else the CPU.
• standardize_constructs() and standardize_groups() map spans and groups to ELSST concepts.
• pairwise_grouping(), clusterer() and label_groups() group and name the spans without a thesaurus.

Data in the package

• ELSST release 5 (CC BY-SA 4.0, © CESSDA) ships in the package. taxonomy="elsst" needs no extra file.

Tests and release

• Fast tests run on each push, on Linux, Windows and macOS. Slow tests load the real models.
• A GitHub Release publishes the package to PyPI through Trusted Publishing.

Development

Set up

git clone https://github.com/rasoulnorouzi/theoryminer.git
cd theoryminer
python -m venv .venv
source .venv/bin/activate
pip install -e ".[notebooks,dev]"

The code uses plain functions with full docstrings. A step never deletes a row: it marks the row and writes the reason. Keep these two rules when you add a step.

Tests

pytest                                  # the fast tests: no model download, a few seconds
pytest -m slow                          # the tests that load the real models (about 1 GB on the first run)
pytest tests/test_harvest.py -k folder  # one test by name

GitHub Actions runs the fast tests on each push, on Linux, Windows and macOS. It also builds the package and tests the built wheel.

Benchmarks

This documentation

The book uses mdBook. The pages are Markdown files in docs/src/. The reference pages come from the docstrings: a script writes them before each build, so the docstrings stay the only source.

python docs/make_api.py     # write docs/src/reference/ from the docstrings
mdbook serve docs --open    # build the book and open it; it builds again when a file changes

To get mdbook, download it from its release page, or run cargo install mdbook.

A push to main builds the book and publishes it with GitHub Pages (.github/workflows/docs.yml).

Releases

A push to main never publishes to PyPI. A release has three steps:

1. Set the new version in pyproject.toml, and in the first cell of tutorials/TUTORIAL.ipynb (%pip install -q "theoryminer==<version>"). A test checks that the two match. Add a section to CHANGELOG.md. Push, and wait for a green CI.
2. On GitHub, open Releases → Draft a new release. Give the tag v plus the version, for example v0.3.0. Click Generate release notes, then Publish release.
3. In the Actions tab, open the “Publish to PyPI” run and approve it (Review deployments).

The workflow stops when the tag and the version are different. PyPI trusts the workflow through Trusted Publishing, so the repository stores no token.

Licence

The code has the MIT licence. See the file LICENSE.

These files keep their own licence:

The papers that you download

download_papers() downloads open-access copies, and it obeys robots.txt by default. The licence of each paper is in the licence column of its result. A licence such as CC BY permits you to share the text; other licences do not. Check the licence before you publish the text of a paper, for example in a notebook output.