harvest()

Read one PDF, or every PDF in a folder, and return the clean sentences and a drop log.

harvest(
    source: str,
    pages: str | int | list[int] | None = None,
    skip_pages: str | int | list[int] | None = None,
    steps: list[str | Callable] | None = None,
    min_words: int = 4,
    max_words: int = 80,
    cache: bool = True,
    find_dois: bool = True,
) -> dict

Source: theoryminer/prepare/harvest.py

The function reads the text of each page, runs the cleaning steps in order, and splits the text into sentences. A step never deletes a row: it marks the row as dropped and gives the reason. A doubtful sentence keeps its row and gets a flag. For one PDF, a bad file raises a ValueError. For a folder, a bad file does not stop the run: the drop log gets one row for the file, with page 0 and the reason “not_pdf” or “unreadable_pdf”. The function looks in the folder only, not in its subfolders. It writes one file per PDF, the page cache (see cache).

Arguments

source

The PDF file, or the folder of PDF files, to read.

• Values: the path of one PDF file, or the path of a folder.
• Default: none. Give a path.

pages

The pages to read. Page 1 is the first page, as in a PDF viewer.

• Values: one page (12), a text (“12-540” or “3,7,9-12”), a list of pages ([3, 5, 9]), or None. For a folder, the same pages apply to every PDF.
• Default None: read every page.

skip_pages

The pages to leave out, after pages has selected its pages.

• Values: the same forms as pages.
• Default None: leave out no page.

steps

The cleaning steps, in the order in which they run.

• Values: a list of step names from STEPS, of your own functions, or of both. A function of your own takes a list of row dicts and returns a list of row dicts. To switch off one step, give DEFAULT_STEPS without that name.
• Default None: run DEFAULT_STEPS, the 10 steps in this order: fix_unicode, strip_markdown, drop_running_heads, drop_junk_lines, join_paragraphs, split_sentences, strip_citations, strip_stats, drop_junk_sentences, filter_length.

min_words

The fewest words that a sentence can have (the step filter_length).

• Values: a whole number, 1 or more.
• Default 4: a sentence with 3 words or fewer is dropped as “too_short”.

max_words

The most words that a sentence can have (the step filter_length).

• Values: a whole number, min_words or more.
• Default 80: a sentence with 81 words or more is dropped as “too_long”.

cache

Whether to keep the text of the pages on disk.

• Values: True or False.
• Default True: write <pdf>.pages.json beside each PDF, and read it on the next run. The next run then skips the slow text extraction. False reads every PDF again and writes no file.

find_dois

Whether to find the DOI and the title of each paper, for each sentence.

• Values: True or False.
• Default True: find them, as extract_dois() does. The DOI comes from the most certain source: the download log (downloads.csv in the folder), the file name of a download_papers() PDF, the PDF metadata, then the first DOI on pages 1 and 2. causenet() copies them to each relation, and draw_map() shows them. False skips this work and gives “” for both.

Returns

A dict with two lists:

• “sentences”: one dict per kept sentence, with the keys

• “dropped”: one dict per dropped row, with the keys doc_id, page, section, drop_reason and text.

Raises

• ValueError: The source does not exist, the folder has no PDF, or the one PDF cannot be read. Also for a wrong setting: a page selection that is not a page, an unknown step name, min_words below 1, or max_words below min_words.

Example

>>> data = harvest("sample_files")
>>> len(data["sentences"]), data["sentences"][0]["sent_id"]
(2531, 'd000p0000s000')