download_papers()

Download the open-access PDFs of the DOIs into a folder. Return one status row per DOI.

download_papers(
    dois: list | str,
    output_dir: str,
    licences: list[str] | None = None,
    email: str | None = None,
    sources: list[str] | None = None,
    respect_robots: bool = True,
    core_api_key: str | None = None,
    serpapi_key: str | None = None,
    save: str | None = None,
) -> list[dict]

Source: theoryminer/download/papers.py

The function asks OpenAlex for the licence of each paper. With licences, it downloads only the papers with one of those licences. Then it tries the sources in turn until one gives a real PDF. A link can give the PDF or a page; for a page, the function follows its citation_pdf_url meta tag once. It waits 1 second between two requests to the same site. A PDF that is already in the folder is not downloaded again, so a stopped run can go on. A failed DOI does not stop the run. The function writes the PDFs and the log downloads.csv into output_dir (the log keeps the rows of earlier runs, and harvest() reads it to know the DOI of each file), and a report file when you give save. It gets open-access copies only: it does not log in to a publisher site.

Arguments

dois

The papers to download.

• Values: the rows of extract_references() (dicts with a “doi” key), a list of DOI strings, or the path of a CSV file with a “doi” column. A text is always read as the path of a CSV file, not as one DOI.
• Default: none. Give the DOIs.

output_dir

The folder for the PDFs.

• Values: the path of a folder. A missing folder is made. The file name is the DOI with “_” for each unsafe character.
• Default: none. Give a folder.

licences

The licences to download.

• Values: None, or a list of OpenAlex licences: “cc-by”, “cc-by-sa”, “cc-by-nc”, “cc-by-nd”, “cc-by-nc-sa”, “cc-by-nc-nd”, “public-domain”, “other-oa”, “publisher-specific-oa”, “none” and “unknown”.
• Default None: download every paper that has an open-access copy, whatever its licence.

email

Your email address. OpenAlex and Crossref answer faster with it (“polite pool”), and Unpaywall needs it.

• Values: None, or a real email address (Unpaywall refuses a made-up address).
• Default None: no email. The function skips Unpaywall, and tmsr-doi-downloader uses the environment variable UNPAYWALL_EMAIL when it is set.

sources

The sources to try, in this order.

• Values: a list of one or more of “openalex” (the PDF links and repository pages of the open-access copies in OpenAlex), “publisher” (the publisher’s direct PDF link, from a per-publisher plugin), “pmc” (the PDF in the PubMed Central open-access data of NCBI), “europe_pmc” (the free PDF links that Europe PMC lists), “unpaywall” (the PDF links of Unpaywall; it needs email), “semantic_scholar” (the open-access PDF link of Semantic Scholar), “crossref” (the PDF links that the publisher gives to Crossref), “preprint” (the open OSF preprint of the paper, from the Crossref relation), “landing_page” (the page that doi.org opens, and its PDF link), “tmsr-doi-downloader” (the package tmsr-doi-downloader; Python 3.11 or newer).
• Default None: the nine open sources above, from “openalex” to “landing_page”, in that order. The default leaves out “tmsr-doi-downloader”. Add it with this setting when the open sources miss a paper.

respect_robots

Whether to obey the robots.txt file of each site.

• Values: True or False. Set False only when you have the right to download from the sites, for example by an agreement of your institution. The wait between requests stays. tmsr-doi-downloader always obeys robots.txt.
• Default True: skip every link that robots.txt does not allow.

core_api_key

A free key from core.ac.uk, for CORE in tmsr-doi-downloader.

• Values: None, or the key as text.
• Default None: use the environment variable CORE_API_KEY when it is set, else skip CORE.

serpapi_key

A key of SerpAPI (a paid service), for Google Scholar in tmsr-doi-downloader.

• Values: None, or the key as text.
• Default None: use the environment variable SERPAPI_KEY when it is set, else skip Google Scholar.

save

Where to write the report rows as a CSV file.

• Values: None, a path that ends with “.csv”, or a folder (then <folder>/downloads.csv).
• Default None: write no report file.

Returns

A list of dicts, one per DOI, with the keys

Raises

• ValueError: when sources is empty or holds a name that is not one of the sources above.

Example

>>> refs = extract_references("sample_files")
>>> report = download_papers(refs, "outputs/round_2", licences=["cc-by"])
>>> report[0]["status"], report[0]["source"]
('downloaded', 'openalex')