FinePDFs: Liberating 3T of the finest tokens from PDFs - a Hugging Face Space by HuggingFaceFW

Hugging Face introduces FinePDFs, a large open dataset built by extracting and cleaning text from millions of PDF documents, reaching trillions of tokens across many languages. The post explains how the pipeline handles messy PDF structure, layout noise, duplication, and low-quality content to produce training-ready text. The goal is to unlock high-value data from PDFs, such as scientific and technical documents, and make it openly available for model training and research.