Goal:
Develop an advanced PDF‑to‑text extraction pipeline tailored for Retrieval‑Augmented Generation (RAG) and local LLMs, with a strong focus on preserving document structure (sections, headings, tables, figures) and producing LLM‑friendly representations. The pipeline should improve over common PDF extractors by reliably detecting paragraphs and sections, extracting tables into useful data formats, handling images via OCR and description, and cleaning headers/footers so that the final text is well‑suited for downstream RAG and question‑answering tasks.
What is expected:
The implementation should rely on open‑source Python libraries without API keys (e.g., PyMuPDF [1], pdfplumber [2], Camelot [3], tabula‑py [4]) that provide low‑level access to PDF layout, text, and tabular data. Building on these, the student designs modular components for:
- Paragraph and section detection based on heading recognition (font size, margins, styling, numbering patterns, table of contents information).
- Robust table identification and extraction (using existing table‑extraction libraries and/or LLM‑assisted detection) into structured formats such as CSV/JSON or pandas DataFrames for RAG.
- Image extraction with OCR for embedded text and generation of textual descriptions of what is shown (e.g., trends in plots, “highest value at …”) to make visual information accessible to LLMs.
- Systematic removal or normalization of page headers, footers, page numbers, and other boilerplate to reduce noise for retrieval.
- Text cleanup (whitespace, broken lines, hyphenation, encoding issues) to produce high‑quality chunks for indexing.
For evaluation, the pipeline is integrated into a simple RAG or QA prototype (CLI interface or an existing chatbot) using local open‑source LLMs (4–8B, e.g. via the Ollama API [7]), and compared against baseline PDF ingestion workflows from state‑of‑the‑art RAG/agent frameworks (e.g., document loaders in LlamaIndex [5] or LangChain [6]). The comparison should cover extraction quality (structure, tables, images), robustness across different document types, and the impact on downstream LLM answer quality.
The code is open‑source and will be hosted publicly on the student’s GitHub repository. A PyPI package is expected so that the extractor can be installed and used like standard PDF/RAG utilities in modern AI frameworks.
Extensions or improvements over existing PDF/RAG extractors or previous bachelor theses are possible (to be discussed with the supervisor).
Recommended requirements:
Implementation in Python
Interest in LLMs, RAG pipelines, and document understanding / PDF processing
Access to computer hardware which can run local 4-8 B LLMs
Interest in building LLM AI solutions
Supervisor:
Dr. Marian LUX - marian.lux@univie.ac.at
Supervision and thesis in German or English
References:
[1] PyMuPDF: High‑performance Python library for text and layout extraction from PDF and other document formats. https://github.com/pymupdf/PyMuPDF
[2] pdfplumber: Precise text and table extraction for machine‑generated PDFs. https://github.com/jsvine/pdfplumber
[3] Camelot: PDF Table Extraction for Humans. https://github.com/camelot-dev/camelot
[4] tabula-py: Simple wrapper for tabula-java, read tables from PDF into DataFrame. https://pypi.org/project/tabula-py/
[5] LlamaIndex: Document loaders and RAG tooling. https://docs.llamaindex.ai/
[6] LangChain: PDF loaders and RAG pipelines. https://python.langchain.com/
[7] Ollama: Local LLMs and Python client. https://github.com/ollama/ollama-python