The problem
Search over a scientific corpus usually indexes prose and throws away everything else. That is a strange choice, because in most papers the figures and tables carry the result. A reader scanning for "the study where accuracy dropped off at high b-values" is looking for a plot, not a sentence.
Indexing figures is harder than it sounds for two reasons. Extracting them reliably is a real engineering problem. And a figure on its own is close to meaningless: the caption is terse, and the interpretation lives in whatever paragraph references it, which may be three pages away.
The output target was therefore not "images from a PDF" but an item per figure or table, carrying its caption, the body text that discusses it, and a structured summary, in a form that a retrieval or topic-modelling pipeline can embed directly.
Why not PyMuPDF
The obvious approach is to walk the PDF and pull out embedded images with something
like page.get_images(). This fails on scientific papers, and it fails
silently.
A large share of scientific figures are not raster images at all. They are vector graphics, sets of drawing operations emitted by matplotlib, Illustrator or R. There is no image object to extract, so an image-object walker returns nothing and reports no error. You get a clean run and a fraction of the figures.
The pipeline uses docling instead, which runs a layout model over each page and renders figure regions from predicted bounding boxes. Because it operates on regions rather than embedded objects, it captures vector and raster figures alike. It also recovers reading order across multi-column layouts, which turns out to be the thing that makes the next stage possible at all.
Architecture
Linking figures to their discussion
The linker scans body text for references to each numbered item and attaches the surrounding paragraphs, capped at five per item so a heavily-cited figure doesn't drag half the paper along with it.
This stage is why the extraction choice mattered. Scientific papers are typically two-column, and a naive text extraction interleaves the columns, producing sentences that jump mid-clause from one column to the other. Reference matching against that text finds fragments and mislinks them. Getting reading order right at stage one is what makes stage two work on arbitrary papers rather than only on single-column preprints.
Structured summarisation
Each figure image and its linked context go to a vision LLM, which returns a fixed
JSON schema rather than prose: figure_type, subject,
scientific_insight, axes_or_elements,
key_data_points and embedded_equations.
Fixed fields rather than free text is the deliberate part. A paragraph of description
is pleasant to read and awkward to index. Discrete fields can be embedded separately,
filtered on, or surfaced as facets, and the schema is a constant in
summarizer.py so it can be reshaped for whatever the downstream index
needs.
Evaluation
Tested across 100 papers from 10 publishers, chosen for variety because layout conventions differ substantially between them and a pipeline tuned on one publisher's template tends to fall over on another.
| Task | Detected | Rate |
|---|---|---|
| Table detection | 320 / 320 | 100% |
| Figure detection | 496 / 540 | 92% |
Table detection was exact across the corpus. Figure detection reached 92%, measured against the figures actually present in each paper rather than against whatever the pipeline happened to return, which is the stricter of the two denominators.
Performance and deployment
A ten-page paper parses in 30 to 60 seconds on CPU. Layout models are a one-time download of roughly 1 to 2 GB. Figure resolution is a knob, defaulting to about 144 DPI, and the summarisation model is swappable per run to trade cost against capability on harder figures.
The Gradio front end runs as a hosted tool on Hugging Face Spaces, which makes it usable by people who won't run a Python script, and it can be consumed programmatically through the Gradio client so a dashboard can use the miner as a service and keep its own UX.
Known limits
- Scanned PDFs need OCR. docling can do it, but it must be enabled explicitly and parsing gets slower.
- Multi-panel figures come out whole. A figure with panels (a) through (d) is extracted as one image, so panel-level summarisation isn't available.
- Equations are described, not parsed. The vision model reads equations inside figures but doesn't emit LaTeX. Getting parseable output would mean adding a dedicated model such as pix2tex or nougat at the summarisation step.
Built at the Predictive Analytics and AI Research Lab, NYU Courant, advised by Prof. Anasse Bari. Try the tool · Code on GitHub.