Scientific research moves fast these days, and machine learning tools are changing the game when it comes to reading and making sense of research papers. At Symby AI, we're committed to enhancing science by leveraging cutting-edge AI technology as the autopilot of scientific research, helping eliminate bad science and improve the scientific method. But even as Vision Language Models (VLMs) achieve remarkable breakthroughs, we've discovered a fascinating challenge lurking in one of the most fundamental tasks: extracting text from scientific PDFs.

The PDF Paradox: Built for Eyes, Not Machines

PDFs were designed with humans in mind—beautiful, visually appealing documents that preserve formatting across different devices and platforms. But this visual-first approach creates a fundamental tension with machine readability. While PDFs look perfect to us, they're essentially complex puzzles for AI systems trying to extract meaningful information.

Our recent investigation into VLM performance on scientific document OCR revealed some surprising insights. We evaluated 10 PDFs spanning diverse scientific fields—from cutting-edge biotech research to distant galaxy observations in astronomy, and complex algorithms in computer science. Using an LLM-as-judge evaluation approach, we uncovered patterns that illuminate both the promise and the pitfalls of current AI technology.

Benchmark Results: The Current State of Scientific OCR

Our testing revealed interesting performance patterns across different models:

While the performance differences appear modest, the nuances in how these models handle different types of scientific content reveal important insights about the current state of AI document processing.

The Multi-Column Mystery

One of the most intriguing findings emerged around multi-column layouts, a staple of academic publishing. Time and again, we observed VLMs missing entire columns during document extraction. What made this particularly fascinating was the disconnect between awareness and execution.

When we directly asked these same models, "How many columns does this document have?" they correctly identified the layout structure. They knew there were multiple columns, yet somehow failed to extract all the content during OCR tasks. This suggests that the challenge isn't visual understanding per se, but rather the complex orchestration required for comprehensive document processing.

It's as if the AI can see the forest and individual trees, but struggles to systematically harvest every branch when tasked with clearing the entire woodland.

The Awareness Gap

What made our multi-column findings particularly fascinating was the disconnect between model awareness and execution. When we observed VLMs missing entire columns during document extraction, we decided to test their structural understanding directly. Remarkably, when asked "How many columns does this document have?" these same models correctly identified the layout structure.

This reveals an intriguing gap: the models understand document structure but struggle to systematically process all content during comprehensive extraction tasks. It's as if they can see the complete picture but lack the methodical approach needed for thorough content extraction.

The Visual Nature of Scientific Knowledge

Academic papers present unique challenges that go far beyond typical document processing. Scientific knowledge is inherently visual, and this creates layers of complexity that even advanced AI systems struggle to navigate.

Consider astronomy papers, where image grids don't just illustrate concepts—they are the data. Each galaxy photograph or spectrograph contains crucial research findings that require contextual understanding to properly interpret. Similarly, the mathematical formulas that define scientific theories aren't just text—they're essentially images that convey complex relationships and calculations.

The Table Trouble

While OCR technology has made significant strides in extracting standard tables, academic documents present a different beast entirely. Scientific papers often feature:

• Nested table structures with multiple levels of organization
• Complex note sections that provide crucial context
• Mixed content tables combining text, numbers, and symbols
• Multi-page tables that span document sections

These aren't the clean, simple tables you might find in a business report. They're intricate data structures that require deep understanding of academic conventions and scientific notation.

Implications for the Future of Scientific Research

These findings reveal both the current limitations and future potential of AI in scientific research. The performance variations we observed—particularly the trade-offs between formula processing and header extraction in specialized models—highlight the complexity of optimizing AI for scientific document processing.

The awareness gap we identified suggests that future improvements may come not just from better visual understanding, but from developing more systematic approaches to comprehensive document processing.

The Road Ahead

The challenges we've identified in scientific PDF OCR are not roadblocks—they're insights that illuminate the path forward. As we continue to develop AI technology that serves as the autopilot of scientific research, understanding these nuances becomes crucial for building systems that truly enhance rather than hinder scientific discovery.

The goal isn't to replace human expertise but to augment it intelligently. By acknowledging where current AI excels and where it struggles, we can design complementary systems that leverage the strengths of both human insight and machine processing power.

As we advance toward a future where AI helps eliminate bad science and strengthen the scientific method, these findings remind us that the most powerful tools are often those that embrace complexity rather than oversimplify it. The challenge of scientific PDF OCR is just one piece of a much larger puzzle—but it's a piece that reveals important truths about the intersection of artificial intelligence and human knowledge.

At Symby AI, we're committed to transparency in our research and development process. These findings represent ongoing work in our mission to enhance scientific research through cutting-edge AI technology. Stay tuned as we continue to explore the frontiers of AI-assisted scientific discovery.