Given the current llm context size limitation, what is the state of art for feeding large doc/text blobs into llm for accurate processing?

I've been thinking lately that maybe we need a new AI-friendly file format rather than continuing to hack on top of PDF's complicated spec. PDF was designed to have consistent and portable page display rendering, it was not a goal for it to be easily parseable afaik, which is why we have to go through these crazy hoops. If you've ever looked at how text is stored internally in PDF this becomes immediately obvious.

I've been toying with an idea of a new format that stores text naturally and captures semantics (e.g. to help with table parsing), but also preserves formatting rules so you can still achieve fairly consistent rendering. This format could be easily converted to PDF, although the opposite conversion would have the regular challenges. The main challenge is distribution of course.

Very cool. I'll probably use it, but not for AI. I have lots of pdfs for which an epub doesn't exist.

Or if anything I'll add it to the projects-that-already-do-this-but-havent-yet-found list.

How does it compare to docling?

I just tested it on one of my nemeses: PDF bank statements. They're surprisingly tough to work with if you want to get clean, structured transaction data out of them.

The JSON extract actually looks pretty good and seems to produce something usable in one shot, which is very good compared to all the other tools I've tried so far, but I still need to check it more in-depth.

Sharing here in case someone chimes in with "hey, doofus, $magic_project already solves this."

There is also Unstract open-source. Structured data extraction + ETL. https://github.com/Zipstack/unstract

I got excited until I read that it was Java/Python based.

I'm looking for a library that can extract data tables from PDF and can be called from a C++ program (for https://www.easydatatransform.com). If anyone can suggest something, I'm all ears.

Just finished migrating to it to replace pdf2docx in a new project I’ve been working on and it is so much better. Thanks for open sourcing OpenDataLoader-PDF!