I'm working on a project that sources a lot of publically accessible but unprocessed, unformatted and unconsolidated data. Currently I'm just running Brave API + Crawlee (Playwright) + AWS Bedrock in Python for realtime processing. This is fine but not exactly scalable.

I have also tried to do some web scraping myself building my own data processing pipeline from scratch and honestly this is a bit of a pain.

So I am asking HN how do you find and process information online. Especially if its not something you can plan ahead for.

I know there's tavily search as well as the new perplexity api but they don't quite fulfil what I need.