Hi this is Nibs. I'm looking for feedback on Scraipe, a python scraping and LLM analysis framework. Scapy does web crawling very well, so Scraipe focuses on versatility; it can pull content from Telegram, CertUA, and other APIs in addition to websites. Scraipe also integrates commercial language models to extract nuanced information from scraped content.
I want to make Scraipe useful for the broader community. The main feedback I'm looking for is:
- For my use case, I compiled web links from large datasets so web crawling was unnecessary. Would Scraipe be useful for you without web crawling?
- What use cases do you have for analyzing website content with LLMs?
- What challenges have you faced in your current scraping workflows?
- What new features or integrations would you most like to see added to Scraipe? (e.g., whatsapp, x.com, or other integrations)
If you're interested in contributing, please let me know too. My goal is to build Scraipe to maturity and fill a niche in the python ecosystem.
snpm•9mo ago
Web demo: https://scraipe.streamlit.app/
I want to make Scraipe useful for the broader community. The main feedback I'm looking for is:
- For my use case, I compiled web links from large datasets so web crawling was unnecessary. Would Scraipe be useful for you without web crawling? - What use cases do you have for analyzing website content with LLMs? - What challenges have you faced in your current scraping workflows? - What new features or integrations would you most like to see added to Scraipe? (e.g., whatsapp, x.com, or other integrations)
If you're interested in contributing, please let me know too. My goal is to build Scraipe to maturity and fill a niche in the python ecosystem.