Ask HN: How do you search the web programmatically these days?

2•coreyp_1•9h ago

For the first time in a long time, I need to query a search engine programmatically, and found that most of them block the use of curl, etc.

So, my question is simple: how do you solve the problem? I've tried searxng with mediocre success, but it seems a bit heavy to have to be running a complete separate service for this one thing that I only need every once in a while. I haven't tried using a service that requires an API key, simply because I'm not sure which direction to go or who to go with.

Just thought I would ask here first.

Comments

pwg•9h ago

> and found that most of them block the use of curl

Try again, but have curl provide a user agent string from one of the real browsers. You'll likely find that the request goes through.

dserban•4h ago

https://pypi.org/project/ddgs/

(Assuming you prefer Python.)

raw_anon_1111•18m ago

Can’t speak for search engines specifically. But I recently had to do a project which required me to crawl the customer’s large site and index it into a vector search for RAG for a call center.

My first attempt was to use crawl it just by doing GET requests (ie same thing as using curl). That got me nowhere. I had to use headless Chrome and Playwright.

Do any modern websites work with just curl even if they don’t block it - ie without being able to run JS?

Gmail label bridge on Claude Cowork just broke

Do I Stop Learning Coding? DSA?

Ask HN: Building a solo business is impossible?

Stop using naive RAG – adding relationships to AI context

Ask HN: Who is using OpenClaw?

Tell HN: Security Incident at Porter (YC S20)

Tell HN: Fiverr left customer files public and searchable

Ask HN: How do you search the web programmatically these days?

Tell HN: 48 absurd web projects – one every month

Ask HN: Teaching life skills through games, am I crazy?

Ask HN: How do you maintain flow when vibe coding?

Ask HN: Getting depressed day by day, how to cope?

Ask HN: How did you get your first users with zero audience?

Aliens.gov Resolves – To a WordPress "Site Not Found" Error

Ask HN: How do you find motivation to do stuff?

Ask HN: How are you using LLMs in production?

Advice for tracking down a listening device?

Opus 4.7 is horrible at writing

Ask HN: Who is your favourite Entrepreneur/Visionary?

Durable Object alarm loop: $34k in 8 days, zero users, no platform warning

Ask HN: How are you actively keeping your thinking sharp while using LLMs daily?

Tell HN: Anthropic no longer allows you to fix to specific model version

Ask HN: Is Claude Getting Worse?

Ask HN: How to highlight talent from untraditional backgrounds?

Ask HN: As an Australian, is it possible to get a remote US role?

GitHub gave webhook secrets away in webhook call

Ask HN: SeedLegals Partnerships in London, worth it?

Ask HN: LeetCode, anyone still doing it?

Tell HN: GitHub might have been leaking your webhook secrets. Check your emails.

Any engineers here with experience of clinical data standards?

Ask HN: How do you search the web programmatically these days?

Comments

Gmail label bridge on Claude Cowork just broke

Do I Stop Learning Coding? DSA?

Ask HN: Building a solo business is impossible?

Stop using naive RAG – adding relationships to AI context

Ask HN: Who is using OpenClaw?

Tell HN: Security Incident at Porter (YC S20)

Tell HN: Fiverr left customer files public and searchable

Ask HN: How do you search the web programmatically these days?

Tell HN: 48 absurd web projects – one every month

Ask HN: Teaching life skills through games, am I crazy?

Ask HN: How do you maintain flow when vibe coding?

Ask HN: Getting depressed day by day, how to cope?

Ask HN: How did you get your first users with zero audience?

Aliens.gov Resolves – To a WordPress "Site Not Found" Error

Ask HN: How do you find motivation to do stuff?

Ask HN: How are you using LLMs in production?

Advice for tracking down a listening device?

Opus 4.7 is horrible at writing

Ask HN: Who is your favourite Entrepreneur/Visionary?

Durable Object alarm loop: $34k in 8 days, zero users, no platform warning

Ask HN: How are you actively keeping your thinking sharp while using LLMs daily?

Tell HN: Anthropic no longer allows you to fix to specific model version

Ask HN: Is Claude Getting Worse?

Ask HN: How to highlight talent from untraditional backgrounds?

Ask HN: As an Australian, is it possible to get a remote US role?

GitHub gave webhook secrets away in webhook call

Ask HN: SeedLegals Partnerships in London, worth it?

Ask HN: LeetCode, anyone still doing it?

Tell HN: GitHub might have been leaking your webhook secrets. Check your emails.

Any engineers here with experience of clinical data standards?