Show HN: Slop or not – can you tell AI writing from human in everyday contexts?

5•eigen-vector•1h ago

I’ve been building a crowd-sourced AI detection benchmark. Two responses to the same prompt — one from a real human (pre-2022, provably pre prevalence of AI slop on the internet), one generated by AI. You pick the slop. Three wrong and you’re out.

The dataset: 16K human posts from Reddit, Hacker News, and Yelp, each paired with AI generations from 6 models across two providers (Anthropic and OpenAI) at three capability tiers. Same prompt, length-matched, no adversarial coaching — just the model’s natural voice with platform context. Every vote is logged with model, tier, source, response time, and position.

Early findings from testing: Reddit posts are easy to spot (humans are too casual for AI to mimic), HN is significantly harder.

I'll be releasing the full dataset on HuggingFace and I'll publish a paper if I can get enough data via this crowdsourced study.

If you play the HN-only mode, you’re helping calibrate how detectable AI is on here specifically.

Would love feedback on the pairs — are any trivially obvious? Are some genuinely hard?

Comments

lucastonelli•59m ago

Hey, congratulations on the final product. It even feels fun. Some are really hard, but some feel blatantly obvious. I don't know why though. I guess it's just because the way we communicate feels off when compared to AI, some times.

eigen-vector•38m ago

Thanks for checking it out! The obvious ones are (hopefully) weaker models :) but yes my experience has been unless you're engaging with human written content consistently the line really blurs easily.

SsgMshdPotatoes•43m ago

Nice idea! Em dashes were giveaways for AIs and typos for human, at least in the ones I did, so those are at least trivial. So might have to do some filtering at least for those.

Some were hard though, yeah (at least if not looking longer than 5-10 seconds). Btw, it seemed more logical to me to just see a green/red card when you click, i.e. right choice or wrong choice. Getting red for the correct answer confused me a bit (but this might just be me).

lucastonelli•33m ago

The coloring is a fair point. I was some times confused if I got the right or the wrong one XD

eigen-vector•31m ago

Thanks for checking it out! The color signal is useful feedback. Let me think about it and rework!

Yeah there are some very obvious tells, but the models that are most capable are very good at writing like human.

Especially when the human responses for reddit or HN prompts were presumably made after reading the content of the article or the post; whilw the model is simply going off of the title.

SsgMshdPotatoes•24m ago

Also for example this one has a giveaway for the human case: "There are lots of great people here at /r/personalfinance" (actually, not sure if that is a giveaway, that was my guess, but depends on how the model was prompted, I guess). And human ones often seem to have two spaces sometimes instead of one, idk why. If you want to get a serious dataset, maybe you could use this one to find all the flaws and perfect it, and then try to get a real dataset from the next one? People will be more eager to help too if they've seen you designed it all very carefully. (Or you could filter the results from this one to make it a good dataset if you get lots of responses.)

eigen-vector•13m ago

You'd be surprised at the nuances we tend to miss :)

This time around I prompted the models not necessarily to be adversarial - i didn't ask them to try and fool the reader. But i gave them contextual info - something to the effect of "you're a user posting on hacker news"

Crypto investor turns $50M into $36,000 in one botched move

US carrying out rescue effort after military aircraft crash in Iraq

Steve Yegge Wants You to Stop Looking at Your Code

Anthropic and OpenAI just exposed SAST's structural blind spot with free tools

US fuel tanker aircraft crashes in Iraq – what we know and don't know

Before you let AI agents loose, you'd better know what they're capable of

To use your brain, first accept the Terms and Conditions

Spacedrive v3: The local-first data engine

I'll probably never use Windows

London Man wore smart glasses for High Court 'coaching'

France's ghost car scandal that allowed one million illegal vehicles onto the

I Built a Modern Million Dollar Homepage with Pixel Wars

How long does it take to get last liquid drops from kitchen containers?

Cameyo by Google: Run any legacy application and turns it into a PWA

Vibe in Go – It's the only way

Trump's AI-Powered World Wars

Live Nation Executives Brag About "Robbing" Ticket Buyers in Slack DMs

The Force Multiplier Generation

Daily multivitamin use may slow biological aging: COSMOS trial results

Fresh Open Claw Documentation

Wvw.dev: world vibe web – A free and OSS federated app store for vibecoded apps

Simulating Catalog and Table Conflicts in Iceberg

DOGE Operative Accused of Planning to Take Social Security Data Is Named

Replit's Jordanian Immigrant Billionaire Founder Shakes Up Vibe Coding

Ask HW: Claude Code design tools

Pi-Autoresearch

The AI coding divide: craft lovers vs. result chasers

Apollo's Private Credit Logic Is a Lot Like Goldman

Show HN: Tokemon, a terminal dashboard to track LLM token usage

Learning Is Forgetting; LLM Training as Lossy Compression