Super Simple "Hallucination Traps" to detect interview cheaters

15•EliotHerbst•7h ago

After testing out Cluely with my team, we suspect that the easiest way to detect interview cheaters is to set simple "hallucination traps" where you ask a question that sounds plausible, but any knowledgeable person would instantly identify as a joke, fake, or just simply say they don't know. Vibe coded a simple app demonstrating the concept - https://beatcluely.com/

Here are some examples of this class of prompts which currently work on Cluely and even cause strong models like o4-mini-high to hallucinate, even when they can search the web:

https://chatgpt.com/share/6865d41a-c720-8005-879b-d28240534751 https://chatgpt.com/share/6865d450-6760-8005-8b7b-7bd776cff96b https://chatgpt.com/share/6865d578-1b2c-8005-b7b0-7a9148a40cef https://chatgpt.com/share/6865d59c-1820-8005-afb3-664e49c8b583 https://chatgpt.com/share/6865d5eb-3f88-8005-86b4-bf266e9d4ed9

Link to the vibe-coded code for the site: https://github.com/Build21-Eliot/BeatCluely

Comments

nrds•5h ago

What do you think is wrong with

> How do you implement a recursive descent algorithm for parsing a JSON file?

That is a 100% reasonable interview question. It's not _quite_ how I would phrase it, but it's not out of distribution, as it were.

EliotHerbst•3h ago

You are completely correct, great catch, that's a (non-AI) hallucination on my part.

leakycap•5h ago

Maybe I don't have the interview volume others do, but aren't you able to tell pretty quickly in your face-to-face or live video call interview that a person is competent or not (such as using a tool to compensate for a lack of experience)

I keep hearing of employers being duped by AI in interviews; I don't see how it is possible unless:

1) The employer is not spending the time to synchronously connect via live video or in person, which is terrible for interviewing

2) The interviewer is not competent to be interviewing

... what other option is there? Are people sending homework/exams as part of interviews still and expecting good talent to put up with that? I'm confused where this is helpful to a team that is engaged with the interview process.

interneterik•5h ago

This is an example that comes to mind where someone can pull of cheating with AI in a realtime interview: https://techcrunch.com/2025/04/21/columbia-student-suspended...

leakycap•5h ago

I'm familiar with this story, this is the person who founded the software being discussed/linked... but what does this do to explain why a competent interviewer was unable to suss out that the person had no idea what they were doing?

Bluffing in interviews is nearly a given. Your interview should be designed to suss out the best fit; the cheaters should not even rank into the final consideration if you did a decent interview and met the person via some sort of live interaction.

EliotHerbst•4h ago

You’re right, a competent interviewer can likely suss out that a person is cheating - but it can depend on the type of interview and role. This can help erase any doubt, as if you are not familiar with what is being discussed, it is hard to differentiate this type of question. We found that some of our existing interviews for roles like technical support could be “cheated” using Cluely to some degree, when asking questions about solving example support issues which might have troubleshooting steps in an LLMs training set and if the interviewee is someone who is loosely familiar and presenting as being more familiar with the topics.

Before these sort of tools [Cluely], there wasn’t a good way that I'm aware of to cheat on this type of question and respond without any interruption or pause in the conversation.

In real support situations, the tool is not useful as you could pass a major hallucination on to a customer, of course.

Reubend•5h ago

Cool idea. Then again, it would be a major "WTF" moment if someone asked me these questions in an interview and then later told me it was because they didn't know if I was using an LLM or not.

Kemschumam•3h ago

My team has been kicking around the idea of using images to trip up candidates using some kind of AI in their ear.

Things like diagrams and questions written on paper the held up to the webcam.

derbOac•3h ago

It's interesting to me that these models confabulate so readily; I'm curious why it happens at all.

Llamamoe•2h ago

Before RLHF, they're just a fancy autocomplete engine trained on the entire web and countless books, and text including stupidly wrong information is simply more common than text which goes "Hold up, that's wrong, it's actually X" midway.

Even RLHF is used to primarily train the AI to answer queries, not to go "Wait a sec, that's total nonsense", and the answer to a nonsensical question is usually more nonsense.

Show HN: Bookmark and organise your mobile links with ease with this free app

Albumentations: Licensing Change and Project Fork

Recreating Laravel Cloud's range input with native HTML

When do pattern match compilation heuristics matter?

Guaranteeing post-quantum encryption in the browser: ML-KEM over WebSockets

Trusting the Boot Process: Inside Bottlerocket's Security Architecture

2-D Digital Waveguide and Finite Difference Modeling of a Sitar (2015) [pdf]

Tenkai: AI-powered no-code platform to extract structured web data from any site

Show HN: Open Dog Registry – free, open-source API for 200 dog breeds

All Rocket launches in 2025 so far, chronologically and to scale

ChatGPT creates phisher's paradise by recommending the wrong URLs

Nintendo locked down the Switch 2's USB-C port and broke third-party docking

Takens Embedding Theorem

Apple is reportedly working on a cheaper MacBook with an iPhone processor

Show HN: Managing VectorDB via Natural Language

The Koka Programming Language

Show HN: VDBbench 1.0: open-source benchmarking for VectorDBs

Monkeys, typewriters, and busy beavers

Why Are SaaS Boilerplates Still This Expensive? So I Built My Own

Show HN: Why not more sales from the same traffic?

Frictionless Effect: Why Brave, Cursor and other "Baked-In" tools keep winning

This Week in Rust 606

Westeros Inside Minecraft

A data center company uses stranded renewable energy

Smoothdev.io – AI Commit Message Generation for Developers

Qualcomm: Release fully-free drivers for modern Wi-Fi chipsets

We Launched Soham AI SDK

Astronomers Detect New Object That May Have Originated Outside the Solar System

Man says ChatGPT sparked a 'spiritual awakening'. Wife says threatens marriage

Remote Debugging over Network for VM