Show HN: A GitHub Action that quizzes you on a pull request

94•dkamm•6mo ago

A little idea I got from playing with AI SWE Agents. Can AI help make sure we understand the code that our AIs write?

PR Quiz uses AI to generate a quiz from a pull request and blocks you from merging until the quiz is passed. You can configure various options like the LLM model to use, max number of attempts to pass the quiz or min diff size to generate a quiz for. I found that the reasoning models, while more expensive, generated better questions from my limited testing.

Privacy: This GitHub Action runs a local webserver and uses ngrok to serve the quiz through a temporary url. Your code is only sent to the model provider (OpenAI).

Comments

frenchie4111•6mo ago

Next week on HN... Show HN: A GitHub Action that uses AI to answer PR quizzes

dkamm•6mo ago

Cluely 2.0

sunrunner•6mo ago

> AI Agents are starting to write more code. How do we make sure we understand what they're writing?

This is a good question, but also how do we make sure that humans understand the code that _other humans_ have (supposedly) written? Effective code review is hard as it implies that the reviewer already has their own mental model about how a task could/would/should have been done, or is at the very least building their own mental model at reading-time and internally asking 'Does this make sense?'.

Without that basis code review is more like a fuzzy standards compliance, which can still be useful, but it's not the same as review process that works by comparing alternate or co-operatively competing models, and so I wonder how much of that is gained through a quiz-style interaction.

dkamm•6mo ago

I imagine the quizzer could ask better questions along those lines with better context engineering (taking entire repo contents, design docs, discussions, etc and compressing those into a mental model). I just took the PR code changes and comments, so there's a lot of improvements that could be made there.

shortrounddev2•6mo ago

Code review, to me, is not about validating the output. It's about a 2nd set of eyes to check for foot guns, best practice, etc. Code review is one step above linting and one step below unit tests, for me.

If someone were to submit this code for review:

    getUser(id: number): UserDTO {
        return this.mapToDTO(this.userModel.getById(id));
    }

and I knew that `userModel` throws an exception when it doesn't find a user (and this is typescript, not java, where exceptions are not declared in the method prototype) then I would tell them to wrap it in a try-catch. I would also probably tell them to change the return type to `UserDTO | null` or `Result<UserDTO>` depending on the pattern that we chose for the API. I don't need to know anything about the original ticket in order to point these things out, and linters most likely won't catch them. Another use for code review is catching potential security issues like SQL injection that the linter or framework can't figure out (i.e, using raw SQL queries in your ORM without prepared statements)

mathieuh•6mo ago

Depends how good your QA is. Where I am it is terrible so most of the time I spend in “code review” is spent checking out the code locally and testing it myself.

shortrounddev2•6mo ago

Yes, this is all on paper. Where I work we don't have QA

johann8384•6mo ago

This lines up with my experience, sometimes it is as simple as "Your way is fine, but we did it this other way over here, and over here, should we make it consistent, even if it is consistent but not as good" or as you pointed out, looking for footguns. I also like the supervillian model of "Show this to an average 5 year old and see what obvious flaw they point out".

donatj•6mo ago

See, I think this is a good idea even for reviewing non-agentic human-written PRs!

We've got a huge LGTM problem where people approve PRs they clearly don't understand.

Recently we had a bug in some code of an employee that got laid off. The people who reviewed it are both still with the company, but neither of them could explain what the code did.

That triggered this angry tweet

https://x.com/donatj/status/1945593385902846118

dkamm•6mo ago

Could definitely be used for human PRs too! Though I'm sure companies would love to track the reviewer scores

SamuelAdams•6mo ago

The only way I’ve ever seen engineers care about PR’s is if the software or product is tied directly to their paycheck. If uptime or bugs directly impact a quarterly bonus, or result in a layoff / getting fired, they spend a lot more time reviewing PR’s. Furthermore, the work and its estimate is expanded to include enough time for the team to thoroughly review the change.

Unless someone is getting fired for bad code the “lgtm” culture will never die.

robotsquidward•6mo ago

What a fun world we devs now live in.

brianjlogan•6mo ago

Remember non-devs are affected just as much by this "new world". Perhaps even worse because they don't understand what's going on.

rmnclmnt•6mo ago

That’s a fun take on a real issue, but…

> Your code is only sent to the model provider (OpenAI)

When has this become an acceptable « privacy » statement?

I feel we are reliving the era of free mobile apps at the expense of harvesting any user data for ads profiling before GDPR kicked in…

stronglikedan•6mo ago

That's not the privacy statement though. I feel like we're reliving the era of RTF... oh wait, we never left.

rmnclmnt•6mo ago

Ok I’ll bite: putting « only » implies this is not a big deal and a lesser of 2 evils, between an AI model provider harvesting prompts for retraining and a 3rd party hosting provider most probably only storing logs for security and accountability…

So yes this is the second part of the privacy statement

throwaway889900•6mo ago

Just submit a PR that removes the action so it doesn't run on the branch before the merge! If devs aren't reviewing the code anyways, will they even catch that kind of change?

xmprt•6mo ago

You could set up some hardcoded rules so that the PR is never merged without human review if it touches the github actions.

LikesPwsh•6mo ago

You could, but it would be mad to skip the code review because it "only" touches customer-facing code rather than GHA.

Xss3•6mo ago

I would probably be putting devs on a pip or firing them if they failed these quizzes often...understanding your own prs is the bare fucking minimum, even without AI help.

inetknght•6mo ago

Won't be long before those people would just get AI to answer the quiz instead.

LtWorf•6mo ago

What makes you think the AI can instead generate the correct answers to double check the developer's answers?

ElijahLynn•6mo ago

This could actually be quite useful.

hk1337•6mo ago

Cute but I wouldn't actually use it.

henriquegodoy•6mo ago

can i automate the process of answering this pr questions too?

bfung•6mo ago

That was my first reaction: now I gotta build a gpt wrapper, oops, I mean agent, to answer questions to this quiz

waynesonfire•6mo ago

Nice! A quiz to ensure you understand your vibe code.

azhenley•6mo ago

I had an NSF grant for a similar project in 2019. Ask the dev questions about their code and validate their answers using program analysis.

The initial idea was applied to classroom settings.

An Inquisitive Code Editor for Addressing Novice Programmers’ Misconceptions of Program Behavior https://austinhenley.com/pubs/Henley2021ICSE_Inquisitive.pdf

h4ck_th3_pl4n3t•6mo ago

This action assumes that LLMs know what they're coding.

They don't, that's why we need the PR in the first place.

drunken_thor•6mo ago

We now are making bots to quiz other bots. This is a nightmare.

klntsky•6mo ago

LLMs are quite bad at understanding intent behind the code if it is original and involves math-heavy tricks. But for most apps it will probably be fine. What's the workflow if it makes a mistake though?

gpi•6mo ago

Is this captcha but for PRs?

tr_user•6mo ago

Saw an actual PR that says "this was generated with claude, please review carefully". Since when did we stop taking responsibility for what is submitted?

Tiny C Compiler

SectorC: A C Compiler in 512 bytes

Speed up responses with fast mode

Show HN: LocalGPT – A local-first AI assistant in Rust with persistent memory

Brookhaven Lab's RHIC concludes 25-year run with final collisions

Software factories and the agentic moment

OpenCiv3: Open-source, cross-platform reimagining of Civilization III

Stories from 25 Years of Software Development

Hoot: Scheme on WebAssembly

FDA intends to take action against non-FDA-approved GLP-1 drugs

First Proof

Vocal Guide – belt sing without killing yourself

Show HN: I saw this cool navigation reveal, so I made a simple HTML+CSS version

Show HN: A luma dependent chroma compression algorithm (image compression)

Show HN: Craftplan – Elixir-based micro-ERP for small-scale manufacturers

Eigen: Building a Workspace

Al Lowe on model trains, funny deaths and working with Disney

The F Word

Start all of your commands with a comma (2009)

Microsoft account bugs locked me out of Notepad – Are thin clients ruining PCs?

I write games in C (yes, C) (2016)

The AI boom is causing shortages everywhere else

Selection rather than prediction

Reinforcement Learning from Human Feedback

Unseen Footage of Atari Battlezone Arcade Cabinet Production

Learning from context is harder than we thought

Where did all the starships go?

Show HN: Look Ma, No Linux: Shell, App Installer, Vi, Cc on ESP32-S3 / BreezyBox

Hackers (1995) Animated Experience

72M Points of Interest

Tiny C Compiler

SectorC: A C Compiler in 512 bytes

Speed up responses with fast mode

Show HN: LocalGPT – A local-first AI assistant in Rust with persistent memory

Brookhaven Lab's RHIC concludes 25-year run with final collisions

Software factories and the agentic moment

OpenCiv3: Open-source, cross-platform reimagining of Civilization III

Stories from 25 Years of Software Development

Hoot: Scheme on WebAssembly

FDA intends to take action against non-FDA-approved GLP-1 drugs

First Proof

Vocal Guide – belt sing without killing yourself

Show HN: I saw this cool navigation reveal, so I made a simple HTML+CSS version

Show HN: A luma dependent chroma compression algorithm (image compression)

Show HN: Craftplan – Elixir-based micro-ERP for small-scale manufacturers

Eigen: Building a Workspace

Al Lowe on model trains, funny deaths and working with Disney

The F Word

Start all of your commands with a comma (2009)

Microsoft account bugs locked me out of Notepad – Are thin clients ruining PCs?

I write games in C (yes, C) (2016)

The AI boom is causing shortages everywhere else

Selection rather than prediction

Reinforcement Learning from Human Feedback

Unseen Footage of Atari Battlezone Arcade Cabinet Production

Learning from context is harder than we thought

Where did all the starships go?

Show HN: Look Ma, No Linux: Shell, App Installer, Vi, Cc on ESP32-S3 / BreezyBox

Hackers (1995) Animated Experience

72M Points of Interest

Show HN: A GitHub Action that quizzes you on a pull request

Comments