Hey, thanks for the question.

So from my experience with the LLMs if you ask them directly "is this a bug or a feature" they might start hallucinating and assume stuff that isn't there.

I found in a few research/blog posts that if you ask the LLM to categorize (basically label) and provide score in which category this issue belongs it performs very very well.

So that's exactly what this tool does, when it sees the failing test it formulates the prompt in a following way:

## SOURCE CODE UNDER TEST: ## FAILED TEST CODE: ## PYTEST FAILURE FOR THIS TEST: ## PARSED FAILURE INFO: ## YOUR TASK: Perform a deep "Step-by-Step" analysis to determine if this failure is: 1. *hallucination*: The test expects behavior, parameters, or side effects that do NOT exist in the source code. 2. *source_bug*: The test is logically correct based on the requirements/signature, but the source code has a bug (e.g., missing await, wrong logic, typo). 3. *mock_issue*: The test is correct but the technical implementation of mocks (especially AsyncMock) is problematic. 4. *test_design_issue*: The test is too brittle, over-mocked, or has poor assertions.

Then it also assigns the "confidence" score to it's answer, based on that either full regeneration of the tests proceeds, commenting on the bug in the test, fixing mocks or full test redesign (if it's to brittle)

While this is not 100% bullet proof, i found this to be quite effective way - basically using LLM for the categorization.

Hope that answers your question!

Hey, thanks for the feedback, i will make sure to make it more visible/less confusing. So the model is actually quite simple.

1 credit - 1 file up to 15 functions. <-- only this tier is available in alpha, due to current limitations in the implementation, i tried generating on bigger files and it took quite a long time, so i am in the workings on solving this issue before enabling larger files support.

2 credits - 1 file up to 30 functions. 3 credits - 1 file 30-35 functions.

P.s if generated tests have <70% pass rate (at which point probably something went horribly wrong, your credits are refunded)

Hope this answer clears things up!

I actually though about it multiple times over at this point.

You're right, this deserves more attention, and is a valid problem going forward with this app. And I had this problem when just started building, it either generated XSS tests for any user input validation method (even if it used other validators) or just 1 single test case.

For now I attempt to strictly limit the amount of tests for LLM to generate.

This is achieved with "Planner" that plans the tests for each function before any generation happens, that agent is instructed to generate a plan that follows the criteria:

- testCases.category MUST be one of "happy_path" | "edge_case" | "error_handling" | "boundary".

And it is asked to generate 2-3 tests for each category. While this may result in the unnecessary tests, it at least tries to limit the amount of them.

Going forward I believe the best approach would be to tune and tweak the requirements based on the language/framework it detects.

Do a structured code review, with a few passes by Claude or Codex. Have it provide an annotated justification for each test, and flag tests with redundant, low, or no utility within the context of the rest of the tests. Anything that looks questionable to you, call it out on the next pass, and if it's not justified by the time you fully understand the tests, nuke it.

You could automate this, but you'll end up getting rid of useful tests and keeping weird useless ones until the AI gets better at nuance and large codebases.

What I see a lot is a generated test for something I prompt, and the test passes. Then I manually break the test and it fails for a different reason, not what I wanted to verify.

Guess I need to make it generate negative tests?

Indeed, the Microsoft Copilot eco-system might be a bit more sophisticated these days.

It so just happens than people around me, including myself, don't use the copilot, we "left" for the next big thing when Cursor was release, and copilot was still a glorified auto-complete.

From your feedback it seems like they became quite good?