Our focus with the dynamic sandbox execution is aimed at finding bugs that are much harder for static analysis to detect. These are bugs like logical flaws in specific execution paths and unexpected interactions between code changes.

Our approach with Jazzberry is specifically focused on the dynamic execution aspect within the PR context. I am seeing that by actually running the code with the specific changes, we can get a clearer signal about functional errors. We're very aware of the need to demonstrate our ability to find those high-severity/exploitable bugs you mentioned, and that's a key metric for us as we continue to develop it.

Given your background, I'd be really interested to hear if you have any thoughts on what approaches you think might be most promising for moving beyond the false positive problem in AI-driven bug finding. Any insights from your work at MIT would be incredibly valuable.

We think that LLMs are able to capture semantic bugs that traditional software testing cannot find in a hands-off way, and ideas from both worlds will be needed for a truly holistic bug finder.

That said, I have found LLMs can find bugs in binaries. It’s not all false positives, as far as I can tell. I have a side project I’ve been working on that does just this (shameless plug): PwnScan.com. It’s currently free and focused on binaries.

The bad news is that you quickly get into a situation where you have too many false positives where it’s sometimes not feasible to sort through them all.

You’re stating the problem with that whole sector.

I really wish product owners, researchers and tool creators had more actual real world experience on the remediation side. I think that’s the reason we have so many crappy tools.

- We need a better way of addressing business logic issues and sensitive data leakage which starts at the data model and flows from there.

- Within large organizations we need better risk data about vulns to aid with prioritization and remediation which is always the larger problem (sifting through noise)

- We need automated threat modeling tools that reduce a teams need to start from zero

Fundamentally a tool is a waste of time if it can’t tell you there’s “x% possibility of downtime or sensitive data leakage.”

Addressing the risk equation (r=il) where the impact and likelihood variables are baked into every tool and based in real world data is where we should be.

Until then, vulnerability scanning and management will continue to suck.

Jazzberry starts by viewing the patch associated with the PR, but can choose to view any file in the repo as it searches for bugs. All the models in use are off-the-shelf for now, but we have been thinking about training a model from the beginning, stay tuned!

There are some other really cool tools built explicitly for this, like QualGent and Operative.sh