Hey I'm Matt! Really excited to answer any questions.

To elaborate a little bit on the "canary" comment --

For a while at Airtable I was on the infra team that managed the deploy (basically click run and then sit and triage issues for a day), One of my first contributions on the team was adding a new canary analysis framework that made it easier to catch and rollback bugs automatically. Two things always bothered me about the standard canary release process:

1) It necessarily treats some users as lower value, and thus more acceptable to risk exposing bugs to (this makes sense for things like free-tier, etc. but the more you segment out, the less representative and thus less effective your canary is). When every customer interaction matters (as is the case for so many types of businesses) this approach is harder to justify

2) Low frequency / high impact bugs are really difficult to catch in canary analysis. While it’s easy to write metrics that catch glaring drops/spikes in metrics, more subtle high impact regressions are much harder and often require user reports (which we did not factor in as part of our canary). Example: how do you write a canary metric that auto rolls back when an enterprise account owner (small % of overall users) logs in and a broken modal prevents them from interacting with your website.

I view what we’re building at Propolis as an answer to both of these things. I envision a deploy process (very soon) that lets us roll out to simulated traffic and canary on THAT before you actually hit real users (and then do a traditional staged release, etc.)

I've been looking for this exact thing. A couple questions:

Are your agents good at testing other agents? e.g. I want your agent to ask our agent a few questions and complete a few UI interactions with the results.

How do you handle testing onboarding flows? e.g. I want your agent to create a new account in our app (https://www.definite.app/) and go thru the onboarding flow (e.g. add Stripe and Hubspot as integrations).

fraud/abuse/compliance is a good usecase for this kinda thing - an abuse vector is kinda like a bug, except that the system does what its expected to do.

testing for abuse stuff ive always found quite difficult, since to work well, you need to both create some real resources so you can delete/clean them up, and also you need to create a new test identity, since your abuse detection system should be deny listing found bad actors. the difficulty is that those sessions probably want to be open for like a week, so they can process both payments and refunds.

can the agents check their email? other notification methods?

Looks like a great idea. Does this fully automate QA testing of websites including removing the human in the loop during testing?

Once again, great product.

Good video, but it looks like it plays twice. Should be ~3.5 minutes...

This seems really interesting. I tried running a swarm on my landing page but didn't get a completed email. I'll try it again, though!

Sounds interesting. Is this handling mobile as well?

Can it find broken UI?

Human can find and report broken UI easily by using common sense.

Even though it is simple for human. Computer has no common sense and I am a machine learning expert. I tried and mostly failed to build a broken UI detector in my previous company. They had automated plugin upgradable process. That periodically broke UI.

I tried to detect it my taking long screenshot, and you could select a image as working version, then later finding diff between 2 images. I kind of worked but not satisfactory.

Does it output playwright scripts?

Neat! How do you handle state changes during tests, for example, in a todo app the agents are (likely) working on the same account in parallel or even as a subsequent run, some test data has been left behind or now data is not perhaps setup for a test run.

I’m curious if you’d also move into API testing too using the same discovery/attempt approach.

Been looking for a solution exactly like this but I struggle to see how it is different than spinning 10 Atlas tabs with a 2 sentence prompt.

Looks interesting! How it would work in terms of latencies? Also, would you be able to run on CI, or even with ephemeral envs? From my experience it is key to be able to run it on each change.

This is a great idea and definitely useful, in particular this would make sense if it can take the build artifacts of recent github action builds and test them comprehensively, particularly to green-light a release. You can already pretty much do this using the standard agent tools and a set of test prompts so anything that makes it easier and repeatable is good.

The pricing sounds quite enterprisey, the risk there is that people will tend towards building their own.

Great to see others working on the problem of validating UI.

We are also building a Web QA agent at https://kodefreeze.com. We are focused on the small and medium sized companies and are offering free usage during our trial period!

    Error loading video
    Please try refreshing the page

    Capturing error: Error: WHEP request failed: 500 - {"message":"\"message\" is required!","error":"Server Error"}

Oh this is sick! I've been complaining about this exact problem for years. The "canary without real users" idea is brilliant - way better than just throwing your free tier users under the bus and hoping for the best.

The thing that really got me was catching bugs in non-deterministic output. We've been struggling with this on LLM features where traditional assertions just don't work. Having agents actually judge quality instead of looking for exact matches is such an obvious solution in hindsight.

Quick question though - how do you handle auth flows with MFA or OAuth redirects?

I did a trial run with a poorly chosen URL. Wanted to rerun with a better one. When i click "Launch Swarm" nothing happens. Am I done? Maybe there should be a message displayed?