• Author
    Benoît Rondeux
  • Date
    19.11.2026
  • Reading time
    5 min
  • Categories
    AI
    UX

User Testing, Bugs & AI Agents

User Testing, Bugs & AI Agents

Everyone agrees that testing a product, a website or an app with real users is the holy grail. Yet when the budget meeting hits, debugging walks away with the cash.

Why? A bug hits you in the face. A flawed conversion funnel bleeds leads in silence.

And let’s be honest, even with a solid methodology like EPIC’s (a bit of self-praise, I admit!), the final weeks of a web project still often resemble a bug hunt: the form that won’t submit on Safari, the filter that clears the page, or the shopping basket that loses an item when reloaded, and so on.

Result: the budget set aside for user testing gets shoved into the “testing and launch” bucket.

The rise of AI agents is changing the math, tilting the balance back toward UX.


Debugging now costs just €0.15

Bugs are a nightmare because they’re obvious and they’re expensive. For example, at Microsoft, a study carried out in 2025 involving 484 developers showed that they spend nearly half a day each week fixing them.

To cut that spend, teams deploy end-to-end (e2e) tests. A bot replays the full user journey in the browser, just like a customer: open the product page, add to cart, pay, get confirmation. Until recently, only big-budget projects ran them because each test demanded hours of coding and constant updates after design tweaks.

But all this is changing, fast. This summer Uber published a demo showing how an agent can handle the job. The bot grabs the intent (“book a ride”), navigates the interface, and blocks any code change that breaks the workflow. A test that once took about a hundred hours now clocks in under four.

On a smaller scale, a university study had an agent write end-to-end (e2e) tests for nine websites. Up to seven out of ten were usable as they were, at a cost of around 15 cents per test. The fifty key workflows on an e-commerce site now cost about as much as a sandwich. Running them with every update still costs a few dollars per run, but nobody writes them by hand anymore.

Caveat: easy to spin up, they still need a watchful eye.

On a complex flow, Slack found nearly half the generated tests failed. When a dev fixes a broken test, a fresh benchmark says one in four still pass by checking the wrong thing.

Flip the script and the bots spot more breakage. Uber simulated failures across 47 critical workflows and dug up 23 risks no human would have scripted; twelve of them flat-out blocked users from booking a ride.


From debugging to UX: the shortcut you shouldn’t take!

On October 23, the Nielsen Norman Group, the global authority on UX research, dropped its annual survey on how 521 UX designers and researchers use AI at work. The results show AI mainly helps them write code and run literature reviews, and is used least with users: six out of ten practitioners have never used it for an interview, and half have never used it for a user test.

Why? Because debugging a site and grasping its users are two very different jobs.

Why shouldn’t AI be used to test the user experience? Some platforms already offer ‘synthetic users’, generated personas that can be interviewed in place of a panel. The Nielsen Norman Group ran the test in October. It asked 604 professionals how they titled their roles and got more than 50 answers. It posed the same question to 120 AI avatars and got a dozen replies, with the majority matching the humans.

The tool nailed the popular answer we already knew and filtered out the other forty. It also ironed out role differences: 93 percent of fictional product managers picked the dominant title, versus 44 percent of humans. A synthetic respondent gives the average answer, or rather the idea we have of a segment.

These tools reproduce what is already written somewhere, and when applied to established usability principles, they perform very well.

Baymard, the e-commerce audit shop, hands its AI the rules it nails 95 percent of the time—half the playbook.

Think of it as the spell-checker in your word processor: it catches every dictionary error but happily ships a sentence that means the opposite of what you intended, because the grammar is clean.

Listen to and observe your users!

Take the checkout process, for example. In Baymard’s surveys, 18 percent of shoppers say they abandoned a shopping basket because they were forced to create an account. When we watch people place test orders, we spot a different pattern: the ‘checkout as a guest’ option exists, yet on nearly half the sites it hides behind the ‘create an account’ button, and the shopper bails without ever spotting it.

Surveys surface the stated reason; observation uncovers the real one.

We saw it firsthand in 2021 with Camber, the Belgian maker of bespoke storage solutions, after its site relaunch. UX Lab, the Liège usability shop founded by the incomparable and much-missed Rodolphe Finamore (RIP), hosted nine participants for an hour each as they navigated the site.

When they hit the “Showrooms” section, nearly half hesitated. Some imagined a 3D virtual tour. Others expected catalog photos. One clicked and said, “I would have said ‘our shops’.” The term is accurate, competitors use it, and no agent would have spotted it. Camber kept it as a branding choice and immediately reworked the wording of two other sections that the same participants found unclear.

Nine people spotted these three issues. The most cited UX stat pushes it further: five participants catch 85 percent of problems.

Ideally you’d need 45 to exceed 90 percent coverage and catch the issues that affect only one in twenty visitors, but that’s expensive. Mix five carefully picked participants from your core segments with smart AI tests and you’re golden.


A roadmap for testing with AI

Back in October, we told you to pour the time AI frees up on production back into the creative process. On the web, same rule: minutes saved on functional testing go straight into UX testing, the kind that makes or breaks the experience.

You have to decide to do it, because the data shows this time doesn’t reinvest itself. Google’s DORA survey, which follows thousands of dev teams each year, showed as early as 2024 that more AI adoption slightly cut the time spent on work the teams themselves called useful. At Atlassian, developers surveyed in 2025 poured the savings into code, features, and docs. The user? Nowhere to be found.

The 2025 edition of the same survey adds a nuance: in teams that base their decisions on the user’s needs, AI boosts performance, whilst in others, it reduces it.

In practice, here’s how we tackle testing, UX, and AI on a site where every conversion counts:

  • Hand the functional stuff to testers. Which user flows can automated tests cover? Regression testing (things that worked yesterday but break after an update), forms, the shopping basket, browser compatibility, and machine-verifiable accessibility rules. Always pair the agent with a human who reviews its fixes.
  • Have the AI prep UX tests. Interview guides, test plans, sorting session recordings, initial audit of known rules: AI saves researchers time, and that’s what practitioners interviewed by the Nielsen Norman Group are already handing off.
  • Keep real customers in the money zones. Product page, basket, checkout, internal search: five smart participants, on a regular cadence, from the segments that matter.
  • Reinvest the hours saved. Route some of the time support agents free up for developers into a dedicated budget line for these tests. It won’t write itself.

Agents find what’s already been written somewhere.
As for what hasn’t been written yet, only your customers can show you. Go and listen to them.