Agents that test mobile apps like real users
"AI-powered QA" usually means a model that writes unit tests from a function signature. That is a code generator. QA is someone opening the app, doing something slightly wrong, and noticing what breaks.
I wanted agents to do that part. Not to reason about the code, but to hold the phone. This is the system that made it possible: real devices and simulators, scenarios written as behaviour, visual automation, and an orchestration layer that turns "test this release" into tickets and fixes while I sleep.
The loop
┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐
│ spec / │──▶│ write │──▶│ run on │──▶│ observe │
│ change │ │ scenario │ │ devices │ │ screen + │
└──────────┘ └──────────┘ └──────────┘ │ logs │
▲ └────┬─────┘
│ │
┌─────┴────┐ ┌──────────┐ ┌──────────┐ ┌────▼─────┐
│ improve │◀──│ learn │◀──│ fix + PR │◀──│reproduce │
│ tools │ │ per app │ │ + regress│ │ twice, │
└──────────┘ └──────────┘ └──────────┘ │ file │
└──────────┘
A change lands. An agent writes or picks the scenarios it touches. The scenarios run on a matrix of devices. The agent watches the screen and the logs, not just the exit code. Anything wrong is reproduced a second time before it becomes a ticket. The ticket comes with steps, a recording and log excerpts. Then the agent proposes the fix, adds the regression, writes down what it learned about this app, and fixes whatever tool made the run harder than it should have been.
The layers
┌──────────────────────────────┐
│ me: charters, priorities, │
│ review of tickets and PRs │
└──────────────┬───────────────┘
│
┌──────────────▼───────────────┐
│ planner ─▶ workers ─▶ judge │
└───────┬──────────────┬───────┘
│ │
┌─────────────▼────┐ ┌─────▼──────────────┐
│ memory │ │ tools │
│ │ │ │
│ scenarios (BDD) │ │ Maestro flows │
│ per-app quirks │ │ adb / simctl │
│ known bugs │ │ screenshot + a11y │
│ device matrix │ │ log + network taps │
└─────────────┬────┘ │ device lease │
│ └─────┬──────────────┘
┌───────▼──────────────▼───────┐
│ evaluation │
│ │
│ reproduce twice │
│ screenshot diff │
│ regression suite │
│ flake tracker │
└──────────────────────────────┘
Three agent roles. A planner turns a charter into scenarios and assigns them to devices. Workers run them, one device each, and report. A judge reads every finding, tries to reproduce it on a different device, and either files it or throws it away. I write charters and read what survives the judge.
Scenarios as behaviour, not clicks
Scripts that say "tap 340,812" break on the next screen size. Scenarios say what a user does and expects.
Feature: buy a position
Scenario: logged-out user on slow network
Given the app is fresh installed
And network is throttled to 3G
When I open a market and tap Buy
Then I see the login sheet
And after login I am back on the same market
And the buy button is enabled within 3 seconds
The agent turns this into a Maestro flow: tap by accessibility label, assert by visible text, wait by condition. The flow is committed next to the feature. Utilities that every flow needs, like login, onboarding and hiding the keyboard, are shared, and the keyboard one exists twice because iOS and Android hide it differently. That detail was learned by an agent on a failing run and became a file.
The agent also writes the scenarios nobody asks for: rotate mid-flow, background and return, kill and relaunch, revoke a permission, deep link into a screen that needs auth, run with the largest font size, run in Portuguese. These are where most of the real bugs were.
What a phone farm does
A phone farm is a rack of devices, USB hubs and a machine that drives them. The classic use is volume: install this build on twenty phones, run the same flow on all of them, collect the screenshots.
host ──▶ adb (usb / tcp) ──▶ device 01 Android 12, 5.5"
──▶ adb ──▶ device 02 Android 14, 6.7"
──▶ adb ──▶ device 03 Android 15, tablet
──▶ simctl ──▶ iPhone SE, iOS 17
──▶ simctl ──▶ iPhone 16 Pro, iOS 18
──▶ simctl ──▶ iPad, iOS 18
per device: install, launch, reset state, set locale,
throttle network, grant / revoke permissions,
screenshot, record, pull logs, reboot
Everything above is a command. Install and uninstall a build. Wipe app data so every run starts fresh. Set the locale, the font scale, the time zone. Throttle the network. Grant or revoke camera, notifications, location. Tap, swipe, type, press back. Screenshot, screen record, pull logcat and the crash reports. Reboot when a device gets weird, which they do.
The farm also handles the boring physical layer: devices that fall off USB, batteries that swell if kept at 100%, thermal throttling that makes a fast phone look slow, one device stuck on a system dialog blocking every run. Each of these got a detector and a recovery, because a farm that needs a person to unplug a cable is not a farm.
What changes when the operator is an agent
A farm run by scripts answers the question you wrote. A farm run by agents answers the question you should have asked.
- It reads the screen. Screenshot plus accessibility tree, every step. A flow can pass while the screen shows a blank list, a wrong currency or a button rendered under the keyboard. Scripts do not notice. The agent does.
- It explores. Given a charter, not a script, it tries the slightly wrong thing: double tap on submit, back during a loading state, a paste with a trailing space in the amount field.
- It reproduces before it reports. Twice, on a second device where possible. Flaky is a category, not a finding, and flakes go to a tracker that points at the tool or the wait condition, not at the app.
- It files tickets a developer can act on. Steps, device, OS, build, a recording, the log window around the failure, and a guess at the cause with the file it suspects.
- It fixes. For a good fraction of findings, the same agent opens a PR with the fix and the regression flow that would have caught it. I review the diff, not the bug.
- It remembers. Per-app quirks, known bugs, which screens are slow, which device is unreliable. The next run starts with that.
finding on device 02
│
├─ reproduce on device 05 ──▶ fails again ──▶ ticket + PR
│ └──▶ passes ──▶ flake tracker
│ └─▶ fix the wait,
│ not the app
└─ record: build, steps, video, logcat, suspect file
Numbers that mattered
Before this, a release took a person most of a day clicking through the same flows on two phones. After it, the matrix above runs on every merge, unattended, and a person reads a short list of confirmed findings in the morning. The interesting measure is not speed. It is that bugs started arriving with a reproduction and a proposed fix instead of a screenshot in a chat.
Why this is not about phones
mobile QA any QA
───────────────────────── ─────────────────────────────
phone farm the environments you test in
BDD scenario behaviour, not coordinates
screenshot + a11y tree observe the output, not the code
reproduce twice a finding is a fact, not a guess
flake tracker blame the tool before the product
fix + regression in one PR the bug closes itself
per-app memory the next run starts smarter
The hard part was never driving a phone. ADB has done that for fifteen years. It was giving agents a way to see, a way to say what should happen, a place to remember, and a judge between them and the ticket queue.
What I did
I built the farm and the device tooling, wrote the first scenarios and the shared flows, set the rule that nothing is filed without a second reproduction, and designed the planner, worker and judge split. Then I stopped clicking through releases and started reading tickets.