A mobile team ships a redesigned checkout on a Thursday, and by Friday morning the nightly suite on the device lab shows forty red tests. Very few of them are bugs. The button label changed from "Place order" to "Pay now," a bottom sheet now slides in a little more slowly, and the Android 13 notification prompt appears one screen earlier than it used to. Someone spends the morning sorting real failures from noise and rewriting locators, and next sprint the profile screen does the same.
The habit behind it sounds reasonable: if a tester checks something twice, automate it. On mobile that gets expensive, because every UI check needs a booted emulator or a free phone, waits on animations and network calls, and breaks whenever product moves a button. The useful question is whether a check will pay back what it costs to keep alive, not whether it can be automated.
This guide walks through what one mobile UI check really costs, which checks to leave with people, which to automate first, how to write assertions that survive a redesign, and a four-factor scorecard for backlog grooming.
What one automated mobile check really costs
Teams usually estimate only the writing. A UI check on a phone carries four costs, and the last two pile up quietly.
Writing it. Locators, seeded accounts, permission state, and OTP waits all take time. A sign-in flow with SMS OTP on Android can take days to stabilize, because the SMS Retriever API, the soft keyboard, and the autofill suggestion strip all compete for the same few hundred pixels.
Waiting for it. A unit test that checks a coupon calculation finishes in milliseconds on the build machine. The same assertion driven through the UI waits for a cold app start, a login, a populated cart, and a checkout screen before it can look at the number. The Android testing strategy guide puts it plainly: big tests have more fidelity but are harder to set up and maintain, so most apps should have many small tests and relatively few big ones. The Flutter testing overview rates integration tests highest for both confidence and maintenance cost.
Triaging it. Every red run forces someone to decide whether the failure is a product bug, a flake, or a stale test. Google published numbers on this years ago in Flaky Tests at Google and How We Mitigate Them: about 1.5% of all test runs reported a flaky result, almost 16% of tests showed some level of flakiness, and around 84% of the pass-to-fail transitions they observed in post-submit involved a flaky test. The post also notes that people start ignoring real failures once a suite cries wolf. Mobile UI tests, with animations, keyboard focus, and OS dialogs landing at unpredictable moments, sit at the touchy end of that range.
Rewriting it. Every time design touches a screen, the tests that walk through it need edits. A checkout redesigned twice a quarter bills you twice a quarter, for every test that crosses it.
To size a proposed check, estimate three numbers: monthly runs, runs that could catch something a person or lower-level test would miss, and monthly hours of triage and edits. If the second is near zero and the third is not, the check is a liability even when green.
Mobile checks that are better left unautomated
Not automating a check is not the same as not testing it. These still need coverage, just from a person with a device, a checklist line, or a lower-level test.
One-off and launch-week checks
Confirming that a festive promo banner shows on the home screen for a five-day sale, or that saved addresses moved correctly into a new schema after the v5.0 update, are checks you need once or twice. A scripted test would be obsolete within a week. Put them on the release ticket, run them on two or three phones, and attach a screen recording.
Checks that need a human eye
Some questions have no reliable yes or no answer. Is the dark mode palette on the order tracking map readable in daylight? Does the confetti animation stutter on a mid-range Android phone with 4 GB of RAM? Does the Arabic layout read naturally when the screen flips to right to left? Does the "slide to pay" control feel right under a thumb? A screenshot test can say pixels changed, not whether the change is good, and a diff against a live map tile fails every run.
Screens still under active design
An onboarding carousel in redesign this sprint, or a home feed running two variants behind a remote config flag, will change several times before release, and each UI test against it gets rewritten each time. Cover the logic with unit or widget tests now, and add the UI check once the screen survives a release unchanged.
Rare paths that are hard to trigger on demand
A low storage warning during a photo upload, an OEM battery optimizer killing background sync on certain Xiaomi or Samsung builds, an incoming call interrupting a video KYC session, or airplane mode switched on halfway through a payment are all real failure modes. Automating them needs device-level hooks that differ per manufacturer and break on OS updates. A time-boxed exploratory session on a few physical phones each release finds more for less effort, as long as the tester logs device model, OS version, and steps.
Third-party screens you do not own
Bank 3-D Secure pages inside a WebView, the Google Pay or Apple Pay payment sheet, and social login consent screens can change without notice, and some actively resist automation. Mock the provider in CI, assert your app handles success, failure, and cancel, and do a manual pass against the real provider before release.
Mobile checks you should automate first
Some checks are so frequent, risky, and stable that leaving them manual is the expensive choice.
The smoke path on every build
The app launches, a test user signs in, the home screen loads real data, and sign out works. Run it on every merge, and keep it short enough to finish before the developer has switched tasks. YAML-driven flows are a quick way to get this running; our Maestro testing guide walks through a first flow.
Money and data paths
Add to cart, apply a coupon, pay with a sandbox card, and see an order number. Transfer funds and see the balance update. Edit a profile field, relaunch the app, and confirm the change stuck. These paths are high risk, run constantly, and change carefully, which makes them the best return on a scripted UI test.
Permission happy paths on settled flows
Granting camera access for a QR scanner, allowing location for a store finder, and accepting notifications after the Android 13 POST_NOTIFICATIONS prompt are worth automating once the surrounding screens have settled. A permission regression often kills a feature silently, and the grant path itself is short and repeatable. Tool support varies: the Appium documentation covers driver options such as auto-granting permissions on Android and auto-accepting alerts on iOS, while the Flutter docs note that the built-in integration_test package cannot interact with native permission dialogs and point to Patrol for that job. Automate the grant path first and leave "denied, then re-enabled in Settings" to exploration for now.
Bugs that already reached users
If a defect made it to the Play Store or the App Store, write the test that would have caught it, at the lowest layer that can catch it. A crash in a price formatter belongs in a unit test, not a UI flow that takes minutes to reach the price.
For help picking frameworks for these checks, the automated functional testing guide compares what to automate first with which tools fit each job. Once the smoke suite runs on every merge, running it in parallel across devices keeps wall-clock time down without cutting coverage.
Write assertions around outcomes, not copy
Even the right checks fail for the wrong reasons when assertions are fragile. The usual offender on mobile is exact on-screen text.
// Fragile: breaks on any copy edit, emoji change, or new locale
onView(withText("Your order is on its way! 🎉"))
.check(matches(isDisplayed()))
// Outcome-based: stable id, a format check, and a backend check
onView(withId(R.id.order_confirmation_root))
.check(matches(isDisplayed()))
onView(withId(R.id.order_number))
.check(matches(withText(matchesPattern("^ORD-\\d{8}$"))))
// testApi and lastOrderId() are your own test helpers
assertEquals("PLACED", testApi.getOrder(lastOrderId()).status)
The first version fails when marketing edits the message, the Hindi translation ships, or the emoji goes. The second asserts what the user actually got: a confirmation screen, a well-formed order number, and an order on the server.
A few habits keep assertions useful through redesigns:
- Assert the result, not the wording. Check the balance moved by the transferred amount, not the sentence announcing it.
- Target stable identifiers. Android resource IDs, iOS accessibility identifiers, and Flutter Keys or Semantics labels survive layout changes far better than XPath or visible text.
- One outcome per check. A test that asserts fifteen elements on a screen turns every small design change into a failure.
- Wait on conditions, not timers. A fixed sleep is too long or too short on real devices. Espresso's idling resources let the test wait until the app is idle, and most frameworks have an equivalent.
- Confirm the backend when the UI only mirrors it. A quick API read beats parsing a label.
The four-factor scorecard
To make "should we automate this?" a quick conversation, score each check 1 to 3 on four factors. Higher always means more suited to automation.
| Factor |
3 |
2 |
1 |
| Frequency |
Needed on every build or merge |
Needed once per release |
Needed once, or a few times a year |
| Risk |
Money, data loss, lockout, or compliance |
A core feature degrades |
Cosmetic or easy to work around |
| Stability |
Screen unchanged for two or more releases |
Small tweaks expected |
In redesign or behind an experiment |
| Judgment |
Pure yes or no outcome |
Some visual confirmation |
Needs taste: feel, motion, tone, legibility |
Then apply the rules in order:
- If Stability or Judgment scores 1, do not automate yet. Send it to an exploratory session if Risk is 2 or more, otherwise keep it as a manual checklist line.
- If Frequency plus Risk is 5 or more, automate it.
- Otherwise, explore it if Risk is 2 or more, and keep it as a checklist line if not.
Here is how a set of checks from a typical commerce app might score:
| Check |
Freq |
Risk |
Stability |
Judgment |
Verdict |
| Sign in with email and OTP on a fresh install |
3 |
3 |
3 |
3 |
Automate (smoke) |
| Pay with a sandbox card and see an order number |
3 |
3 |
3 |
3 |
Automate |
| Allow camera, then scan a QR code to pay a store |
2 |
3 |
3 |
3 |
Automate |
| Deny notifications, then re-enable them in Settings |
2 |
2 |
2 |
3 |
Explore |
| Photo upload when storage is nearly full |
1 |
2 |
2 |
2 |
Explore |
| Dark mode legibility on the order tracking map |
2 |
2 |
2 |
1 |
Explore |
| New onboarding carousel, redesigned this sprint |
2 |
2 |
1 |
2 |
Explore |
| Festive sale banner on home for five days |
1 |
1 |
2 |
3 |
Manual checklist |
Scores move. The carousel scores 1 on stability today and 3 after two unchanged releases, when it becomes an automation candidate, so re-score the Explore column each release. And "Explore" does not mean "ignore." It means a person with a device and a short charter, such as "break notification permission recovery on the newest Android and iOS releases," spends a fixed block of time and writes down what they find.
Putting the scorecard to work this sprint
Score the noisiest tests first. List the UI tests that failed most last month without a real bug behind them, and score each. Anything in Explore or Manual leaves the blocking suite, into a non-blocking lane or deleted with a ticket naming its replacement. Google's post warns quarantine can hide a real race condition, so give each quarantined test an owner and a review date.
Score new tests before anyone writes them. Add the four numbers to the test ticket template. A thirty-second grooming chat costs less than a test written, stabilized, and deleted two sprints later.
Push logic down a layer. If a check is really about a calculation, a validation rule, or a state transition, it belongs in a unit or widget test. The Flutter docs recommend many unit and widget tests plus enough integration tests for the important use cases, which holds on native apps too.
Where autonomous testing fits
The Explore column is where teams feel stretched, since sessions on real phones cost people time every release. QApilot, for example, crawls an app's screens to build a knowledge graph of its flows, which helps a team see which journeys exist before deciding which deserve a scripted test. Its MCP server lets a developer describe a check in plain English from Claude or Cursor and run it on an Android device or emulator while a screen is still changing, without committing a brittle test to the suite. Neither replaces the scorecard; they make unscripted checks cheaper to cover.
The goal was never to automate as much as possible. It is to give each mobile check the cheapest form of coverage that still catches the failures that matter, and to revisit that choice as screens settle.
