Android phones on the board, and the bug that hid them

If you opened the benchmarks page any time in the last five weeks, the “By Android device” table had one row in it: Other Android. Hundreds of Android phones had run the benchmark by then. The table should have been showing Samsung, Xiaomi, Pixel and the rest. That was our bug, and it is fixed as of today.

What went wrong

A phone’s browser does not tell a website “Redmi Note 13 Pro”. It hands over a model code, and that code is all we have to go on. We match it against a list of patterns and keep only the family, such as Galaxy A-series or Xiaomi / Redmi, never the code itself.

The list was out of date in ways that mattered. Xiaomi’s codes start with the year and month the model was registered, then five to seven more characters. Our pattern expected five digits up front and a long tail. 2312DRAABG is a Redmi Note 13 Pro, and the letter in fifth place broke the match. 2201117TG, a Redmi Note 11, was too short. Between them, those two cases covered roughly half of current Xiaomi, Redmi and POCO phones. Honor codes and newer Huawei codes did not match anything. Neither did OnePlus phones from before 2021, iQOO, or Lenovo tablets. The Galaxy S26 and Pixel 10 had no family of their own. Every one of those ended up in Other Android.

The second problem was arithmetic. A family needs 30 runs before the table shows it, because a median from a dozen runs moves too much to rank anything. With a few hundred Android runs split across 26 families, almost nothing reached 30, even when the label was right.

What changed

The patterns now cover all of the phones above. Two pairs share a row because their codes look the same and can’t be told apart. Huawei and Honor use one format. So do OPPO and OnePlus, which have used the same CPH codes since 2021. Those rows say so in their names.

The table also rolls families up. A family with 30 runs still gets its own row. Below that, it joins its brand, so a dozen Galaxy S24s and twenty Galaxy A-series phones show up as Samsung rather than disappearing. Once the A-series reaches 30 it gets its own row again and the rest of the brand becomes Samsung (other). A brand that is still under 30 in total goes into Other Android. Every run is counted in exactly one row.

One thing we could not fix: runs already stored keep the label they were given. We never keep the model code, only the family, so there is nothing to relabel from. Other Android will shrink as new runs replace old ones. It won’t empty out overnight.

Where Android phones land

These are the numbers on the benchmarks page this morning, for every run of benchmark 2.5.0 so far.

The order matches what you’d guess, but the gaps are bigger than you might expect. A typical phone with a Snapdragon chip scores 1.6 times a typical phone with a Mali GPU. That’s partly a price split rather than a chip one. Most budget and mid-range Android phones run MediaTek chips with Mali graphics, and most flagships outside Google’s own run Snapdragon. A typical iPhone scores 1.7 times a typical Snapdragon phone. The Snapdragon spread is wide, though: the top quarter of those phones score above 2,901, comfortably ahead of the typical iPhone.

The Snapdragon median has also come down a long way. In the first two weeks of 2.5.0, with 39 Snapdragon phones counted, it was 2,460. Across 177 it is 1,357. We read that as the early runs leaning towards flagships. People with a new top-end phone are the first to want to see what it does. A median from 39 runs is a hint, and this is what a hint looks like when more data comes in.

Snapdragon and iPhone win different scenes

The total hides a split. The benchmark has six scenes, and phones don’t keep their order across them. In the same two-week window:

A typical Snapdragon phone does more than twice the iPhone’s work in the two GPU scenes. The iPhone takes the CPU physics scene by almost four to one, and the 2D canvas scene by two to one. In those two weeks, the totals for Snapdragon and iPhone came out within a few percent of each other, having agreed on almost nothing along the way. If a phone is mostly for games, the GPU rows say more about it than the total does.

These numbers come from 39 Snapdragon phones, 51 iPhones and 29 Mali phones. That’s enough to see the pattern, not enough to trust the third digit.

Android finishes clean less often

A run is complete when every scene found a load the phone could hold steadily. It is throttled when the phone measured fine but slowed down as it warmed up. Of Android runs in those first two weeks, 54% finished complete, 29% throttled and 17% could not measure at least one scene. iPhones finished complete 78% of the time.

Most of that is heat. A phone has no fan, and three minutes of full load is enough to make one back off. If you want a number that reflects your phone rather than how warm it got, run it plugged in, on a cool surface, with other apps closed. If the result comes back throttled, run it again after the phone has cooled down.

Getting your phone on the table

Run the benchmark with sharing switched on and your phone’s family joins the count. Phones with few runs show up under their brand first, then as their own family once there are 30. What we store about a run is listed on the privacy page: the score, the scene results, the screen’s numbers, and coarse families for the GPU, browser and, on Android, the phone. Nothing on the benchmarks page is finer-grained than that.