Smartphone benchmarks used to be a very big deal. They’re not quite as big anymore, especially since everyday users don’t really see incremental differences between chips these days, but that doesn’t mean no one cares at all anymore. MediaTek apparently cares quite a bit, apparently.
MediaTek benchmark scandal
So here’s what’s going on, and stop me if you’ve heard this story before.
Anandtech was using the Chinese version of the OPPO Reno3 Pro and the European Reno3, which use a MediaTek P95 CPU and MediaTek Dimensity 1000L CPU, respectively. The Dimensity 1000L is a newer and faster chip that should (and generally does) perform better, but PCMark scores say otherwise. Which is strange, to say the least.
When they pursued an anonymous version of the PCMark app to do another benchmark, the scores were much more realistic. By “cheating” in PCMark scores, this made the MediaTek P95 CPU score about 30% overall than it should have, which is a pretty significant bump.
Turns out, there’s a specific configuration file in the device’s firmware that whitelists certain benchmark applications to let the processor run faster and hotter to artificially bump up those scores. As we’ve seen before, this doesn’t help real-world performance, but it does make those numbers look better.
And, to be clear, thanks to Anandtech’s digging it doesn’t look like OPPO was doing this, especially since it doesn’t happen on all OPPO devices. Other smartphones using MediaTek chips have similar whitelist configuration files in their firmware, some of which date all the way back to 2016. This is apparently something that’s been happening for a while.
MediaTek responded and said they follow acceptable industry standards and simply optimize performance of their hardware to match whatever gets thrown at it, more or less. Pretty standard PR-friendly answer, but hey, what would you expect?
This isn’t the first time we’ve seen this kind of thing happening. Huawei has been caught cheating, and Samsung goes way back with some of their Exynos CPUs. However, those companies were all pretty much shamed into stopping the practice of trying to cheat benchmarks. MediaTek has managed to fly under the radar so far, and if this has been going on for four years it doesn’t seem like it’ll stop anytime soon.
Benchmarks are great, but know what you’re looking at when you do any research related to them.