Huawei caught cheating benchmark test for P20

Photo by Vlad Savov / The Verge

Huawei has been caught optimizing some of its top smartphones to over-perform on benchmark tests. On Tuesday, AnandTech discovered that Huawei’s P20 had been programmed to maximize performance specifically when running 3DMark, a popular benchmarking app. Today, the company behind 3DMark followed up with a statement saying that it had confirmed the findings and would delist the P20, as well as three other Huawei phones with similar behavior, from its benchmark leaderboards.

The delisted phones include the P20, P20 Pro, Nova 3, and Honor Play. Huawei admitted to this behavior in a statement given to Android Authority, saying that its phones are designed to adjust their performance based on the app that’s running.

But the way that Huawei implemented that behavior isn’t allowed. While phones can adjust their performance as part of their typical behavior under high workloads, they can’t be hard coded to maximize their behavior just because a specific benchmark app is running. That’s what Huawei seems to have done, according to UL, which is behind the 3DMark software.

When UL ran an internal version of 3DMark, which Huawei’s phones couldn’t recognize the name of, the phones performed worse in the test. That indicated that the phones weren’t actually smart enough to identify high performance demands on their own, which meant the benchmark score wasn’t an accurate reflection of how the phone would handle a typical app without special attention from Huawei.

As punishment, 3DMark has removed these phones’ rankings from its leaderboard and adorned their listings on its website with a note that the phone’s “manufacturer has not complied with UL benchmark rules.” Many of their results have been removed as well.

Huawei is far from the first company to get caught toying with benchmark results. Samsung got busted for the same behavior on its flagship phones in 2013, and just last year, OnePlus was found to have done the same. This timing is particularly unfortunate for Huawei, since just weeks ago it got caught trying to pass off a DSLR photo as a photo from one of its phones.

What’s funny about all of this is that benchmarks don’t really matter that much. Tweaking a phone to optimize benchmark apps might produce some numbers that make a small subset of nerds drool, but those numbers don’t correlate to the actual experience of using the phone. They might speak to how well the phone performs under heavy stress, like while gaming, but a better test is to just play a game with it and find out what happens.

Huawei even admits this. In its statement, the company said it “always prioritizes the user experience rather than pursuing high benchmark scores — especially since there isn’t a direct connection between smartphone benchmarks and user experiences.” And yet, it still coded its phone to deliver higher performance when running a benchmark test.

But Huawei also claims that its phones’ include AI that’s smart enough to optimize performance based on whatever app is running, and clearly that’s not the case. If it were, then boosting performance for benchmarks would be fair game — but as it is, it’s not a true reflection of how the phone performs in demanding situations.

Comments

What year is it?

Year of the benchmarkgate. Last year it was diselgate.

Following Samsung’s footsteps. At least Samsung wasn’t a security threat.

No just safety threat with their phones exploding all around.

apple invented exploding phones

Grow up. Also, #whataboutism

Huawei isn’t a security threat. You need to stop.

I am not making this up. Pay attention to the news.

Sensationalised news with no proof whatsoever. Nice sources.

This is right up there with Iraq’s WMDs. Still looking for the proof!

No one can prove what a chip in your phone, or a few lines of code can do. On the other hand, it is proven that Huawei is partly controlled by PLA. They also provide AI technology and equipment that is used for racial profiling and incrimination over the Uighur minority in XinJiang. These are openly known truths. Not even Huawei denies either of them.

Did you read the reason why our top spy agencies suggested consumers don’t buy their products?

Spy agencies, notorious for their honesty and impartiality.

Don’t get me wrong, I’m not taking them at their word. But how do they benefit from telling consumers to stay away from these products?

Breitbart? Fox?
Honeslty….show me an article or publication from a well respected journalist, University or company that confirms that there are backdoors built into their chips.
If you can´t finally please shut up.

Fact – China is an authoritarian state which suppresses dissent with an unelected government.

Fact – This unelected authoritarian government routinely subjects companies operating within the jurisdiction under its control to extra-judicial demands.

Fact – China, as a dictatorship, has both state and private media under special control and direction in order to censor all news or ideas regarded as detrimental to state security. So any news about backdoors in any Chinese equipment, real or speculated, will be suppressed and never acknowledged.

Fact – China tolerates no VPN operation within its borders that it cannot decrypt. As Huawei is the World’s largest telecoms equipment maker, it goes without saying that the Chinese intelligence services MUST have some sort of backdoor into that equipment so as to be able to decrypt traffic on these networks.

We can all do the math here. I need no further proof than the above to know that the Kirin chips in Huawei phones may as well be called the ‘Spychip Series 1000’.

Nothing you posted is proof that Huawei spies on you. You’re just buying into the US agency propaganda.

Look, if all that bothers you that much, how about some other facts.

-Some of the critical ingredients inside the battery come from places where child are forced to work, blood is shed and people killed just to secure the cobalt extraction. I guess that won’t bother you.
-fact is also that most of the components are built in China, as long as that happens there is a very real probability of tampering with the hardware.
-all your points are facts – just like every US manufacturer has its stuff built within China by Chinese companies. I also assume that you have no problem with that, right?

The issue is that there is not a single proven incident where Huawei phones showed signs of having backdoors installed. Not one, but you Breitbart guys state and insist on every post that it is not only likely but a proven fact.

I know that proof means nothing to you, but for me it does. I do not own a Huawei phone but I still think that people should not accuse a company of something that is in no way proven.

That wasn’t the claim being made though.

Nothing in the news was proven at all. It’s just US propaganda at this point until proven otherwise.

"Chinese company installed secret backdoor on hundreds of thousands of phones"
(Huawei was one of the companies busted releasing phones with this preinstalled.)
https://arstechnica.com/information-technology/2016/11/chinese-company-installed-secret-backdoor-on-hundreds-of-thousands-of-phones/

China’s antiterrorism law requires backdoors to be installed on their phones.
https://thediplomat.com/2016/01/chinas-comprehensive-counter-terrorism-law/

I wonder why none of the Huawei apologists above haven’t responded to this yet.

I already covered this in another thread, stop with the nonsense.

Because it’s BS. The article he posted even has the quote:

Huawei takes our customers’ privacy and security very seriously, and we work diligently to safeguard that privacy and security. The company mentioned in this report is not on our list of approved suppliers, and we have never conducted any form of business with them.
(Huawei was one of the companies busted releasing phones with this preinstalled.)

This is a lie. They have had ZERO dealings with AdUps.

Also, BLU phones have no restrictions when they were the very phones that had this issue.

You are the very definition of fake news.

View All Comments
Back to top ↑