DxO’s mobile photography test has become the go-to benchmark for comparing smartphone cameras. But the test, created in 2012, wasn’t designed to measure a lot of features that are now common in phones, so DxO is updating it to start considering a bunch of new factors, like second lenses and depth effects.
The updated DxOMark Mobile test has three big new focuses: zoom, bokeh, and low-light performance. The first two are meant to take into account additional lenses and focus effects, like on the iPhone 7 Plus. A phone with an additional telephoto camera will necessarily perform better on a zoom test than a phone that’s relying on digital zoom. And while the bokeh test will look for how well the camera is able to pleasantly blur the background of a photo in a traditional sense, artificial depth effects will be considered for this category, too, which again gives dual-camera systems an advantage.
The iPhone 7 Plus is now ranked higher than the iPhone 7
DxO already measured low-light performance, but its tests now measure scenes lit no brighter than candlelight for both photos and videos. “Consumers are also now expecting their phones to capture images in very-low-light conditions, so we are testing smartphone cameras in lighting down to 1 Lux (essentially candlelight),” writes DxO’s David Cardinal. “This exposes differences in performance between phones that otherwise might rank very similarly.”
In addition to those three factors, the test now takes into consideration motion blur, image capture speed, better testing of HDR and dynamic range, and deeper video tests, including focuses on stabilization and adapting to new lighting conditions.
Beyond just doing a better job of testing phones, DxO says its updated test does a better job of distinguishing between top performers. So while Apple’s iPhone 7 Plus, Google’s Pixel, and HTC’s U11 are still leading the pack, the updated test shows that Apple does better with zoom and bokeh, the Pixel does better with autofocus and exposure, and the U11 does better with color and texture.
To give people a better idea of how the new test will work, DxO went back and retested several top phones, including the iPhone 7 and 7 Plus. Originally, DxO only tested the iPhone 7 and gave that phone a rating of 86 out of 100. But under the new test, the 7 gets an 85 and the 7 Plus gets an 88. The difference comes down entirely to the new categories of bokeh and zoom: since the 7 Plus has that extra camera, it stands out in both areas.
While the new version of DxOMark Mobile should do a better job assessing cameras, all of this still needs to come with some caveats. Benchmark tests try to simulate and test real-world conditions inside a lab, but ultimately they aren’t testing what it’s like to shoot with a phone in the real world. DxO’s tests are a handy guideline to check for how different cameras perform, but they aren’t a replacement for actual experience. The test’s new factors also place weight on specific features, like telephoto lenses, that may not matter to everyone.
That all said, DxO has done a very good job lately with its camera rankings. Its original test ranked the U11 a point above the Pixel, which lined up with my colleague Vlad Savov’s experience. Back in July, he wrote, “Even though I always say benchmarks don’t tell the whole story, I have to agree with them on this count. The U11 is the first phone I’ve yet used that doesn’t make me miss the Pixel’s camera.”
Under the new test, the Pixel and U11 are now dead even, with both holding a 90 point score.