If you want to show the common link between iPhones, iPads, Samsung Galaxy devices, and Google Pixels, you’ll find it in the underlying CPU technology that they’re all based on. ARM is the company, recently acquired by SoftBank for an immense $31 billion, quietly residing at the heart of the mobile revolution, designing the processors and graphics systems that go into our most-desired devices. This week ARM has expanded its portfolio to also include image signal processors (ISPs), which is the first major product from its year-old Imaging and Vision Group and a major foray into pushing mobile camera technology forward.
I met with ARM this week and was given an introduction to the new ISP product, which makes its debut in the Mali-C71 ("C" for "camera") intended for automotive applications. Steve Steele, one of the leading figures of ARM’s IVG, wasn’t shy about setting high expectations: "We do feel, genuinely, that our ISP is the best that’s available." Having experienced the recent marvels produced by Google’s image processing and knowing Apple’s traditional strength in this field, I reminded him of both and he remained resolute in his assertion. The best image processing. From the company that sets the agenda for all processing in mobile devices. I couldn’t resist being thrilled by that prospect.
Every mobile device with a camera today has an ISP — it’s the visual equivalent of analog-to-digital converters in audio, taking the raw photonic data gathered up by an image sensor and turning it into something that can be usefully displayed on the device, or encoded into a familiar format for sharing, or even sent to a computer vision subsystem that can recognize objects in the frame. Each of those tasks has unique requirements, but it’s adequate to just understand the basic components here: a digital camera uses a lens to focus light onto a sensor, and then an ISP to turn whatever the sensor spits out into a (hopefully accurate) representation of the scene in a digital format. The problem with all of that? Even with a great amount of competition out there, most ISPs are actually quite terrible.
Think of all the Chinese smartphone makers that proudly proclaim they have Sony’s latest and greatest imaging sensor. They tell you the exact model name of the sensor, and they usually accompany it with an exploded view of their multi-lens array. It’s impressive engineering, to be sure, but then you get the phone in your hands and you find that, strangely, the pictures aren’t quite as awesome as the undoubtedly excellent Sony hardware would lead you to expect. That’s where the ISP disadvantage rears its head. Meizu, Huawei, Xiaomi, and their ilk don’t have the years of research and development that industry leaders like Apple and Samsung have poured into their imaging pipelines. They can get great optics and sensors, but not a great ISP.
ARM’s ISP exists specifically to address that weakness of the mobile market. Its Mali-C71 iteration targets autonomous cars and all the situational awareness they need to achieve through imaging, so it’s a little overpowered for smartphones, but the company tells me that its mobile solution is already released to silicon partners and we can expect it to become official some time over the next few months.
In terms of integration into a mobile chip, the ISP is little different to ARM’s GPU designs. A silicon vendor licenses the intellectual property from ARM and then works to integrate the ISP into its system-on-a-chip design, which would also include the various CPU and GPU cores as well as power management and other integrated parts. Then a smartphone maker can just order an all-in-one solution to build a new smartphone or tablet device around: CPU, GPU, and ISP all nicely tucked into the same chip.
Qualcomm is one of ARM’s silicon partners that already offers such a solution, but I haven’t been hugely impressed by the implementations of it that I’ve seen out on the market. ARM’s lofty promise to be the absolute best, allied to its history of sober business decisions and avoidance of overstatement, makes me hopeful for what this new development might deliver.
Just look at the multiplicity of steps involved in capturing a single image. ARM’s ISP puts the image sensor’s data through 15 stages of refinement and correction: de-noising, dead pixel correction, de-mosaicing, tone mapping, white balance, color space conversion, gamma correction, sharpening, and then final adjustments to account for whether you want to show the image on the device’s own display or export it elsewhere. A smartphone vendor that buys a chip with this ISP built in is essentially outsourcing all of that complex work and making the task of engineering a new device much simpler. They’ll still be able to tune the ISP to their liking, so there’s no danger of every camera churning out the same imagery anytime soon, and ARM recommends that each company spends at least two months doing exactly that.
For a fuller breakdown of the geeky details of the Mali-C71, I recommend reading ARM’s blog post on the subject as well as AnandTech’s comprehensive coverage. The two provide a solid grounding in the technical aspects of this, but I keep returning to the simple promise that’s embedded in this new technology: better image processing from chips that will be available to all, not just an exclusive few. Phone cameras have consistently lagged behind display technology and industrial design, both of which have grown to be amazing even on mid-range devices, but this new ISP tech promises to bring them up to parity in a hurry.