Skip to main content

Sony’s first AI image sensor will make cameras everywhere smarter

Sony’s first AI image sensor will make cameras everywhere smarter

/

On-device AI promises to be faster, cheaper, and more secure

Share this story

Sony’s new image sensor could one day appear in cameras like this one.
Sony’s new image sensor could one day appear in cameras like this one.
Photo by Amelia Holowaty Krales / The Verge

Sony has announced the world’s first image sensor with integrated AI smarts. The new IMX500 sensor incorporates both processing power and memory, allowing it to perform machine learning-powered computer vision tasks without extra hardware. The result, says Sony, will be faster, cheaper, and more secure AI cameras.

Over the past few years, devices ranging from smartphones to surveillance cameras have benefited from the integration of AI. Machine learning can be used to not only improve the quality of the pictures we take, but also understand video like a human would; identifying people and objects in frame. The applications of this technology are huge (and sometimes worrying), enabling everything from self-driving cars to automated surveillance.

But many applications rely on sending images and videos to the cloud to be analyzed. This can be a slow and insecure journey, exposing data to hackers. In other scenarios, manufacturers have to install specialized processing cores on devices to handle the extra computational demand, as with new high-end phones from Apple, Google, and Huawei.

From left to right: the IMX500 as a bare chip and IMX501 as a package product.
From left to right: the IMX500 as a bare chip and IMX501 as a package product.
Sony Electronics Inc.

But Sony says its new image sensor offers a more streamlined solution than either of these approaches.

“There are some other ways to implement these solutions,” Sony vice president of business and innovation Mark Hanson told The Verge, referencing edge computing, which use dedicated AI chips not attached directly to the image sensor. “But I do not believe they will be anywhere close to as cost effective as us shipping image sensors in the billions.”

The IMX500 is destined for commercial clients, not consumer hardware

Sony’s huge presence in the image processing market will certainly push this technology to clients at a huge scale. Hanson notes that the company has more than 60 percent market share, and shipped about 1.6 billion sensors last year. Among Sony’s customers is Apple, which uses the company’s sensors in its iPhone line.

This first-generation AI image sensor, though, is unlikely to end up in consumer devices like smartphones and tablets, at least to begin with. Instead, Sony will be targeting retailers and industrial clients, which are beginning to use computer vision technology more widely.

Hanson references Amazon’s cashierless Go stores as an example of this. In Amazon’s Go stores, the retailer uses scores of AI-enabled cameras to track shoppers and charge them for objects they grab from the shelves. “They put hundreds of cameras, and they’re running petabytes of data, on a daily basis through a small convenience score,” says Hanson. “But if we can miniaturize that capability and put it on the backside of a chip we can do all sorts of interesting things.”

Reports suggest that the resulting hardware costs have slowed the roll-out of Amazon’s stores, though Sony’s IMX500 sensors would not yet be capable of processing the large data-loads needed for this sort of functionality. Instead, they could perform simpler tasks, like mapping the flow of customers around any given store.

Amazon Opens First Cashierless Convenience Store In Seattle
Many applications of AI computer vision, like Amazon Go, require lots of expensive cameras.
Photo by Stephen Brashear/Getty Images

In addition to cost savings there are privacy benefits. If the AI chip is stuck directly onto the back of the image sensor then object detection can be done on-device. Instead of sending off data to be analyzed, either to the cloud or a nearby processor, the image sensor itself performs whatever AI analysis is necessary and simply produces the metadata instead.

Benefits include greater privacy and faster processing speeds

So, if you want to create a smart camera that detects whether or not someone is wearing a mask (a very real concern right now) then an IMX500 image sensor can be loaded with the relevant algorithm which allows the camera to send off quick “yes” or “no” pings.

“Now we’ve eliminated what would normally be a 60 frames per second, 4K video stream to just that one ‘hey, I recognize this object,’” says Hanson. “That can reduce data traffic [and] it also helps things like privacy.”

Another big application is industrial automation, where image sensors are needed to help so-called co-bots — robots designed to work in close proximity to humans — from bashing their flesh-and-blood colleagues. Here the main advantage of an integrated AI image sensor is speed. If a co-bot detects a human where they shouldn’t be and needs to come to a quick stop, then processing that information as quickly as possible is paramount.

Continental AG annual general meeting
AI cameras are also useful for keeping robots designed to work alongside humans safe.
Photo by Julian Stratenschulte/picture alliance via Getty Images

Sony says the IMX500 is much faster for these sorts of tasks than many other AI cameras, with the ability to apply a standard image recognition algorithm (MobileNet V1) to a single video frame in just 3.1 milliseconds. By comparison, says Hanson, competitors’ chips, such as those made by the Intel-owned Movidius (which are used in Google’s Clips camera and DJI’s Phantom 4 drone) can take hundreds of milliseconds — even seconds — to process.

The big bottleneck, though, is the ability of the IMX500 to handle more complex analytical tasks. Right now, says Hanson, the image sensor can only work with pretty “basic” algorithms. That means that more sophisticated and varied tasks, like driving an autonomous car, will certainly require dedicated AI hardware for the foreseeable future. Instead, think of the IMX500 as a simple, single-application device.

But this is only the first generation, and the technology will undoubtedly improve in future. Right now, cameras are smarter because they send their data to computers. In the future, the camera itself will be the computer, and all the smarter for it.

Test samples of the IMX500 have already started shipping to early customers with prices starting at ¥10,000 ($93). Sony expects the first products using the image sensor to arrive in the first quarter of 2021.

Update, Saturday May 16th, 6:15AM ET: The wording of this article has been updated to clarify that Sony’s Mark Hanson suggested Amazon Go as a potential application for computer vision more generally, rather than a specific target customer for Sony.

Today’s Storystream

Feed refreshed Sep 24 Striking out

E
External Link
Emma RothSep 24
California Governor Gavin Newsom vetoes the state’s “BitLicense” law.

The bill, called the Digital Financial Assets Law, would establish a regulatory framework for companies that transact with cryptocurrency in the state, similar to New York’s BitLicense system. In a statement, Newsom says it’s “premature to lock a licensing structure” and that implementing such a program is a “costly undertaking:”

A more flexible approach is needed to ensure regulatory oversight can keep up with rapidly evolving technology and use cases, and is tailored with the proper tools to address trends and mitigate consumer harm.


A
Youtube
Andrew WebsterSep 24
Look at this Thing.

At its Tudum event today, Netflix showed off a new clip from the Tim Burton series Wednesday, which focused on a very important character: the sentient hand known as Thing. The full series starts streaming on November 23rd.


A
The Verge
Andrew WebsterSep 24
Get ready for some Netflix news.

At 1PM ET today Netflix is streaming its second annual Tudum event, where you can expect to hear news about and see trailers from its biggest franchises, including The Witcher and Bridgerton. I’ll be covering the event live alongside my colleague Charles Pulliam-Moore, and you can also watch along at the link below. There will be lots of expected names during the stream, but I have my fingers crossed for a new season of Hemlock Grove.


A
Andrew WebsterSep 24
Looking for something to do this weekend?

Why not hang out on the couch playing video games and watching TV. It’s a good time for it, with intriguing recent releases like Return to Monkey Island, Session: Skate Sim, and the Star Wars spinoff Andor. Or you could check out some of the new anime on Netflix, including Thermae Romae Novae (pictured below), which is my personal favorite time-traveling story about bathing.


A screenshot from the Netflix anime Thermae Romae Novae.
Thermae Romae Novae.
Image: Netflix
J
Twitter
Jay PetersSep 23
Twitch’s creators SVP is leaving the company.

Constance Knight, Twitch’s senior vice president of global creators, is leaving for a new opportunity, according to Bloomberg’s Cecilia D’Anastasio. Knight shared her departure with staff on the same day Twitch announced impending cuts to how much its biggest streamers will earn from subscriptions.


T
Twitter
Tom WarrenSep 23
Has the Windows 11 2022 Update made your gaming PC stutter?

Nvidia GPU owners have been complaining of stuttering and poor frame rates with the latest Windows 11 update, but thankfully there’s a fix. Nvidia has identified an issue with its GeForce Experience overlay and the Windows 11 2022 Update (22H2). A fix is available in beta from Nvidia’s website.


A
External Link
If you’re using crash detection on the iPhone 14, invest in a really good phone mount.

Motorcycle owner Douglas Sonders has a cautionary tale in Jalopnik today about the iPhone 14’s new crash detection feature. He was riding his LiveWire One motorcycle down the West Side Highway at about 60 mph when he hit a bump, causing his iPhone 14 Pro Max to fly off its handlebar mount. Soon after, his girlfriend and parents received text messages that he had been in a horrible accident, causing several hours of panic. The phone even called the police, all because it fell off the handlebars. All thanks to crash detection.

Riding a motorcycle is very dangerous, and the last thing anyone needs is to think their loved one was in a horrible crash when they weren’t. This is obviously an edge case, but it makes me wonder what other sort of false positives we see as more phones adopt this technology.


A
External Link
Ford is running out of its own Blue Oval badges.

Running out of semiconductors is one thing, but running out of your own iconic nameplates is just downright brutal. The Wall Street Journal reports badge and nameplate shortages are impacting the automaker's popular F-series pickup lineup, delaying deliveries and causing general chaos.

Some executives are even proposing a 3D printing workaround, but they didn’t feel like the substitutes would clear the bar. All in all, it's been a dreadful summer of supply chain setbacks for Ford, leading the company to reorganize its org chart to bring some sort of relief.