Skip to main content

An AI-powered bionic hand will know what it's grabbing

An AI-powered bionic hand will know what it's grabbing


The hand uses computer vision to pick the right grip for the object in front of it

Share this story

If you buy something from a Verge link, Vox Media may earn a commission. See our ethics statement.

Image: Newcastle University

When the wearer of a prosthetic arm wants to grab something, there are a number of ways they can communicate this signal. With a basic prosthetic, the grip mechanism might be controlled mechanically, with a cable attached to the opposite shoulder, for example. With most contemporary limbs, the signal to grip is detected using myoelectric sensors — which read muscle activity from the skin. And with the most advanced (and most expensive) prosthetics, sensors that can measure nerve signals are actually implanted inside the muscle. But what if the arm itself could see?

That’s the idea being developed by biomedical researchers from Newcastle University, who have developed a prototype prosthetic limb with a AI-powered camera mounted on top. The camera uses the sort of computer vision technology that big tech companies have developed, with researchers using deep learning to teach it to recognize some 500 objects. When the wearer of the limb moves to grab, say, a mug, the camera takes a picture of the object, and moves the hand into a suitable “grasp type.” (For example, a pinching motion for picking up a pen; or a vertical grip for grabbing a soda.) The user then confirms the grip action with a myoelectric signal.

“Using computer vision, we have developed a bionic hand which can respond automatically — in fact, just like a real hand, the user can reach out and pick up a cup or a biscuit with nothing more than a quick glance in the right direction,” said Dr. Kianoush Nazarpour, a biomedical lecturer at Newcastle University, in a press statement. “The beauty of this system is that it’s much more flexible and the hand is able to pick up novel objects — which is crucial since in everyday life people effortlessly pick up a variety of objects that they have never seen before.”

The end result is a hand that is much quicker to use than contemporary prosthetics — up to 10 times faster than others on the market, say the researchers. The system is also cheap. The camera that allows the hand to “see” is just an ordinary Logitech webcam, and the AI software that recognizes objects can be trained cheaply. The whole training process is described in a study published in the Journal of Neural Engineering.

“It’s a stepping stone towards our ultimate goal.”

There are difficulties to overcome, of course. The neural network used to recognize objects wasn’t always correct (it had about an 80 to 90 percent success rate) and amputees who tested the prosthetic had to be able to override its actions when necessary. If the system was implemented in the real world, there would also have to be a mechanism for adding new objects to the AI’s memory, too. The next step, say the researchers, is to add more sensors to prosthetics, so they can detect things like temperature and pressure. With this information they’ll become more and more like organic limbs.

“It’s a stepping stone towards our ultimate goal,” said Dr. Nazarpour of the prototype arm. “But importantly, it’s cheap and it can be implemented soon because it doesn’t require new prosthetics — we can just adapt the ones we have.”

Today’s Storystream

Feed refreshed Sep 24 Striking out

External Link
Emma RothSep 24
California Governor Gavin Newsom vetoes the state’s “BitLicense” law.

The bill, called the Digital Financial Assets Law, would establish a regulatory framework for companies that transact with cryptocurrency in the state, similar to New York’s BitLicense system. In a statement, Newsom says it’s “premature to lock a licensing structure” and that implementing such a program is a “costly undertaking:”

A more flexible approach is needed to ensure regulatory oversight can keep up with rapidly evolving technology and use cases, and is tailored with the proper tools to address trends and mitigate consumer harm.

Andrew WebsterSep 24
Look at this Thing.

At its Tudum event today, Netflix showed off a new clip from the Tim Burton series Wednesday, which focused on a very important character: the sentient hand known as Thing. The full series starts streaming on November 23rd.

The Verge
Andrew WebsterSep 24
Get ready for some Netflix news.

At 1PM ET today Netflix is streaming its second annual Tudum event, where you can expect to hear news about and see trailers from its biggest franchises, including The Witcher and Bridgerton. I’ll be covering the event live alongside my colleague Charles Pulliam-Moore, and you can also watch along at the link below. There will be lots of expected names during the stream, but I have my fingers crossed for a new season of Hemlock Grove.

Andrew WebsterSep 24
Looking for something to do this weekend?

Why not hang out on the couch playing video games and watching TV. It’s a good time for it, with intriguing recent releases like Return to Monkey Island, Session: Skate Sim, and the Star Wars spinoff Andor. Or you could check out some of the new anime on Netflix, including Thermae Romae Novae (pictured below), which is my personal favorite time-traveling story about bathing.

A screenshot from the Netflix anime Thermae Romae Novae.
Thermae Romae Novae.
Image: Netflix
Jay PetersSep 23
Twitch’s creators SVP is leaving the company.

Constance Knight, Twitch’s senior vice president of global creators, is leaving for a new opportunity, according to Bloomberg’s Cecilia D’Anastasio. Knight shared her departure with staff on the same day Twitch announced impending cuts to how much its biggest streamers will earn from subscriptions.

Tom WarrenSep 23
Has the Windows 11 2022 Update made your gaming PC stutter?

Nvidia GPU owners have been complaining of stuttering and poor frame rates with the latest Windows 11 update, but thankfully there’s a fix. Nvidia has identified an issue with its GeForce Experience overlay and the Windows 11 2022 Update (22H2). A fix is available in beta from Nvidia’s website.

External Link
If you’re using crash detection on the iPhone 14, invest in a really good phone mount.

Motorcycle owner Douglas Sonders has a cautionary tale in Jalopnik today about the iPhone 14’s new crash detection feature. He was riding his LiveWire One motorcycle down the West Side Highway at about 60 mph when he hit a bump, causing his iPhone 14 Pro Max to fly off its handlebar mount. Soon after, his girlfriend and parents received text messages that he had been in a horrible accident, causing several hours of panic. The phone even called the police, all because it fell off the handlebars. All thanks to crash detection.

Riding a motorcycle is very dangerous, and the last thing anyone needs is to think their loved one was in a horrible crash when they weren’t. This is obviously an edge case, but it makes me wonder what other sort of false positives we see as more phones adopt this technology.

External Link
Ford is running out of its own Blue Oval badges.

Running out of semiconductors is one thing, but running out of your own iconic nameplates is just downright brutal. The Wall Street Journal reports badge and nameplate shortages are impacting the automaker's popular F-series pickup lineup, delaying deliveries and causing general chaos.

Some executives are even proposing a 3D printing workaround, but they didn’t feel like the substitutes would clear the bar. All in all, it's been a dreadful summer of supply chain setbacks for Ford, leading the company to reorganize its org chart to bring some sort of relief.