Facial recognition systems have made huge technological leaps in recent years — offering definitive identification in seconds, from as far as 350 meters away — but there’s one scenario where they still fall short: the car. Modern systems still struggle to identify faces through glass, particularly the slanted windshield of an automobile. Current systems at land borders in Hong Kong perform facial recognition through rolled-down windows, simply to avoid the confounding effect of the glass.
But over the past few months, a new system has emerged to solve that problem, developed at Oak Ridge National Laboratory at the request of US Customs and Border Patrol. Still largely secret, the system draws on light-field sensor arrays to mitigate the glass barrier, allowing cameras to focus past the reflections on the windshield to the faces of the drivers and passengers. If effective, it could pave the way for far more aggressive deployment of facial recognition at automotive crossings.
The system arose out of an initiative called biometric exit, which mandates a face or fingerprint verification of every US visitor as they exit the country. At airports, that has meant a massive push for facial recognition cameras at departure gates and elsewhere — but land borders present unique challenges. Most land visitors arrive in cars, and windshields make it difficult to reliably capture facial images. According to a GAO report released in February, the available technologies “would require all passengers to stop and exit their vehicle to be photographed or scanned.” The result would create unworkable delays at some of America’s busiest crossing points, particularly at the Mexican border.
In June, Customs reached out to Oak Ridge National Laboratory for a solution, and recent statements indicate that they’ve found one. The lab still hasn’t publicly discussed its findings (a representative declined to comment for this piece), but at a conference in April, CBP executive director Colleen Manaher said the lab had made progress with a system called a plenoptic camera.
“The camera they have developed can go into a vehicle through tint and glare,” Manaher said, when asked about the system. “I’m looking at it through the naked eye and I can’t see in, but there on that screen are two people, the passenger and the driver, in facial recognition quality.” The camera is still in the prototype phase, but for Manaher and others at Customs, it presents a unique chance to break the technological stalemate.
The biggest problem for in-car facial recognition is the glass barrier, which adds reflections that confuse many algorithms, and also cuts down on the total amount of available light. The details of Oak Ridge’s plenoptic camera are still largely undisclosed, but the general technology could address both problems. Also called light-field cameras, plenoptic systems use an array of sensors to capture as much light information as possible, far more than would be necessary to construct a single static photograph. That means more light overall, but also enough information to perceive depth, a crucial element in distinguishing glass reflections from the features of the face behind it.
While the technology is relatively new to government agencies, it’s been available to consumers for years. The Lytro camera, first released in 2012, uses a similar system to let consumers tweak the focus and perspective of a photo in post-processing. Colvin Pitts, a senior architect at Lytro, says the camera’s depth-sensing capability could be particularly useful when cleaning up an image for facial recognition. “Because it lets you separate objects in space, you can imagine focusing one image on the tinted windshield and another image where you expect the driver to be,” Pitts says. “With some clever image processing, you could remove some portion of what you get from the windshield.”
It’s still unclear how reliable those techniques would be in the field, and Pitts cautioned that showing early success might be easier than developing a reliable system. “Solving some of the cases is a lot easier than solving all of the cases,” he told The Verge. That’s consistent with Manaher’s early assessment, which described the system as “very prototypey.”
As the technology develops, it could enable ambitious new facial recognition projects, particularly when combined with automatic license plate recognition or ALPR technology. Last Year, New York governor Andrew Cuomo announced a plan called New York Crossings that would install facial recognition cameras alongside ALPR systems at every bridge and tunnel leading into Manhattan, although experts say there are still significant logistical challenges to such a system. If implemented, New York Crossings would give officials identifying information on hundreds of thousands of individuals as they enter and exit New York City, the vast majority of which would not be suspected of any crime.
While it may be years before the systems are deployed, the issue is already raising privacy concerns. Clare Garvie, an associate at Georgetown Law’s Center on Privacy and Technology, says on-road systems like New York Crossings raise serious user consent issues. “It doesn’t matter how much notice is given the the public in advance,” Garvie says. “If these cameras are installed on gantries over main arteries, there is very little practical way to opt out of this system.”