First Click: The real world is fast becoming a digital colony

This thought occurred to me after playing Pokémon Go for the first time this week — a game which uses augmented reality to become present everywhere in your life. Your house, your office, the street you live on, the pubs and bars you meet your friends: they’re all part of the game now. And though you can choose not to participate, if something is popular enough, it will force its reality onto you. Just look at the case of Boon Sheridan, who lives in an old church, and when Pokémon Go was launched, found his house was now a gym.

The core technology that enables Pokémon Go (along with our increasingly powerful mobile networks and smartphones) is the mapping data from the game’s creator, former Google subsidiary Niantic. This data turns the real world into something that can be sorted, categorized, and managed by computers, and without it there would be no game. So what happens when other types of "real world" data also become digitized? We're about to find out in a big way, thanks to the advent of machine vision and object recognition.

Getting computers to understand out what’s happening in pictures and video has always been tricky, but in recent years, advances in deep learning have made it much, much easier. Now, Google can automatically sort your holiday photos based on whether you’re at the beach or the bar, and Facebook can describe every image on its social network for the benefit of the visually impaired. Photographs and videos are being turned into just another sort of data that's as easily quantifiable as text.

machine vision is going to turn photos and videos into just another sort of content

This is a seismic change, and its effects are going to be felt everywhere in the tech world and society. But one clear upshot is going to be the continuing colonization of the real, in line with what has happened with Pokémon Go.

Consider a recent patent application from Snapchat, which describes a technology to serve users’ ads (in the shape of sponsored filters) based on objects it recognizes in pictures. That means that if a user takes a snap of a particularly dreamy looking latte, Snapchat’s technology will recognize it as such and offer them a filter from Starbucks to place over the picture. In fact, the company’s patent application even includes specifications for an auction system, so that companies can bid on the advertising rights for certain objects. That means that if Dunkin’ out-bids Starbucks, well, you’ll get a different filter altogether when you photograph your coffee.

This is a relatively small example and one, I think, that sounds worse on the surface than it is in reality. (After all, Snapchat’s system only proposes to offer users image filters, not pop-ups.) However, it does illustrate how digitizing information about the real world — be that street names or what a cup of coffee looks like — leads to tech companies impressing their reality on our experience. In the case of both Pokémon Go and Snapchat, these realities are entirely optional (and essentially about advertising), but, just like digital maps, if enough people use them, they'll become the norm.