How do you price a bottle of milkshake? Everyone from Ayn Rand to Karl Marx will give you a slightly different answer, but the majority of solutions boil down to some combination of cost, competition, popularity, and good old trial and error. That basic formula hasn’t changed for centuries.

The modern rise of supersized, sell-everything stores has, however, fundamentally altered the method of calculation. With over 200 different varieties of canned tomatoes on Walmart shelves, the complexity of putting a sticker on any one of them has exploded. To take the guessing out of this process, big store chains are now turning to data. Data of the scale and breadth to match their football-pitch-sized supermarkets. Big data.

This past week I visited the offices of Dunnhumby, a company that started off as a charming husband-and-wife operation in 1989 and has since grown into one of the biggest customer analytics firms in the world. It specializes in collecting and making sense of the masses of information supermarkets collect on a daily basis — both about their own operations and their customers’ particular shopping habits.

"It’s all about ensuring logic at the shelf," says VP Sean Dangler as he walks me through a demo of the KSS Retail PriceStrat software. KSS Retail is a subsidiary of Dunnhumby, which is itself owned by Tesco, the UK’s dominant supermarket chain and the world’s second-largest retailer after Walmart. Tesco recently deployed PriceStrat across its UK stores, meaning that the system I was looking at was the very same that sets the price of a loaf of tiger bread.

Tesco-clubcard-560

16 million loyalty cards can tell you a lot about the way people think

Dangler takes me down the virtual yogurt aisle to demonstrate his concept of shelf logic. We see no fewer than 517 varieties to choose from. He needs to set a price for each one that coheres with every variant nearby and makes sense to the browsing shopper. It’s not enough to simply know that a 12-pack should cost less per item than a six-pack, nor that branded yogurts should command a price premium. This system is aware enough to tell its user by exactly how much each of those prices should vary. Dynamically and in real time. The key to all of this is your store loyalty card.

Dunnhumby got its big breakthrough with the introduction of the Tesco Clubcard as the UK’s first customer loyalty card in 1995. Today, it continues to manage all the incoming data from the roughly 16 million Clubcard members and it’s that additional insight that the PriceStrat software taps into. Knowing customers’ names, addresses, family situation, and shopping habits, it can detect such nuanced things as your yogurt flavor preferences and internal hierarchy of brands. It’s capable of identifying the precise price point at which you would switch between brands or how much incentive you’d need to buy the bigger pack. It’s this extra layer of personalized data, which is combined with two years’ worth of checkout receipts, that makes PriceStrat formidably knowledgeable. Even if you’ve never stepped foot in a Tesco store before, your shopping habits are now predictable to the big supermarket chain, which can synthesize a model of your preferences by referring to demographically similar shoppers.

Anonymous data is good, but it only provides an average. The customer-specific tracking that comes from the use of loyalty cards is exponentially more valuable in adding social and economic dimensions to what stores know about their customers. It also sets the stage for PriceStrat’s next big trick: forecasting sales as prices change.

Kss-retail
PriceStrat can simulate the impact of any pricing change to a 95-percent certainty.

Once Dangler has set up his basic pricing rules, he’s ready to start testing out potential discounts and special offers to try and improve sales. He goes for an aggressive price cut on the own-brand natural yogurt, cutting the profit margin to a few pennies, and the volume of predicted sales balloons as a result. It turns out that people are really price-sensitive when buying cold desserts. Alas, a large proportion of the gains is offset by a drop in branded sales, meaning the idea would probably result in worsening relationships with suppliers in exchange for a modest increase in profits. We keep searching for the optimal solution, with every small change having an immediate trickle-down effect on related products. It’s like a chaos theory testing suite, with each price being a flap of a butterfly’s wings. The only thing missing is a button to make the system automatically optimize everything, you still need humans to input scenarios.

It’s like a chaos theory testing suite, with each price being a flap of a butterfly’s wings

Along the way, I discover phenomena like asymmetric cross-price elasticity — an eight-pack’s price affects sales of four-packs more strongly than vice versa — and the fact that a "buy one, get one free" offer is more cost efficient than a straight 50 percent price cut (that’s because some people will still take just one). Melding such insights with seasonal trends and household-specific behavior tracking can lead to incredibly accurate predictions of the price a supermarket needs to set to get its next shopper to buy yogurt. Dangler tells me that the PriceStrat sales estimates are consistently within 5 percent of the actual numbers — no matter what weird and multilayered scenario you throw at the system.

I notice that almost none of this sophistication addresses the classic considerations of cost and competition, and Dangler tells me that’s quite intentional. Companies like his are trying to invert the traditional structure of looking at per-item profit margins and competitors’ prices first, choosing to take customer preferences as their starting point. His software includes price comparisons and direct costs, but only as pieces in the greater puzzle of organizing a vast array of goods and products in an economically optimal way.

The amount of information we produce, not just consume, was the subject of a recent blog post by new Microsoft CEO Satya Nadella, who believes we’re on the brink of an era of "ambient intelligence." With the use of digital devices becoming ubiquitous and generating an ever-expanding "data exhaust" from each individual, Nadella thinks that the future of computing lies in correctly interpreting and learning from that mass of information. IBM, which offers a similar 360-degree customer analytics platform to PriceStrat, reported in 2011 that 90 percent of the world’s data had been produced in the previous two years.

Though Nadella and guys like those I met at Dunnhumby are universally optimistic about this trend, it does create an obvious privacy concern. "What scares me," remarked Lord MacLaurin, Tesco’s boss at the time of the launch of the Clubcard, "is that you know more about my customers after three months than I know after 30 years."

"What scares me is that you know more about my customers after three months than I know after 30 years."

In an act that is simultaneously democratizing and disquieting, supermarkets are starting to place the customer at the center of their pricing strategies. They know more about who we are, what we like to buy, and how we tend to buy it than ever before, and they’re reaching out for even more. Tesco has successfully rolled out its Hudl Android tablet and a Clubcard TV online service as a pair of extra incentives to encourage its users to volunteer more of their information. In many ways, it’s just an effort to keep pace with data-driven online companies like Amazon and Google, whose profitability relies on knowing as much about their users as possible.

My experience with the PriceStrat software has been a study in better profitability through science: it’s a mathematical system that accounts for a wide spectrum of consequences, but takes no regard of ethical or socioeconomic considerations. Nonetheless, the actual functioning of the software, the vast databases it can harness, and the oftentimes counterintuitive insights it can produce are undeniably impressive. It might never be smart enough to determine the fair price of a milkshake, but it can certainly figure out the most profitable one.