clock menu more-arrow no yes

Filed under:

Twitter's powerful new search engine includes every tweet ever sent

New, 8 comments

"HI TWITTERS . THANK YOU FOR A WARM WELCOME. FEELING REALLY 21st CENTURY ."

Twitter's always been about speed and immediacy. It's where news breaks, it's where topics trend, it's where your timeline flies by so fast you're bound to miss things. In an effort to make itself a more permanent home for all that information, Twitter has announced today that you can search through every tweet sent on the service since 2006.

Twitter has always stored tweets permanently (that's not just up to the Library of Congress), but used to purge tweets from the search index after only a few days or weeks. For a few years, though, it's been working to fix that. The new search engine means that you'll be able to search for that tweet from '08 where your friend DM-failed and never noticed, or that link someone tweeted at you two years ago. But it also means you'll be able to read back through tweets during the epic World Cup Final. You'll be able to find the guy who may have inadvertently live-tweeted the raid on Osama Bin Laden's hideout. You'll be able to follow conferences, events, and breaking news in full, even after the fact.

Search finds old things, but still favors the new

For now, it's a fairly simple engine: you search through words and hashtags to find what you're looking for. You can search a date range as far back as you'd like, or just type "Bieber" and watch a teenager take over the world one reply at a time. This has always been possible, sort of, through services like Topsy and of course Google, but Twitter's made it faster, easier, and far more reliable than ever.

Doing so, and balancing archival retrieval with the need to search and surface recent tweets, was a hard problem to solve. Yi Zhuang, a Search Infrastructure Engineer, wrote a lengthy blog post about how Twitter managed to build such a pervasive, ever-expanding search engine without going bankrupt and without losing focus on the here and now. (It's largely made possible by solid-state storage and the speeds with which it can read and write data.) It's an interesting read, but your time might be better spent looking up all the times people tweeted the word "fart" in July of 2007. It was a lot of times.