If you were compiling a list of the world’s 100 oddest objects — just the weirdest stuff that human civilization has excreted over the millennia — then you’d have to leave room somewhere for the Voynich manuscript. It’s 600 years old, written in a language no one can read, and full of diagrams no one understands. It is a genuine, bonafide, world-class mystery. This is presumably why when newsrooms around the world had a chance this week to publish stories claiming it’d been “decoded by artificial intelligence,” they leapt at the opportunity.

Except, of course, it hasn’t. Not at all. According to experts, the Voynich manuscript remains as inscrutable as ever. But understanding why this new research fails to “decode” the text, and what exactly it does add to the annals of Voynichology, has its own value. It also emphasizes (if further emphasis were needed) that this manuscript is one extremely odd cookie.

The research that sparked the coverage is a paper named “Decoding Anagrammed Texts Written in an Unknown Language and Script.” It was published in 2016, but it was presented at a conference last year and picked up by journalists earlier this month. In it, computer science professor Greg Kondrak and graduate student Bradley Hauer describe a method for finding the source language of ciphered texts, before turning that method on the manuscript itself, and deciding that it was originally written in Hebrew, before being encoded in its current form.

It’s a claim that, if true, would be a glacier-sized break in an ice-cold case. The 240-page Voynich manuscript is written in an unknown alphabet that’s never been seen before or since. The script is comprised of roughly 25 to 30 individual characters (interpretations vary) written from left to right in a single, elegant hand. Scattered throughout are illustrations of unidentifiable plants, astrological diagrams, doodles of castles and dragons, and a particularly odd section that shows naked women bathing in pools connected by flowing tubes. It looks like the map of an ancient water park, but scholars suggest it might be medical or alchemical in intent.

Most assume that the manuscript is written in what’s called a substitution cipher. This is one of the simplest and most ancient types of codes, in which letters of an established alphabet are swapped for invented ones. The problem is that hundreds of years of study have been unable to work out which language the Voynich manuscript was originally written in.

“No one has ever made a convincing case for any particular language,” Lisa Fagin Davis, the executive director of the Medieval Academy of America and a keen Voynich scholar, tells The Verge. “I’ve seen suggestions that it’s encoding Arabic, Aztec, Roma, Latin, Italian.” Davis says people tend to study the “paleographic, forensic, and artistic evidence” to find a country of origin, and with that, a source language, but she adds that computational analysis is also used.

It’s this tool that Kondrak and Hauer picked up in their attempt to deconstruct the manuscript. They figured, like many cryptologists before them, that by computing certain qualities of the text — like, for example, how often each letter and each combination of letters appear — they could create a statistical fingerprint that could be compared to other languages.

So, they trained a number of algorithms to pick out these metrics, using the Universal Declaration of Human Rights as their sample text in a whopping 380 languages. (Despite what some coverage suggested, this process did not involve neural networks or deep learning — just good old-fashioned statistical analysis, aka lots of counting and percentages.) And it worked! Not too badly anyway. According to Professor Shlomo Argamon, a computational linguist at Illinois Institute of Technology, the preliminary test results are “perhaps slightly questionable, but not more so than many other results often published in the scientific literature.” And so, with their algorithmic pattern-matcher trained and tested, Kondrak and Hauer turned to the Voynich manuscript. Here, say experts, is where things really started going downhill.

The problem is not any single mistake, but a series of assumptions and omissions that give Kondrak and Hauer more leeway in interpreting their results than is scientifically rigorous.

The first is quite straightforward: their algorithm was trained on modern-day languages, but the manuscript is carbon-dated to the 15th century. So, if it was originally written in Hebrew, it would have been written in 15th-century Hebrew. “The grammar, spelling, and vocabulary would have been quite different, especially for a manuscript like the Voynich that is scientific (as opposed to Biblical or liturgical) in nature,” says Davis.

The second is that although Kondrak and Hauer’s algorithm can produce suggestions for source languages of ciphered texts, it doesn’t evaluate the likelihood of these matches. So when the pair say that Hebrew was the highest scoring match for the manuscript without rating the likelihood, this is a bit of a meaningless boast. “Someone has to have the highest score,” says Argamon. “They mention some of the other top matches. As I recall, one was Malay, which is a language very, very different from Hebrew.”

The third assumption is perhaps the biggest: Kondrak and Hauer claim that as well as being a substitution cipher, the Voynich manuscript is also written in anagrams, so the letters in each individual word are scrambled. This is not a new suggestion in the world of Voynichology, but it’s far from an established truth. It also perfectly sets up the final flourish of Kondrak and Hauer’s research: translating the opening sentence of the Voynich manuscript into English.

The sentence in question is this: “She made recommendations to the priest, man of the house and me and people.” Kondrak says, “It’s a kind of strange sentence to start a manuscript but it definitely makes sense.” But even within the paper, he and Hauer describe how they had to fudge the translation to produce this result. Their first attempt was “not quite coherent,” said a speaker of modern Hebrew, and they had to make “a couple of spelling corrections” before feeding the characters into Google Translate to produce the result above. (“Any time you have to resort to Google Translate over someone who has actually studied the language, you’re going to lose some credibility,” notes Fagin.)

But this is where the assumption that the manuscript was written in anagrams becomes even more crucial. Argamon notes that written Hebrew is what’s known as an “abjad,” meaning a script with no vowels. If you assume that the manuscript was written in Hebrew and that it’s written in anagrams, then it becomes much, much easier to “translate.” Then, not only can you rearrange all the characters in a word to find something that makes sense, but you can add in your own vowels. This means “lots and lots of just random sequences of letters form coherent words,” says Argamon. Add this with the fact that Kondrak and Hauer made spelling corrections and relied on Google Translate (a piece of software that looks so hard for meaningful content, it often turns gibberish into coherent sentences), and you can see why the experts are skeptical.

“The point here is that their method [...] gives them huge latitude in doing this sort of impressionistic interpretation,” says Argamon. “They take this decoded sentence, squint at it through thick eyeglasses, and say that’s good enough for us.” Nick Pelling, a Voynich expert who’s written extensively on the subject, is more blunt. When asked by The Verge what chance he thinks the paper’s conclusions are correct, he says: “So close to 0% as makes no practical difference.”

So, “AI decodes mysterious 600-year-old manuscript”? Not so much.

In fairness to Kondrak and Hauer (and as is often the case with these stories), the media certainly deserves a significant amount of blame for the exaggeration. The pair admits that their research is only a “starting point,” and experts we spoke to recognized the utility of their underlying algorithms. The experts just say too many steps were missed to start making any claims about the manuscript itself.

And in many ways, it makes sense that attempts to crack the Voynich manuscript using “artificial intelligence” would be covered so breathlessly. A New Yorker article on the history of the manuscript describes it as “the perfect canvas on which to project our worries about the difficult and the frightening and the arcane,” and the same could be said about AI. In the contemporary media landscape, this diverse and complex group of technologies is often used as a stand-in for fears about automation and unknowable (and uncontrollable) machine intelligence. Pitting AI against the Voynich manuscript is like watching Godzilla fight Mothra: the spectacle is so fun that we don’t care or think too hard about the details.

Still, for experts, the fact that the manuscript remains impenetrable might be a relief. After all, if you’ve spent years and years of your life trying to decode a mysterious document, it would probably be a bit of a blow if some bloodless machine cracks it overnight.

As Pelling said in a final email: “Through my book [...] and my blog, I’ve probably written more actual historical research about the Voynich than anyone else alive: I’ve given talks on it, and made a TV documentary on it, and have been interviewed about it on radio and TV numerous times… And I still can’t read it. :-)” The mystery lives on.