When Microsoft added a chatbot to his Bing search engine this month, people noticed it was offering all sorts of false information about the Gap, Mexican nightlife, and singer Billie Eilish.
When journalists and other early testers had long conversations with Microsoft’s AI bot, things turned crass and terrifying creepy behavior.
In the days since the Bing bot’s behavior became a global sensation, people have struggled to understand the peculiarity of this new creation. More often than not, scientists have said that humans deserve a lot of the blame.
But there’s still a bit of mystery about what the new chatbot can do – and why it would. Its complexity makes it difficult to parse and even more difficult to predict, and researchers view it through a philosophical lens as well as through the hard code of computer science.
Like any other student, an AI system can learn bad information from bad sources. And that strange behavior? It could be a chatbot’s distorted reflection of the words and intentions of the people using it, said Terry Sejnowski, a neuroscientist, psychologist and computer scientist who helped lay the intellectual and technical foundations for modern artificial intelligence.
“This happens when you go deeper and deeper into these systems,” says Dr. Sejnowski, a professor at the Salk Institute for Biological Studies and the University of California, San Diego, who has a research paper on this phenomenon this month in the scientific journal Neural Computation. “Whatever you’re looking for – whatever you want – they’ll provide it.”
Google too boasted a new chatbot, Bard, this month, but scientists and journalists soon realized it was writing nonsense about the James Webb Space Telescope. OpenAI, a San Francisco start-up, kickstarted the rise of chatbots in November with the introduction of ChatGPT doesn’t always tell the truth.
The new chatbots are powered by a technology scientists call a large language model, or LLM. These systems learn by analyzing vast amounts of digital text from the Internet, including amounts of false, biased, and otherwise toxic material. The text that chatbots learn from is also a bit outdated, as it takes months for the public to use it.
As it analyzes that sea of good and bad information from the internet, an LLM learns one thing: guess the next word in a series of words.
It works like a giant version of the autocomplete technology that suggests the next word as you type an email or an instant message on your smartphone. Given the “Tom Cruise is a ____” sequence, it might be guessing “actor.”
When you chat with a chatbot, the bot doesn’t just draw on everything it’s learned from the internet. It draws from everything you’ve said to him and everything he’s said back. It’s not just guessing the next word in the sentence. It guesses the next word in the long block of text that contains both your words and their corresponding words.
The longer the conversation lasts, the more influence a user unconsciously has on what the chatbot says. If you want it to get angry, it gets angry, said Dr. Sejnowski. If you persuade it to get creepy, it will get creepy.
The alarmed reactions to the strange behavior of Microsoft’s chatbot overshadowed an important point: the chatbot has no personality. It provides instant results spat out by an incredibly complex computer algorithm.
Microsoft seemed to curb the strangest behavior when it put a limit on the length of conversations with the Bing chatbot. That was like learning from the test driver of a car that if you go too fast for too long, the engine burns out. Microsoft’s partner, OpenAI, and Google are also exploring ways to control the behavior of their bots.
But there’s a caveat to this reassurance: Because chatbots learn from so much material and put it together in such a complex way, it’s not entirely clear to researchers how chatbots produce their final results. Researchers watch to see what the bots do and learn to set limits on that behavior – often after it has happened.
Microsoft and OpenAI have decided that the only way they can find out what the chatbots are going to do in the real world is to let them loose – and bring them in if they stray. They believe their large, public experiment is worth the risk.
Dr. Sejnowski compared the behavior of Microsoft’s chatbot to the Mirror of Erised, a mystical artifact in JK Rowling’s Harry Potter novels and the many films based on her inventive world of young wizards.
“Erised” is “desire” spelled backwards. When people discover the mirror, it seems to offer truth and understanding. But it doesn’t. It shows the deep-seated desires of everyone who stares into it. And some people go crazy if they stare too long.
“Because humans and the LLMs both mirror each other, over time they will tend to a common conceptual state,” said Dr. Sejnowski.
Unsurprisingly, he said, journalists began seeing creepy behavior in the Bing chatbot. Consciously or unconsciously, they pushed the system in an uncomfortable direction. As the chatbots take in our words and bounce back to us, they can reinforce and reinforce our beliefs and persuade us to believe what they tell us.
Dr. In the late 1970s and early 1980s, Sejnowski was among a small group of researchers who began seriously investigating a form of artificial intelligence, a neural networkthat powers today’s chatbots.
A neural network is a mathematical system that learns skills by analyzing digital data. This is the same technology that allows Siri and Alexa to recognize what you say.
Around 2018, researchers from companies like Google and OpenAI started building neural networks that learned from huge amounts of digital text, including books, Wikipedia articles, chat logs, and other things posted on the Internet. By pointing out billions of patterns in all this text, these LLMs learned to generate text themselves, including tweets, blog posts, speeches, and computer programs. They could even continue a conversation.
These systems are a reflection of humanity. They learn their skills by analyzing text that people have posted on the internet.
But that’s not the only reason chatbots generate problematic language, says Melanie Mitchell, an AI researcher at the Santa Fe Institute, an independent lab in New Mexico.
When generating text, these systems do not repeat word for word what is on the Internet. They produce new text themselves by combining billions of patterns.
Even if researchers trained these systems solely on the basis of peer-reviewed scientific literature, they could still make statements that are scientifically ridiculous. Even if they learned only from the text what was true, they could still produce falsehoods. Even if they only learned from text that was sane, they could still generate something creepy.
“There’s nothing stopping them from doing this,” said Dr. Mitchell. “They’re just trying to produce something that sounds like human language.”
Experts in the field of artificial intelligence have known this for a long time this technology exhibits all sorts of unexpected behavior. But they don’t always agree on how to interpret this behavior or how quickly the chatbots will improve.
Because these systems learn from far more data than we humans could ever understand, even AI experts can’t understand why they generate a particular piece of text at a given time.
Dr. Sejkowski said he believed the new chatbots had the long-term power to make people more efficient and give them ways to do their jobs better and faster. But this comes with a warning for both the companies building these chatbots and the people using them: they can also lead us away from the truth and into dark places.
“This is terra incognita,” said Dr. Sejkowski. “People have never experienced this before.”