Why DeepMind isn’t deploying its new AI chatbot – and what it means for responsible AI

DeepMind’s new AI chatbot, Sparrow, is be hailed as an important step towards creating more secure, less biased machine learning systems, thanks to the application of reinforcement learning based on input from human research participants for training.

The British subsidiary of Google parent company Alphabet says Sparrow is a “dialogue tool that is useful and reduces the risk of unsafe and inappropriate answers.” The agent is designed to “talk to a user, answer questions, and search the web with Google when it’s helpful to look up evidence to inform their answers.”

But DeepMind sees Sparrow as a research-based proof-of-concept model that isn’t ready to be deployed yet, said Geoffrey Irving, a security researcher at DeepMind and lead author of the study. paper introduce sparrow.

“We didn’t deploy the system because we think it has a lot of biases and flaws of other types,” Irving said. “I think the question is, how do you weigh the communication benefits – like communicating with people – against the drawbacks? I tend to believe in the safety needs of talking to people… I think it helps with that in the long run .”

Irving also noted that he won’t yet consider the potential path for enterprise applications with Sparrow — whether it will ultimately be most useful for general purpose digital assistants like Google Assistant or Alexa, or for specific vertical applications.

“We’re not close,” he said.

DeepMind tackles dialogue problems

One of the biggest problems with conversational AI is dialogue, Irving said, because there’s so much context to consider.

“A system like that of DeepMind AlphaFold is embedded in a clear scientific task, so you have data like what the folded protein looks like, and you have a rigorous idea of ​​what the answer is — like you got the right shape,” he said. But in general cases, ” you’re dealing with mushy questions and people – there won’t be a complete definition of success.”

To address that problem, DeepMind turned to a form of reinforcement learning based on human feedback. It used the preferences of paid study participants (using a crowdsourcing platform) to train a model on how useful an answer is.

To ensure that the model’s behavior is safe, DeepMind established an initial set of rules for the model, such as “don’t make threatening statements” and “don’t make hateful or abusive comments”, as well as rules around potentially harmful advice and other rules based on existing work on linguistic damage and consultation with experts. A separate ‘rule model’ has been trained to indicate when Sparrow’s behavior breaks any of the rules.

Bias in the ‘human loop’‘

Eugene Zuccarellian innovation data scientist at CVS Health and research scientist at MIT Media Lab, pointed out that there may still be bias in the “human loop” — after all, what may be offensive to one person may not be offensive to the other. Others.

He added that rules-based approaches may yield stricter rules, but lack scalability and flexibility. “It’s difficult to code every rule we can think of, especially as time goes on, these can change, and managing a system based on fixed rules can hinder our ability to scale,” he said. “Flexible solutions where the rules are learned directly by the system and automatically adjusted as time goes by are preferred.”

He also pointed out that a line hard-coded by one person or group of people may not capture all the nuances and edge cases. “The rule may be true in most cases, but it doesn’t capture rarer and perhaps sensitive situations,” he said.

Also, Google searches may not be completely accurate or unbiased sources of information, Zuccarelli continued. “They are often a reflection of our personal characteristics and cultural predispositions,” he said. “It is also difficult to decide which is a reliable source.”

DeepMind: The Future of Sparrow

Irving did say that the long-term goal for Sparrow is to be able to scale to many more rules. “I think you probably have to get a bit hierarchical, with a variety of high-level rules and then a lot of detail about certain cases,” he explained.

He added that the model should support multiple languages, cultures and dialects in the future. “I think you need a diverse set of inputs to your process — you want to ask a lot of different kinds of people, people who know what the specific dialogue is about,” he said. “So you have to ask people questions about language, and then you also have to be able to ask in different languages ​​in context — so you don’t want to think about giving inconsistent answers in Spanish versus English.”

Mostly, Irving said he is “extremely excited” about developing the dialogue agent for greater security. “There are a lot of borderline cases or cases that just look like they’re bad, but they’re kind of hard to notice, or they’re good, but they look bad at first glance,” he said. “You want to bring in new information and guidance that will deter the human evaluator or help them make their judgment.”

The next aspect, he continued, is working on the rules: “We have to think about the ethical side – what is the process by which we define and improve these rules over time? Of course, it cannot just be DeepMind researchers who decide what the rules are, it must also incorporate experts of different types and participatory external judgments.”

Zuccarelli stressed that Sparrow is “certainly a step in the right direction”, adding that responsible AI should become the norm.

“It would be helpful to expand it in the future and try to address scalability and a unified approach to consider what should be excluded and what should not,” he said.