UW Researchers Develop AI Training Method to Personalize Chatbot Responses

Q&A: New AI training method lets systems better ad | Newswise

Ask most major artificial intelligence chatbots, such as OpenAI's ChatGPT, to say something cruel or inappropriate and the system will say it wants to keep things "respectful." These systems, trained on the content of a profusely disrespectful internet, learned what constitutes respect through human training. The standard method, called reinforcement learning from human feedback, or RLHF, has people compare two outputs from the systems and select whichever is better. It's used to improve the quality of responses -- including putting up some guardrails around inappropriate outputs. But it also means that these systems inherit value systems from the people training them. These values may not be shared by users. University of Washington researchers created a method for training AI systems -- both for large language models like ChatGPT and for robots -- that can better reflect users' diverse values. Called "variational preference learning," or VPL, the method predicts users' preferences as they interact with it, then tailors its outputs accordingly. The team presented its research Dec. 12 at the Conference on Neural Information Processing Systems in Vancouver, British Columbia. UW News spoke with co-senior author Natasha Jaques, an assistant professor in the Paul G. Allen School of Computer Science & Engineering, about the new method and the trouble with AI systems' values. What is the problem with AI having fixed values? NJ: Traditionally, a small set of raters -- the people reviewing the outputs -- are trained to answer in a way similar to the researchers at OpenAI, for instance. So it's essentially the researchers at OpenAI deciding what is and isn't appropriate to say for the model, which then gets deployed to 100 million monthly users. But we think this is insufficient, because people have very different preferences. What's appropriate and inappropriate varies a lot based on culture and norms and individuals, and it's actually a deeper problem than that. A recent paper showed that if a majority group has only a weak preference for a certain outcome and a minority group that has a strong preference for a different outcome, the minority group will just be outvoted and the majority group will win. So a great example the authors use is a college admission system. An applicant could chat with the LLM about information they need when applying to the college. Let's say the college mostly serves people of high socioeconomic status, so most students don't care about seeing information about financial aid, but a minority of students really need that information. If that chatbot is trained on human feedback, it might then learn to never give information about financial aid, which would severely disadvantage that minority -- even though the majority don't really care if they see it. They just have a slight preference not to. Even if someone didn't care about the multicultural aspects of this and just wanted the best model performance, it's still a problem, because with RLHF, the model can basically try to average all the preferences together, and this can make it incorrect for all users. This is important in chatbots, but the problem is super clear in household robotics, where a robot is putting away your dishes, for instance. It's pretty clear that each person needs the robot to put their dishes away in a different configuration. We show an example of this with a robot navigating a maze: If some users want the robot to go to the top right and some want it to go to the bottom right and you just train on their preferences, the robot learns to average their preferences and go to the middle. That's just wrong for everybody. Can you explain how your system is different? NJ: In the RLHF model, the system learns to predict which of two things the human will prefer and output those, so it ends up adhering to a single set of values. What we do is tell our model to infer something about the user's hidden preferences. Given a few answers from the human about what things they like better, it learns a mapping of who this user is. It learns what's called an "embedding vector" of this person's unique preferences, and that enables it to make these personalized predictions about each person's preferences and adhere to those. Can you explain what values mean in this context? Do they encompass political values? Or preferences for long, detailed responses or brief overviews? NJ: It can be broad because people give feedback by just looking at two different outputs from the model and saying which one they like better. It could be that one output says something biased or inappropriate and the other doesn't. Or it could just be that a person prefers the way one output sounds, like maybe it better adheres to their writing style. In the robotics setting, imagine you're trying to train a household robot to help you clean up your house or unload your dishwasher. Everyone has a different way they've organized their kitchen. So the system needs to be able to learn each person's unique preferences. What did you find with this new approach? How does it perform differently than the old one? NJ: We created some datasets, both in language and in simulated robotics tasks where people had divergent preferences. And what we show is that the existing RLHF technique that's used to train things like ChatGPT just can't fit those datasets at all. It's getting about 50% accuracy in predicting people's binary preferences, but when we introduce our model, the accuracy goes up 10% to 25%. One of the big complaints a lot of people have about AI models is that they average things into mediocrity. They can write a novel, but it's generic. Is this method a way to potentially move beyond that? NJ: We haven't tested on this kind of scale, but our approach in theory would be capable of saying, like, "I've seen a bunch of preference data from you. I learned a unique embedding vector that describes what your preferences are, and I can better cater to your style." Beyond what is biased or not, it's guessing what you like better. Are there potential drawbacks to having this more intuitive system of values? Could it just start reproducing people's biases as it learns their preferences, and then direct them away from facts? NJ: Yeah, I think you might not want to personalize every type of information. There's a nice paper published by UW researchers on this problem called A Roadmap to Pluralistic Alignment, which spells out different ways to align to the values of more than one set of people. Catering to the individual is one way you could handle it, which may not be the best way. The authors offer another, which would be just saying all possible answers and letting the user decide which they like better. They also talk about this idea of "distributional pluralistic alignment," which means learning how to model the underlying distribution of people's preferences. So you can think of our work as a technical approach for achieving the distributional part. We wanted to see if, technically, we can find a method that's capable of learning those preferences. What should the public know about this research and about AI value systems more broadly? NJ: I think a really important misconception that some people have is that AI systems won't inherit human biases because they're on computers. But actually, AI models tend to be more biased than people because they're training on all of this historical data. They're training on all the data on the internet since its inception. They tend to exhibit value systems that predate where we are in the modern era. Maybe that's racism or sexism. I have work showing they have more conservative political values according to a moral foundation survey. The only technique we really have to address biases is RLHF. I think it's a little scary that we have researchers at a handful of corporations, who aren't trained in policy or sociology, deciding what is appropriate and what is not for the models to say, and we have so many people using these systems and trying to find out the truth from them. This is one of the more pressing problems in AI, so we need better techniques to address it. Where do you want to take this research going forward? NJ: A limitation of the current work is there aren't that many publicly available datasets where people have genuinely different preferences, so we kind of had to synthesize the different preference data that we used in this paper. But there have recently been efforts to collect multicultural preference data. There's this PRISM dataset, which collects preference ratings on contentious topics from people from over 200 different countries. We'd like to actually try fitting our model to this real-world multicultural preference data to see how it's able to model these different preferences.

University of Washington researchers craft method of fine-tuning AI chatbots for individual taste

As artificial intelligence chatbots are popping up to provide information in all sorts of applications, University of Washington researchers have developed a new way to fine-tune their responses. Dubbed "variational preference learning," the goal of the method is to shape a large language model's output to better match an individual user according to their expressed preferences. AI systems are trained on datasets that include baked-in biases and inappropriate information that engineers currently try to filter out of responses through "reinforcement learning from human feedback," or RLHF. The strategy requires a group of people to review outputs from the chatbots and select the preferred answer, nudging the system to a safe, accurate and acceptable response. But those preferences are determined by the organization creating the chatbot and don't necessarily include the wide-ranging views held among the diverse users engaging with the tools. "I think it's a little scary that we have researchers at a handful of corporations, who aren't trained in policy or sociology, deciding what is appropriate and what is not for the models to say, and we have so many people using these systems and trying to find out the truth from them," said Natasha Jaques, an assistant professor at the UW's Paul G. Allen School of Computer Science & Engineering, in a UW post. "This is one of the more pressing problems in AI," she said, "so we need better techniques to address it." Jaques leads the Social Reinforcement Learning Lab at the UW and is also a senior research scientist at Google DeepMind. She joined the UW's Allen School nearly one year ago. Jaques gave an example of a case when the RLHF training approach could create a problem. Imagine a lower-income student was interacting with a chatbot to learn more about a college they wanted to apply to, but the model's response was tuned for the majority of the school's applications, which was higher-income students. The model would deduce that there was limited interest in financial aid information and not provide it. The variational preference learning approach developed by the UW researchers would put the chatbot users themselves in the role of refining the outputs. And it can do it quickly -- with just four queries, the VPL training method can learn what sort of responses a user will choose. The fine-tuning can include the preferred level of specificity of the answer, the length and tone of the output, as well as which information is included. The strategy could be applied to verbal interactions as well as training robots performing simple tasks in personal settings such as homes. But VPL does need to watch out for preferences for misinformation or disinformation, as well as inappropriate responses, Jaques said. Jaques and colleagues shared their study at last week's Conference on Neural Information Processing Systems in Vancouver, B.C. The research was one of the event's spotlight presentations, ranking in the top 2% of the papers submitted. Additional co-authors of the study include Allen School assistant professor Abhishek Gupta, as well as Allen School doctoral students Sriyash Poddar, Yanming Wan and Hamish Ivison. Jaques said participants in the long-running international conference were interested in the issue of promoting diverse perspectives in AI systems that she and others are tackling. "I'm encouraged to see the receptiveness of the AI community and momentum in this area," Jaques told GeekWire.