In 2019, Michael Kearns, National Center Professor of Management & Technology in Computer and Information Science (CIS), and Aaron Roth, Henry Salvatori Professor in Computer & Cognitive Science in CIS, published The Ethical Algorithm: The Science of Socially Aware Algorithm Design.
Rather than rehash the social ills caused by AI-powered systems, Kearns and Roth’s book enumerates technical improvements to the algorithms that increasingly govern our lives. “Curtailing algorithmic misbehavior,” they write, “will itself require more and better algorithms — algorithms that can assist regulators, watchdog groups, and other human organizations to monitor and measure the undesirable and unintended effects of machine learning and related technologies.”
As part of the recently announced Raj and Neera Singh Program in Artificial Intelligence, Kearns will teach CIS 4230/5230: Ethical Algorithm Design, which introduces students to the challenges to privacy and fairness that many standard algorithms pose. For his part, Roth teaches a number of upper-level CIS courses on topics like fairness and privacy, all of which will be available to students in the program.
In The Ethical Algorithm, you define algorithms as “nothing more than a precisely specified series of instructions for performing some concrete task.” What are some examples of algorithms that people encounter in their everyday lives?
Aaron Roth
There’s a division that we make in the book between algorithms that are “hard coded” by human beings and algorithms that are derived from machine learning.
For traditional algorithms, one type that you encounter frequently is shortest-path algorithms: you type into your phone when you’re driving somewhere, and an algorithm finds the route in the road network that will minimize commute time.
You also encounter more complicated algorithms that are trained using machine learning models. If you use voice-to-text transcription on your phone, that’s also an algorithm. But nobody was hard coding the mapping from different phonemes to letters — that was trained algorithmically.
Michael Kearns
Another one is your Facebook feed. The input to the newsfeed algorithm is the entire inventory of things that Facebook could show you. The output is some reduction of that set to what Facebook thinks you would find the most interesting or that you are most likely to buy.
As you make clear in the book, computer scientists have largely focused on designing algorithms to achieve certain technical goals. You argue that algorithms should also be designed with ethical considerations like privacy and fairness in mind. What led you to conclude this should be a component of algorithm design, as opposed to just tech policy?
Michael Kearns
We saw the rapid advance in power and scope of machine learning starting around 2010 — the dawn of the deep learning era. Then we started seeing reports in the early 2010s of algorithms being used for purposes that touch end-users, and then what we would call audits of algorithms. You would see people going to commercially available face recognition engines and seeing if they underperform on different skin tones.
Aaron Roth
It became apparent that privacy was going to be a problem well before it became apparent that fairness was going to be a problem. Fairness concerns arise when you start using machine learning algorithms to make important decisions about individual citizens.
Before the deep learning revolution, people were using simpler machine learning algorithms to make relatively unimportant decisions, like recommending what movies to rent on Netflix. As soon as you’re training some model on the data of all Netflix users, or you’re coming up with a model on Facebook to recommend people you might want to be friends with, you start to be worried about privacy.
Michael Kearns
In short, we think it’s a good idea to bake ethical considerations into algorithms when it’s sensible and possible. It is not always sensible and possible — and many effects of algorithms are exogenous to the development of the algorithms themselves.
For instance, algorithms that predict criminal recidivism risk have downstream consequences in the legal system. There’s the algorithm and then there’s what the legal system is doing with the algorithm; you might not be able to address the way the system is using the algorithm in the algorithm’s design itself.
But from the beginning our stance has been that if there are things about algorithms that we don’t like, we should change the way we design those algorithms to avoid these behaviors. The first thing you should do is try to fix the problem in the technology itself rather than wait for the harm to happen and then regulate it in the courts.
The first thing you should do is try to fix the problem in the technology itself rather than wait for the harm to happen and then regulate it in the courts.
Michael Kearns, National Center Professor of Management & Technology in Computer and Information Science (CIS)
As you note in the book, it’s extremely difficult to give computers precise definitions of concepts like privacy and fairness. After all, many humans don’t agree on what privacy and fairness should look like. What do you see as the challenge and promise of trying to come up with quantitative definitions of “fuzzy” ethical concepts?
Michael Kearns
The words “fairness” and “privacy” are fuzzy because they mean many different things to many different people. If you’re going to explain ethical concepts to a computer, those concepts need to be in a very precise, quantitative form.
The definitions of fairness that we adopt technically in order to enforce better behavior in algorithms capture only a very narrow notion of what people might mean by that word in general. I think that most people, if they looked at the fairness definitions that are being enforced, would consider them to be necessary, but not sufficient. They’re statistical definitions that treat individuals as generic members of various groups — like their age group or their gender group or their racial group — and then try to equalize some statistical notion of harm across those groups.
These definitions, in particular, don’t give any promise of individual fairness. If you are a creditworthy individual who happens to be from a particular group and are falsely rejected for a loan that you would repay, your consolation is supposed to be the knowledge that creditworthy people from other groups are being rejected for loans at the same rate as your group. But that definition of fairness doesn’t change your outcome and it doesn’t give any promises to you as an individual.
Aaron Roth
To even get to the point where you can manage precisely specified trade-offs, you need to be able to quantify what the goals of an algorithm are. That was easier with the kind of machine learning that was around when we wrote the book.
One thing that has happened as the focus of machine learning has transitioned to generative models, that is, models that create text and images, is that computer scientists often no longer have as well-specified performance definitions — what do we want from these models? What does “good” look like and how do you measure it?
Michael Kearns
To take one example, ChatGPT is solving a very well-defined, precise problem: predict the distribution over the next word given the context so far. Just like with consumer lending algorithms, you measure false positive and false negative rates. The problem is that nobody knows what they want to do with this technology yet.
Compared to when we wrote the book, there’s a big gap between the objective that is being optimized and why people are interested in the technology. And this is basically a problem for the entire industry right now. Everybody’s fascinated by ChatGPT but also waiting for practical use cases that are much better than the technology we have.
You hear about using ChatGPT for generating ad copy or for summarizing meetings — these are all plausible uses, but until you settle on your use case, it’s hard to know what the ethical concerns are. Our view is that we will regain the ability to have science frame the societal debate around these algorithms when we break down tools like ChatGPT into different use cases.
Let’s think about your tolerance for toxic language in the output of a large language model. If I’m using it as an aid for bedtime story generation for children, my tolerance for toxic output is zero. If I’m using it for writing literary fiction, I probably have a much higher tolerance for toxicity in the output.
Aaron Roth
Privacy and accuracy are really in tension when you need to trade them off at some societal level. It’s politics and people fight about it — they need to try to find compromises. Figuring out how to trade off the legitimate goals of different stakeholders is not something that has a technical solution.
In the book, you describe a number of different algorithmic approaches and concepts related to privacy, fairness and other ethical domains, from generative adversarial networks (GANs) to the Pareto frontier. In the time since the book was published, what has changed, challenged or confirmed your perspective on the need for ethical algorithms?
Aaron Roth
The problems we talk about have only become more important, because we are embedding machine learning technologies in more and more products and systems. It used to be that to identify a privacy violation, you needed to be clever — now you can just talk to the model. People worry about copyright violation. Maybe DALLE outputs a picture of Iron Man — that’s copyrighted, the algorithm’s not supposed to do that. You can worry about a language model just telling you something about an individual that happened to appear in its training data. So the problems are only more important, and we know less about how to solve them, in part because we just don’t know very much about how these technologies work.
Michael Kearns
We’re optimists and remain optimistic. That said, I think we’ve lost some ground scientifically. One of the current frontiers in governing tools like ChatGPT is so-called “guardrail” models. A guardrail model sits next to ChatGPT watching what the user types in and how ChatGPT replies, and occasionally intervenes when it detects toxicity. To me this is a step backward scientifically compared to where we were, but it’s because the technology is more powerful.
Aaron Roth
As Michael says, we need fine-grained control to make models that don’t produce toxic content from the beginning. We don’t have a firm scientific understanding of how to train models that are good at downstream task A, like predicting the next word, and don’t do downstream task B, like producing toxic or biased content.
We don’t have a firm scientific understanding of how to train models that are good at downstream task A, like predicting the next word, and don’t do downstream task B, like producing toxic or biased content.
Aaron Roth,
As part of the new major in artificial intelligence at Penn Engineering, students will receive training in ethics. Can you describe what that ethical training will look like? In other words, how will Penn Engineering prepare the next generation of engineers to develop and use AI responsibly?
Michael Kearns
I teach an undergraduate- and master’s-level course called CIS 4230/5230: Ethical Algorithm Design. If you think of our book as a layperson’s treatment of why these topics are important and what science does or doesn’t have to say about them, the class is the technical version. We have a multi-week module on fairness, in which we introduce various technical definitions of fairness. I introduce students to the concept of the Pareto curve and then we talk about algorithms for actually constructing the models along that curve that present you with tradeoffs between fairness and accuracy.
Aaron Roth
I teach a number of upper-level courses, technical deep dives into privacy or robustness or fairness — they’re technically graduate courses, but are open to undergraduates. All of these courses will be available to students in the AI major.
In the book’s acknowledgements, you thank Raj and Neera Singh, after whom the new AI major is named, who also supported the Networked and Social Systems Engineering (NETS) program, as well as Fred and Robin Warren, who created the Warren Center for Network and Data Sciences. What does their support mean to ensuring that the new AI degree programs at Penn Engineering will lead to responsible AI development?
Michael Kearns
The NETS program was foundational in introducing a Penn Engineering major that was technical at its core, but was outward-looking and outward-thinking about the implications of that technology.
Over the decade that these programs have been in place, they have funded so much research and so many students interested in shaping the societal implications of technology — the new AI major will bring all of those resources together.