Sutton's recipe for discovery

Good day, ladies and gentlemen. I regret that I'm unable to be with you all today to engage in a back-and-forth discussion, but I'm nevertheless pleased to be able to share with you via this recording some high-level thoughts about the current and future state of artificial intelligence, and in particular about AI's relationship to science and mathematics, which is, as I understand it, um the central focus of this meeting and of the Xprize Foundation. I would like to start with an old joke. I'm sure you've all heard it before. This is the one about the researcher whose work is being evaluated, and the review comes back and says, "This work is both novel and good. Unfortunately, the parts that are good are not novel, and the parts that are novel are not good."

My first point about AI is that this assessment it applies exactly to large parts of AI as we know it today. Not all of AI, but a large part of it, pretty much all of what we mean by generative AI, which includes large language models, and the images, and the video models, and even new methods for learning world models. All of these AIs take a large number of examples and produce a model which behaves similar to the examples, that is, which generates text like people, or images like artists or nature, and videos like we find on the internet.

Now, don't get me wrong, generative AI can be extremely useful. There's no doubt about that. But the assessment of the joke still applies. These systems can produce output that is both novel and good, but not at the same time. In many ways, this This just absolutely not a problem. you ask any AI for an answer from the internet or to summarize a document, we don't want it to be novel. We're happy if the if the quality of the answer, the goodness, comes from the source material. From the people who wrote the document or the articles on the internet. Um if the AI's answer is novel, it means going beyond the source material, adding something beyond it. This is what we call hallucinations. In most cases, we don't like it when when AI makes something up, when it adds something novel. One exception, of course, is when you're looking for the facts, when when we're not looking for facts or reality, but for fiction and entertainment. We might ask for a bedtime story for a child or for an image based on existing images, but which is nevertheless different and distinct from them. In these cases, it is never easier easier for us to know how creative the the AI is actually being, as we do not know how close the AI's story or poem or image is to the source material. In a real practical sense, we can't know this. We can't know this cuz the internet is too big. The possible sources that the AI may draw upon are too numerous. Now, when we ask for a fiction or a novelty, the AI can give it to us because its processing is in part stochastic. Every decision that the AI makes can go multiple ways, and it will go different ways, produce a different trajectory every time. The trajectory can be random and thus novel, or it can be based on the training data, and thus good because the training data is good, sourced from people or reality. Thus, the trajectory generated by the AI system is either novel or good. It's either based on randomness or it's based on data, but it is never both. It's never both at the same time.

So, really I think it's okay if the output of generative AI is never good and novel at the same time. You know, for the researcher in the joke, uh this is a devastating criticism. Uh but for most things in the world, for most things that you do, uh it's not terrible to not be both novel and good. And particularly for generative AI, it's fine. Generative AI is meant to be a mimic. This is This is what supervised learning is for. Uh generative AI can be extremely useful even when it just mimics. If it is faster or cheaper or smaller or more cost customizable or more copyable than the thing being mimicked. It is okay if creative AI, if generative AI cannot be both novel and good at the same time. It's still a transformative technology. But But this inability to be both novel and good is a limitation, an important limitation.

I remember we are here to use AI for science and mathematics. And for these areas, the assessment of the reviewer in the joke is is devastating. For these areas, we need true creativity and we need true discovery. Generative AI, I sometimes I call it mimicking AI, it will never get us get us there. For these, we need something more. And indeed, we have something more in other parts of AI. We have AI systems that that can give us more. We have AlphaGo with its world-changing move 37 or AlphaZero with its brilliant original chess-playing style. And we have GT Sophie, the drive simulated race cars better than any person. We have AlphaFold and AlphaProof and AlphaCode which have brought us advances in science and mathematics and science and and programming. We have RL lift, which optimizes the assignment of cars to passengers in the ride-sharing business. All these systems have found things that are both novel and good. And truth be told, some language models have been augmented in ways that make them more than generative AI, uh more than based just on supervised learning. But to the extent that they are, this limitation applies.

Now, all these other systems that I've listed have some additional features which make them more, uh which makes them capable of true creativity and true discovery. And it's important for us to recognize this, that this uh ability to to create and discover is not present ordinary garden garden variety generative AI. It is It is something that cannot come from just supervised learning. Uh cannot come just from learning from examples. So, what is this thing, this this additional ingredient? Well, it's a simple thing, a common sense thing. It's not really new. You have many names We we have many names for it, but unfortunately, we don't have a really good name for it. Let me just call it discovery as I have. Basically, discovery is just the idea of trying many things and seeing which work, then keeping those that have worked the best. So, evolution by natural selection works this way. The scientific method works this way. And just ordinary life and learning works this way. We try things and we remember what works. What could be more obvious? In this behavioral case, the last case, uh psychology has two names for this discovery. It's two names are instrumental learning and operant conditioning. In machine learning we have the name of reinforcement learning. Reinforcement learning is basically this discovery ability in computational learning agent. We also see discovery in planning and in combinatorial search. Really anything that involves the idea of generate and test, keeping the best.

Okay, the essence of discovery is to combine three steps: variation, evaluation, and selective retention. These three steps are the are key. And it's more than supervised learning, which doesn't have the evaluation or retention. Now now I'm not the first to identify these three things and point out that they're important. This combination of three things that I'm calling discovery are important. Uh it's important to science and then natural selection and to animal behavior. Uh there particularly I think of papers by Donald Campbell and Daniel Dennett and Gary Chico. Uh what is new in my remarks today is to directly relate the idea of discovery to modern AI and to help us see that it's not present in supervised learning and generative AI and in particular that that discovery is not present in back propagation or gradient descent.

Well, let me be a little more explicit. What is missing from generative AI? As as as I remarked, uh these generative AI systems do have stochastic aspects, so they do generate a variety of trajectories and behavior. The missing part is the evaluation step. The generator of genera- generative AI was pre-trained by supervised learning, and that leaves it no way at runtime to evaluate what it generates. And of course, without evaluation, there can be no selective retention, and thus no discovery. It has variation, that can bring novelty, but without evaluation, there's no discovery. And arguably, there's no creativity. That I That is, I would say, that creativity requires that the things generated be evaluated. Without evaluation, and retaining the best, there is nothing created. The novelty flickers into existence, but if its value is unrecognized, then it flickers away and is lost. I think what we really mean by creativity uh requires not just random generation, arbitrary generation, but a recognition of value, and a retention.

Now, in many cases, uh when we use AI, uh we we the evaluation part is provided by the human users. Uh just as like when we use generative AI AI to make a bunch of pictures for us, and then we pick the picture that we like the best. So, there's the random generation, there's the human evaluation, and the overall process is a discovery. A discovery of a good picture, or good writing, or a good poem. But, the more important case is when the evaluation comes from a clear objective. Uh so, some moves are better than other Some some moves lead lead to checkmate. Uh some steps Some mathematical steps lead to a proof, and some don't. Uh some actions that you take in the world resi- result result in high reward, and others don't. Some genotypes make more copies ultimately. Some theories explain the data better. So, the evaluation comes from the domain and what you're trying to do. So, in these important cases where we have discovery, we have evaluation coming from a clear objective that's known to the system.

Now, let's talk a little bit more about the variation step. Some people like to call it blind variation, where blind here means uninformed variation, a shot in the dark sort of thing. It It But, you don't need for your variation to be completely uninformed. Like, a good scientist does not select the theories that he tests at random. Uh so, we don't want to be completely uninformed, but it also can't be completely informed. It can't be completely determined. There must be a some uncertainty about where the answer lies in order for there to be discovery when you find it. So, in practice, the variation is always partly informed and partly blind, but it's the blind part that really corresponds to the discovery.

Okay. So, now let's briefly go all the way to modern deep learning, to the backpropagation algorithm. At first, it might seem that backpropagation is incapable of discovery as I've defined it, because backpropagation is deterministic and thus incapable of variation. But, this is not entirely correct. Uh the weight updates of backprop are deterministic, but the weights are initialized to small random values, and this provides some variation to start with. Now, this random initialization process is often downplayed, uh but it's in fact a necessary part of backprop, and it must be done properly to get good results. So, in backprop variation is done once when the network is initialized, so its effect is is only temporary. And later the network does tend to lose its ability to learn, its plasticity. And this is the weakness of deep learning that is alleviated by a new algorithm that my group presented in nature a couple of years ago. Our continual backprop algorithm made one small change that's every so often a less used neuron is reinitialized to small random weights and this allows the variation to continue and plasticity to be retained. Okay.

Now, of course, there's much more that can be said about creativity and discovery, but but I need to wrap up. So, let me emphasize the main point. Uh creativity and discovery they are more than what can be done by supervised learning, more than what can be done by pattern recognition or prediction, even more than what can be done with world modeling. These things are all really important, but they alone will not bring us to discovery. Discovery requires evaluation from a person or from an explicit goal. And only in that latter case with an explicit provided goal available to the system will we attain fully autonomous AI. So, this is my call to arms. If we want the full power of AI scientists, then we should share the goals with them, so that they can create, evaluate, discover, and in these ways fully participate in achieving the goals. Let's be bold. Let's automate creativity and discovery. Thank you very much for your attention.