Memory and Continual Learning: Engram's Dan Biderman and Jessy Lin

[00:00] What about pre-training or even post-training makes it possible for the models to generalize in these magical emergent ways and controlling that process so that a company has a set of private data? How do we make the models learn that just as well as the models know, like the capital of France or, you know, like how to write Python? So I think it's a really fun problem to think about. [00:24] *music* [00:41] Welcome to Training Data. We are delighted to have Don Biederman and Jesse Lin, co-founders of Ngram today. Ngram is a neolab focused on memory and continual learning, and two of the hottest topics in all of AI research today. And Sean and I are delighted to dig in on those topics with you today. Awesome. Happy to be here. [00:59] Great. So maybe to kick off, the Engram website says, we don't see the world through the lens of pre-training or post-training. [01:06] Our models are always training. What does that mean? [01:09] So I think like models today, obviously, you know, a lot of things. They're incredibly smart. But we kind of think the bottleneck for making these models more useful these days is not really raw intelligence, but understanding like new and evolving context.

[01:39] And this is kind of why we think of ourselves as working on these fundamental problems of memory and continual learning, which are really two sides of the same coin. How do you make the models learn new things and bake them deeply into the weights of the model? And is your premise then that memory as a separate... [01:54] database or separate [01:56] you know, thing that you shove into the context window is not true memory and it's not true continual learning? I think all of these tools will kind of come together. So these days, like the way that people are solving these problems is with context engineering. So you take like a huge prompt, maybe you like keep talking to the model over many, many turns and hours and, you know, reorganize the context to better understand like what you're trying to do. And we think like these kinds of things like tool use, context engineering will play a part. But I think an under leveraged [02:26] Thank you. [02:26] training pipeline or framework or kind of workflow that the frontier labs are using to make these models really good at frontier math or code, but applying that to every kind of domain, every kind of context that you have, like, let's say in a company. Yeah. And to me, it's like as an individual taking notes and having sticky notes is a very valuable thing. We should never discard this. But whenever we get back to business the next day, we always have some sort of. [02:53] trace of memory in our brain, some new intuition about how things should be and where should we look. So these two things should come together. [03:00] And current solutions are more kind of externalized memory. And this has two issues. One is that the amount of tokens we will all collectively, individually generate is going to be in the tens of millions of tokens per day soon. So just keeping it and searching through it is going to be and rereading it is going to be pretty expensive. But it's going to also be pretty hard, pretty confusing for the models unless we have major, major breakthroughs. Tens of billions of tokens for Sean.

[03:26] That's good. Depends on the day. [03:30] Tell us a little bit about the Engram architecture, the Engram product and how it works. Yeah, I mean, at a high level, I think what we're trying to do is take any context. Like there's all these different workspaces, let's say. So we're working with partners like Notion, Microsoft and Harvey that have these places where people are doing a lot of work over a long period of time. [04:00] having conversations, giving them feedback, and figuring out how to have a model that deeply understands that context. So not just reading the files at test time, but really understanding it the way that an employee that's worked at your company for years has. So you kind of understand at a high level, oh, these are the initiatives across the company. This is the way that we do things. You've studied how to run the hiring pipeline or how to do this kind of thing within [04:30] as like anybody else can in the company. And so what we're doing is training per team models within these workspaces that deeply understand those contexts and can improve with time on the things that people care about. So the way that we do this at like a technical level maybe is training these into weights. So we do a lot of like adapter fine tuning. So adapters of many types, like I think people have looked into this for decades at this point,

[05:00] architectures. I think like all of these tools are at our disposal and then figuring out what the right data is. So how do you turn any kind of raw like document or interaction into useful training signal for the model? So again, we have like a variety of tools now, like supervised fine tuning, you know, RL, you know, on policy distillation, like all of these things that, you know, the field has kind of developed and trying to fit these pieces together into a model [05:30] the things that people care about. Yeah. And it's not a bet that tools are not there. Like our models always work under the assumption that some knowledge is externalized, some tools are always there. But what you need to do is you need to figure out, and that's the hard task, is what needs to be internalized and what can be externalized. And even for stuff that's externalized, many individuals and companies have their own bespoke tools and ways of doing things. Not everyone has the same, you know, Bash CLI tools that, you know, the frontier models are training [06:00] setup, I think, is its own interesting thing. [06:03] And so is the premise then that my notion agent will be... [06:08] a custom agent that is Laura fine-tuned or, you know, some way with an adapter, you know, [06:14] So that's constantly learning on new content that's added into my notion workspace. Is that the premise? Yeah, and they're working with many models and they're the early users of all the frontier models and they're probably going to keep doing that. Does this approach work on the frontier models or the closed frontier models? We need white, you know.

[06:32] We need white box access to the weights, right? So, you know... [06:36] We can partner with companies that have closed source weights and do this with them. But it's easiest for us to do it with open source models. But any model that's a transformer model, [06:47] we can do our thing to it. And what's the trade-off then when people are comparing the before and after using you? Is it that they're no longer [06:57] sending so much context [06:59] And so the trade-off is like you burn more compute up front to learn your company's way of doing things into the weights, and then you're sending less context to the model on every inference pass. Is that the rough trade-off? Yeah, that's one thing. The fact that you don't have to research things and reread things and the fact that you don't have to write monstrous system prompts, that can give you two orders of magnitude reduction in token inference consumption. It's not like... [07:26] you know 50 or it's it can be 100x fewer tokens because many things especially things that relate [07:32] to people and teams and organization and priorities. These are things that you can't really find in one document unless like you really have it really regimented and document everything. These kinds of things, the model can kind of implicitly learn by training on some of the data and answer, you know, within a hundred tokens, what the best frontier models would consume a hundred thousand tokens doing. So these kinds of examples are interesting. And also the quality, you know, there are tasks that are, you know, not supernatural for the current generation of the models. [08:02] be consistently this gap of like,

[08:04] three to six months ahead where there's certain things that are bespoke that people are just exploring. The models are not fully great for them. The models will at some point be great for them. But if you can autonomously learn in a very lightweight way, it will give value in that time in terms of capabilities. [08:19] Why train on the workspace level versus the individual level, for example? Either is fine for us. It's just easier to start with, you know... [08:27] Teams of people are more disciplined in how they collect context and in the amount of context they have over years. And it's easy for us to start there. But every person's computer and every person's phone one day is a useful target for our technologies. In fact, it will be very interesting to go there. We just think the big deposits of information are now in teams of people collaborating in knowledge work. [08:51] Is it a feature or a bug that there is so much... [08:54] fact memorization basically built into the large language models. And there's a school of thoughts that, you know, [08:59] The models just wrote memorizing the fact that the capital of France is Paris is actually a bad thing. And what we would prefer for the models to do is, you know, abstractly learn the concepts of countries and capital cities, but not to memorize all these facts in the weights. And so I'm curious what you think about disentangling. [09:18] memorization, [09:19] versus learning how it's done in the models today and then how you're thinking about approaching it. [09:24] Yeah, I think it's a really interesting question. Like, to some extent, you kind of need to remember stuff in order to like compose them into more complex concepts. I think the thing that's kind of missing is figuring out what's important to remember. And I think even now, when you think about like learning new knowledge, if you look at a lot of these academic benchmarks, it's like, how can we learn very specific facts, like, you know, the length of a bridge in this like African country. And that's not something that you really want the models to devote capacity for. And it's not something that we devote capacity to.

[09:54] So I think if you look at human memory, I mean, you can say a lot more about this, but like it's lossy because... [10:00] part of the future of intelligence is compressing what's important and separating that from what's not important. And so I think like you can't really separate. [10:11] fact learning from like non fact learning or skill learning as some people would like to think. Like if you take a model and like some people have done this with models where you like strip out, you know, like all the facts and just have it like the pure core or something like this. It's very unnatural as a model. It doesn't. [10:28] no basic things, and you kind of need that. Why do you need that? Why can't you look up facts and then just have... [10:37] I think if you look at how the models think, if you need to recall basic facts in order to take the next step in your thinking, you can't get very far. Maybe that's a high level intuition, but it's part of the reason why we think training is really important. In order to think more and more complex and different, [10:54] deep thoughts about things, you kind of need to internalize something so that you can compose them into more abstract concepts. Yeah, and there have been efforts before that were hard to scale to try and [11:04] you know, disentangle the two and pre-train the models in a way that's [11:08] you know, allows it to retrieve and search for things and not internalize them. It's just the recipe we know to hill climb on collectively right now. [11:16] is this, you know, fact pre-training step. And I think the magic of, or the mystery of this approach is that, you know, traditionally in CS, we would have, you know,

[11:26] databases as its own curriculum and we would have algorithms and the databases is like facts about the world and capitals of whatever, store them or query them. There's also algorithms of how do you efficiently manipulate information and get some answers in a sample efficient way. And I think the magic of deep learning is that these two things are now mushed together and we need all these smart people and anthropic interpretability to try and break them apart. And I think a lot of [11:56] is that these things are gradually separating again, where companies have their own context and they really handle them with care and engineer them with care. And there's a generic model that's completely a stranger through these contexts. And the model is operating on them. But for us, it's clear that [12:12] There needs to be a certain convergence, at least with some cadence, where the facts and the stories and the details are getting mixed into the model. It has disadvantages as well, because if you have to, you know, capitals of countries are, you know, they can change, but it's not very frequent. But there's many other facts that are changing all the time and just imprinting them into weights is a challenging thing to do. I see. So you're saying it's a false dichotomy to try to separate. [12:38] algorithms from databases here. What really matters is like how to distinguish what's important to remember versus what's not important. Exactly. And it's an open question. I guess it's part of how we dream and are you guys... [12:50] taking any inspiration from that in terms of ranking? Very, very loosely, I think. Just the idea that that's kind of a phase that's missing, maybe, where you take a context and you deeply internalize it. Right now, it's like everything happens at test time. You look at the context that the user gives you, and you do some thinking on the fly. But again, you can't get very far. Or you can get so far, maybe. And you make mistakes along the way. Like, how do you digest that back into the model so that next time you do it, you do it the right way?

[13:19] and make even more progress. - Yeah, and what are dreams? Dreams are pretty crazy things to say, we wanna build an AI that's like our dreams, it sounds a little bit like a nut thing to do. There's not a lot of coherence there, but what's interesting there is like, [13:32] what happens in our dreams. We, we, we see things, we talk to ourselves and we, we experiment with the affordances of what can we do and can't we do in the world and social situations. And, and, you know, any, any, it's heavily biased towards social stuff. Right. So for us too, with things we're building is, you know, you, [13:50] We give the models. [13:51] The time to then go back, retreat from the actual interaction and experiment with its affordances. What can it do in an environment? What does it know? How fast can it handle these kind of tail extreme things, the same ones that we dream about at night? [14:08] Do you guys... [14:09] come from academic backgrounds. Like what's a canonical example that motivates this problem or, or, [14:16] you know, [14:16] Or that's a win so far. [14:19] I have one example. Maybe Jesse can give another one. A hypothetical one, for example. Imagine one of the AI labs, say OpenAI, has to win some [14:27] Math Olympiad in a week time from now. [14:30] Would they construct a catalog of all the math textbooks and really have people annotate which chapters to get and which graphs to see? Or will they actually collect this, synthesize some training data, launch a training job, see where it lands in five, six days, start evaluating it? [14:47] and stuff like that. So it's obvious for anyone who's trained models that there's a superior way to integrate across the ideas and capabilities and it involves this kind of

[14:54] magic of training. And we are clear that this has to happen in those high stake domains of math and coding and cyber and stuff. We just think much of this magic can actually end up in the hands of many more people in interesting ways. [15:09] Like, why isn't it? [15:11] just the foundation model labs that own the [15:15] end product here? How do you go between giants? Yeah. So I think the worldview that we have is a bit different from the Frontier Lab worldview, where it's like, we want one model that's bigger and bigger, that's more and more intelligent across a variety of domains. Instead, how we see it, we kind of imagine this world where everybody has their own model. A lot of the things that people want to learn are either private, things that will never see the light of day in a post training data set, or even conflicting, like, oh, the way that I want to do the task is different [15:45] another individual wants to. And I think a lot of these things we're already seeing are hard to train into the models with the same tools that we have used for like decades in machine learning, which is like, you have really clean supervision, you have like ground truth reward signals, and you like create a nice environment and you like train the model to like use the tools to better accomplish this like coding process. [16:08] task. And instead, a lot of the things that actually happen out in the world are very ambiguous or like, it's hard to say, like, what makes something good. And so I think a lot of these things are very specific to individuals. And I think very kind of misaligned or not very aligned with how the Frontier Labs think about the whole training pipeline and what kind of models will exist in the longer term.

[16:38] pretty close with them. It's getting to AGI, getting this one generic model that's extremely capable. [16:44] and coding and math and then using it to automate the economy or to solve really hard, you know, long term problems. [16:50] cryptography and defense or whatever. And it's pretty clear what needs to happen to push this, you know, more pre-training, bigger models, more data, more RL, more inference time compute, that kind of stuff. That's P0. That's where the majority of expenditure and talent goes. And definitely all of them are thinking about memory and all of them are thinking about continual learning. It's just more of a product kind of effort right now. We think it deserves [17:21] We think breakthroughs need to happen there. And Demis and the Sequoia event about a month ago said pretty clearly that we need new breakthroughs around these topics. And obviously, they're thinking about them. We're just focusing exclusively on this. And we think certain things are on incentives of where the data is and who owns the model are pretty interesting. So if you could learn from many humans or organizations at scale without necessarily sending someone to work with them shoulder to shoulder, that would be a pretty big unlock. [17:50] point on that is like I think a lot of things need to look different in the world so one is there needs to be new research breakthroughs two is new infrastructure for training like you know small models for everybody rather than like one big model one big run and then the third I think is a different way of kind of combining research and product so right now I think like there's like researchers in these frontier labs they kind of train the model they throw it over the fence to the product team who then like

[18:16] prompts or context engineers like new products surfaces on top of the core models. But in this world where the models are always training, I think the inputs that users provide are very intricately tied to what the models learn from, like what the training signal is. And so there needs to be a lot more of a kind of integrated loop between like research and product. And so like while we're focused on tackling a lot of the core research challenges, and that's our background, I think we're also very focused on like how to deploy this as quickly as possible to like [18:46] actual feedback in the real world. [18:47] What motivated you to work on this problem? [18:50] I think it's obviously one of the grand challenges in AI. I think everybody's talking about it these days because the models are so smart. What else is left? I think learning at the edges, learning the remainders of what makes these models useful. It's not just about raw intelligence anymore, it's about learning new things. And I think it also feels very fundamental because it kind of goes back to really understanding what makes the model so good. [19:17] So right now the models kind of incidentally know a lot of things from pre-training and we don't really understand why. It's like the Internet was just, you know, this gift granted to us where there's like a diverse set of data that contains like all of these different examples of coding and like writing and all these other things. And it just happened that way. And now to figure out how to crack this problem of continual learning, it's about, [19:42] figuring out what about pre-training or even post-training makes it possible for the models to generalize in these magical emergent ways and controlling that process so that, you know, a company has a set of private data. How do we make the models learn that just as well as the models know, like the capital of France or, you know, like how to write Python? So I think it's a really fun problem to think about. And Dan, you came from the neuroscience world, is that right? Yes. Yes. So,

[20:11] I was initially interested in questions around consciousness and the human condition and things like that. Are the models conscious? I don't have any advanced thoughts on this more than you would read. I don't think so, but it's important that smart people are thinking about it. I would say I was interested in how humans think, how humans perceive. And as Amos Tversky, the Israeli psychologist, used to say, he's not interested in artificial intelligence. He's interested in natural stupidity. [20:41] see how people and animals experience the world. [20:44] And gradually, you know, my inclinations took me to the stats and AI domains. And there I figured that so many of the same problems of memory and continual learning are really, really urgent. And the kind of solutions we have in the current systems are pretty far from what we have in biology. And I'm not one of these people who would say that the machine should be like, you know, like the animal or the human brain. I don't think so. There's many things computers can do better than us. [21:14] store a whole code base or you can use a computer. You don't even need AI on the computer to store everything losslessly and just get it. But the human brain evolved to work in these constraints of information capacity and to have these fuzzy representations that can then be abstracted and form connections and form the next day. Current systems don't really have that beyond the generic pre-training step. And I was really interested in what are ways to build that in? What are ways to

[21:44] This is more of a philosophical question. [21:46] You mentioned in the brain, there's a bunch of different... [21:49] real estate, different co-processing units, whatever. [21:52] modern computer architecture, there's CPUs, GPUs, you know, memory, there's different co-processors with the bitter lesson [22:01] Do you think that [22:03] What's happening is that [22:05] Like-- [22:06] LLMs are, you know, converged to say like one... [22:11] coprocessor that's just [22:12] totally dominant. It's like [22:14] Everything all compute is going to happen in [22:17] you know, the GPU equivalent of [22:19] like a language model? Or do you think that... [22:22] these models are kind of [22:24] building... [22:25] a bunch of coprocessors, like... [22:27] you know, emergently inside the model? Like, you know, and take with memory, do you think that [22:34] the models themselves will just [22:37] build whatever [22:38] part of the brain equivalent would be that's, [22:40] good at memory or [22:42] Or do you think there needs to be like... [22:44] another standalone architecture with it. [22:47] Yeah, like is memory an emergent property? Exactly. And almost everything. Like is everything that we need in intelligence will just be emergent with [22:56] better training data and more scaled compute. Yeah, I would say just on a more like a superficial perspective on the current deployment of AI, it's way more than just GPUs. And we're seeing all these, you know, sandboxes exploding and models operating on other computers, trying things. I'm more mean on the model architecture level rather than on the... So other experiments, there have been many previous experiments, different architectures that we contributed to, like the state space family and others to try and handle very, very

[23:26] The thing with all these methods, it ends up being a trade-off, usually a trade-off between memory and accuracy. And memory not in the behavioral cognitive sense, memory in the computer sense, right? Instead of having the memory footprint of the transformer attention, which is quadratic in the sequence length, these models have… Some are claiming they have sub-quadratic. Yeah, some are claiming. And some do have it, right? [23:56] The thing is that in our hands, we find that you always compromise accuracy for this memory. There's no free lunch. And what we're saying is like, look, if you're really bitterless and pilled, what you want to do is you want to think, how can I burn more compute? And how can I burn it on new contexts that I have not seen before? So we're as bitterless and pilled as anyone else. And we are not betting that the overall direction of AGI is going to end anywhere soon. [24:26] scale and if I truly want to understand Sean and Sean's work and Sean's context [24:29] Just like rereading files is not going to make it, especially for a special person like you. We got to train 100 trillion parameters for this guy. Cosign, cosign. What are you finding... [24:46] that people care most about their models learning? Like, is it memorizing facts about the organization? Is it remembering like, ah, no, we do CI this way? Is it like, what are people actually hoping to do

[24:57] And then maybe this feeds into how you do the ranking of memory slots and all that. Yeah. Well, I think if you look at what people are spending their time in the app layer doing these days, it's a lot of just trying to make the model work well for your use case. Like, oh, I want the model to, like, you know, let's say, like, design my website with my brand style. Like, that's, like, a, you know, very common example these days. [25:27] you know, kind of your particular way of like writing, let's say. So there's many, many kinds of things. And honestly, like, I think when we think about these methods, kind of going back to this distinction between like facts and skills, there really is none. I think the methods are kind of agnostic to that. Yeah, to me, it's like the natural thing, almost all the app layers are [25:51] basically, you know, [25:52] a frontier model wrapped in a loop with search tools and stuff. And what they're all interested in doing with us is finding ways to kind of interface with their data in a way that's, you know, [26:04] faster, more efficient, and also is more contextual. So almost all of them, it's like [26:10] We want to have our firm knowledge be encoded in something that's more efficient that I don't have to research. We want to have the model know in a targeted way who's the person I should triage a thing to. And we're just showing them that with pretty lightweight training, these things can be instinctual to the models. They don't have to.

[26:28] have these very involved long REPL loops to solve them. So it's, in a sense, it's like, you know, it's a rag killer kind of thing. Again, we can always do rag and we can always retrieve, but that's the thing that people are interested in, interfacing with very large data planes and automating very repetitive things this way. Yeah. And I want to double click on this rag killer thing, and I'm sorry to beat a dead horse, I just don't fully grok it yet. Yeah. [26:54] Is the premise that there's some trade-off between doing RAG versus updating your model weights? [26:59] Is it an idea that you should be doing both? [27:01] Like what types of things should be done in the weights versus what types of things should be externalized to rag? I think it's an unsolved problem. I don't think anyone has answered to it. We're all working on it. [27:14] it's also the fundamental question of like biological memory, what should be internalized versus what not. Um, [27:21] I do think that things that are like, you know, do you need to internalize the room number in a hotel that you were in like a year ago? Probably no, not in your neural tissue. Probably that's good to write down. But do you need to internalize maybe the password to your home right now? Probably it's useful for the next few years to have that imprinted somewhere. So, yeah. How does this translate into like knowledge, work and products? This is still something we figure out and we try to take the approach that [27:48] We try to use as few heuristics as possible. It's easy to run filters on the data and say, like, I'm going to keep this, discard that, train on this, train on that. But as humans, you know, we watch TikTok and we, you know, get exposed to a lot of garbage and still the brain is able to learn and not completely go off the rails. And we think models should be the same as well.

[28:05] Yeah, maybe concretely in the short term, I think a lot of what people are worried about these days is the huge inference costs of running these agents like. [28:14] for days on end. High inference costs a good thing. I mean, consuming tokens for what? Sonia works with fireworks. She really loves Zyant. I love inference. We love inference too. Yeah. So I think it's like in the short term, I think that's the immediate pain point. Like, why are you reading the same files over and over again? You know, even in the same query, but like definitely, you know, across people in the same company, they're running the same queries on the same documents over and over again. And that should be something the model just knows. [28:44] Like in the same way you ask an employee, they don't, you know, type into the search box, like, what was I working on yesterday? They just know. But doesn't caching kind of solve that? [28:53] I think to some extent, yeah. But I think going back to this, like, [28:57] question of [28:58] what should be internalized versus what's like something you retrieve at test time. I think, again, like a lot of it is about building on your knowledge. So if you are always doing raw, you can't make associations like, oh, you know, I see somebody, you know, on the team is doing this kind of research. And I kind of like recall. [29:18] at an abstract level, oh, there's this related thing that you might want to know about. You didn't even ask about it, right? But I think these kinds of associations can only happen in weights because they're not really about, you know, you asked me to search for this, I'm going to search for this. Yeah, and also I think the main limitation with retrieval systems in general...

[29:36] and AI specifically is like, [29:39] The problem is not so much what to store and where to put it. The problem is how to address it, how to query the thing. Do you know what to look for even? And this involves some sort of intuition that sometimes the models don't have, interestingly enough. They don't know where to look. And especially if you're limited to the current way of doing things, which is keyword search. It is just easier to scale an RL and least involved in terms of infra for embeddings and stuff. So, yeah, knowing what to search is something that's intuitive and can happen in the way. [30:09] Like, [30:10] Much of this company started with us taking a deep dive into KV caches and caching. [30:16] And this is a fascinating thing, right? KVCache is a monstrosity of the current way of doing things that, you know, think about it, a KVCache for a single like Wikipedia article for some reason. [30:29] Taylor Swift or something like this, it will be like... [30:31] 80 gigabytes of HBM memory on the GPU and an entire llama. It's for, say, a 70 B llama model. And the entire weights of the model would be about 100 gigabytes. And, you know, with with some distortion, they remember the entire Internet. [30:47] And how come this thing is so, one thing is so bit efficient? And we have this proof of existence that [30:54] Gradient descent can pack a lot of information in very few numbers. [30:58] Whereas this... [31:00] KV cache thing, you take a few tens of kilobytes of article and it becomes those 80 gigabytes of brain state. So make sure you can cache this, you can load this, you'll have issues with disk to HBM stuff, people are working on it, it's pretty interesting, but what if we can take those 80 gigabytes of brain state?

[31:18] Spend some compute offline. [31:20] maybe also in fireworks, but then compress it and make it really, really small. [31:26] so that the thing we load in cache is like, [31:28] 1000x smaller. That would have tremendous implications for how we load things, how fast we can do things, and what the fidelity of the representation is. [31:37] Super interesting. Yeah. What are some of the things that could happen... [31:41] in the next. [31:42] year or two that would be like the chat gpt moment of memory [31:46] Or do you think that that's not how... [31:48] things will play out it's a good question um [31:51] I don't know. I think like... [31:53] the first proof of concept of the thing that people keep talking about with continual learning, which is you have an intern that you can teach things over time and it actually gets better. I think everybody's waiting to see that. [32:03] And no matter how sophisticated the context engineering approaches are these days, they're not getting there. So I think you need all of these tools at your disposal to make that happen. But I think it will be something like that, where it's like the model's actually getting smarter. Like, whoa, it's different from yesterday. Yeah. And it's important to say that the ChatGPT model was not anticipated. We just read about all the different product directions that certain people had before ChatGPT was different. [32:33] example is like look if you [32:35] you know, resign from your job today and your sole mission was [32:38] to make a model that's better for you. And you would use OpenAI, Anthropic, and all these frontier models, and you would just 24/7 engineer the context right skills [32:47] your way to move the needle is very limited as an individual. You'll just be better off waiting for the next version of the model, and you'll take it from there.

[32:54] And we would like to see a future where actually the more time you spend on the thing actually translates to the quality of performance, at least in the things and domains you care about. [33:05] And this is pretty hard to achieve and the only reason it [33:09] we think it could be achieved is if you start scaling compute and training on these data without destroying them all importantly, which is pretty hard. [33:17] just a couple, like, [33:19] This is just for fun, like rapid fire questions going off. Yeah. Just memory. When's the last time you reached surprised about something in AI? [33:27] in any area when reading about fundraising. [33:30] A lot of surprises every day. I would say all of us felt a little bit of a change around the capabilities of the coding agents. That's true. But we've been dabbling with these things and trying to make them work in more effortful ways before, so it didn't come as a complete surprise. [33:48] But yeah, I think... [33:50] To me, the main events were... [33:51] GitHub co-pilot, that for me was just the main event and chat GPT. And then seeing the agentic stuff, we all anticipated, I think, and different people had different expectations on how far it can go and how long horizon it can go. [34:06] but I feel, yeah, it's, we're, we're yet to see something fundamentally different and people are working on completely new ways of doing things now. Um, [34:13] But yeah, to me, its model is actually changing in a way that's not [34:17] harmful and learning new things on the fly that are personally and economically viable. [34:24] That's interesting.

[34:25] Right now there's this idea of like we're each going to have a token wallet that we're going to bring around to companies or to different apps. [34:32] Um... [34:33] different workspaces. Do you think that we're gonna end up with like a memory bank, a memory wallet that we're gonna [34:39] move around, [34:40] across the digital world as we go. [34:43] I think it's an interesting question. I don't know if we've... [34:47] fully figured out what the right kind of like products, [34:50] form factor is in the sense, in a way, even with like chat GBT memory, let's say I kind of don't want it to remember across my like personal and work context. Oh, yeah. Like, it's like, oh, you know, you might like these sheets because you trained a model on a GPU last week. It's like, that's totally irrelevant. And to some extent, it's like because the memory is flawed. But also, I [35:12] I guess tools and the products that you use to be separated to have control over that. So I personally think like there needs to be some separation there but [35:21] I guess, to be determined what that might look like. Yeah. And like, I think the Holy Grail is like, [35:26] you go to work and you just burn through all these tokens and you create all this value and [35:31] And somehow, you know, all the IP and stuff stays with the company, but somehow the skills you learned, the things you invented, your ways of doing things, some of them you can take with you as well to your next job in a way that's, you know, sanitized and not, you know, harmful to any other company's IP.

[36:01] push the frontier and implement AI more deeply in our companies, in our individual life, and then be rewarded for it. [36:08] I started a PhD in the SaaS firm in 2007. [36:11] at Stanford and [36:14] AI was boring as hell at the time. It was all statistical learning. And there's basically two areas, like computer vision and NLP. So like vision and... [36:24] language were kind of the two areas, and I think that's still true. [36:27] In 2012, Alex, that happened. Vision was dominating for six years or whatever. Are you guys surprised that... [36:36] language seems to be like the language approach seems to be like dominating over vision in progress. Question two, [36:43] Do you think Vision has any chance of coming back? How do you think about this? [36:48] Yeah, I think it is pretty surprising to me. I mean, some people maybe saw it coming, but I think I've always kind of been interested in language as like... [36:58] I don't know, I guess like a medium for communication and like so many kind of complex abstract things can be done in language. I do think like [37:08] I imagine in the longer term, language and vision will combine in this more unified system where we take in inputs from all of these different modalities and understand them in this abstract way. [37:23] Yeah, to me, I've never been interested in language. It seemed to me... [37:27] such an advanced capability that, you know, is very, the entire animal kingdom has very different forms of speech and language than what we, you know, and how we communicate with ourselves in writing. And I was always, as many other leaders in AI had this thought that, you know, the natural thing is you have to experience the world, act in it, in vision, in action, that will be the key.

[37:57] do some work at mosaic and stuff like that to learn how the sausage is made on the, on the NLP side. And the thing that's striking is that like the, the, [38:04] Language should be pretty hard, like each word has this one hot embedding vector that's [38:11] as dissimilar to any other word than it is, you know, [38:15] It's a completely high-dimensional space, and it's really artificial in a sense. And we learn it with models that are order of magnitude bigger than the best vision models. [38:26] And still, you know, things work pretty well. I do think there's a lot of juice to be squeezed in an image and video. And I think you guys are doing good investments in this space. But I think the two would keep being interesting in different ways. [38:41] I mean, I'll tell you my, that was my lead up. Now I'm going to tell you the crackpot theory. [38:45] I like, [38:46] And this podcast is not for me to pontificate. It's for you guys. But this is something I've been thinking a lot about, and you're the right people to share this with. I was pretty shocked that language kind of surpassed vision and... [39:03] I underestimated what was happening with LLMs in like 2018, 2019, 2020 because... [39:09] I just had this bias towards vision. And when I look back on it now, I think what's basically happening is that in biology... [39:17] Like vision has a massive fundamental advantage over language in biology and [39:23] And maybe I'm wrong, but basically like the... [39:26] Bitrate

[39:27] that you're [39:28] brain can process [39:31] optical data through the eye, [39:34] is... [39:34] I'm not a biologist. This is just kind of my dumb... [39:38] assessment, it seems many orders of magnitude greater. And there's a lot of like, [39:43] optical processing that happens like even before you reach you know like [39:48] electrons, [39:49] And so it's just like the total... [39:51] bitrate that is of training data that's kind of [39:55] being processed and then making it to your brain seems many or so 90 greater than the audio [40:01] data where, you know, it's [40:04] sound waves where sound waves are fundamentally like, [40:07] much slower bit rate than light. Yeah. And then there's almost like an upscaling from the acoustics to electronics, which make it into your brain, where it's like a downscaling. [40:20] from photons to electrons with vision, whereas in computers, [40:25] Today, everything is electronic. So it's kind of like, [40:29] You nerfed vision. [40:31] and you promoted [40:33] language where the [40:34] It's like all processing is on the same playing field. It's all electronic. And I just... [40:40] I think this is like my crazy-ass dumb... [40:43] non-technical crackpot. [40:46] theory, but I think this might be part of why [40:49] just like from an information theory perspective that, [40:52] Like maybe language and vision are on a similar playing field by the time you get to like

[40:57] LLMs and then LLMs are [41:00] We're just a really, really smart architecture that's better suited for [41:04] language than for [41:06] Vision. [41:07] um how dumb does this out especially to you don the neuroscientist Jesse also has some background in cognitive [41:14] computational science, right? So I would say my point here is like, look, much of what we're doing in knowledge work, we haven't evolved to do, right? We're sitting on these computers, reading these things, writing these memos, whatever. [41:26] We are not evolved to do this. It's new to us. Our brains are not wired for this. Still, nevertheless, it's useful to have LLMs to do this for us. And, you know, as humans, we're heavily vision biased. You know, other rodents are more olfactory biased, and I've worked on these things myself before. So what's the real estate in the brain that's allocated to vision and, you know, occipital lobes versus like language areas, temporal lobe, probably more vision. I'll have to check with chat GPT, but I think that's the situation. [41:56] No, no from memory? No, man. I'm externalizing. I'm a big rag believer in my personal lifestyle. But I think- In the limit, it's all rag. I internalize just important things like my emotions to you. No, just kidding. Sorry. [42:17] Yeah, and vision is dominating. When people are training vision language models, they end up... [42:20] The language ends up dominating the vision content there. But yeah, it's hard to say that because a certain brain is more biased towards a certain modality, it doesn't mean necessarily that we're going to more efficiently do it. I do think that efforts on brain-computer interfaces should take this into account. How do you then relay it back to the brain? That's where I think it's really important to think, what real estate do we have there right now? But for knowledge work, it's equally fine if it's text, I think. Last question.

[42:50] We're all in it. [42:52] I think I'm imagining like a world where everyone has their own model that is really different from the other person's model and from the frontier model. And all of these kind of serve different purposes. And to have a model that really, you know, I think people often talk about like knowing you, but also like kind of like helping you in the ways that make sense to you personally, whether it's like an individual or a team. [43:22] of intelligence everywhere. [43:24] Yeah, and to me actually, [43:26] It's a variant of the story where like [43:28] you know, [43:29] In neuroscience, we know that memory and navigation are pretty closely related. Same circuits in the brain that represent landmarks in space are in charge of some elements of episodic memory and things like this. [43:40] And for me, I think the company can be [43:44] the actual LLM interface to the data plane for everyone. So sharing some similarities to [43:51] great companies like Databricks and Oracle, where we form these memories that happen to be neural memories with models that happen to be personalized and happens to be there's hundreds of millions of them, but they're basically [44:01] and neural interface to the data plane in a way that's very different from what we know. And it's more efficient, it's more associative. It's not representing the file system as it is, it's representing a brain state of that file system. So that's for me a vision. [44:14] Beautiful vision to end on. Thank you guys so much for coming by the show with your building. Awesome. Love it. Thank you guys.

[44:20] Thank you.