Google's NotebookLM AI will change my life-maybe yours too

By Fred Vogelstein

For the past two years - since ChatGPT ignited the modern AI revolution - I’ve wondered aloud to anyone who would listen about when AI was going to enable the large language model of me. Instead of crawling the public internet, it would create a queryable database exclusively based on the digital file cabinets of my personal and professional life.

I want this because I’m in the idea generation business. The best way to generate ideas is to identify the intersections of vectors - things that are happening or might happen - before anyone else. We do that today by reading and listening to smart people. But many of us also know there is a wealth of information in our digital files that would be super useful if only it didn’t take so long to read through them all and process the information within. Desktop search programs help a little. But even the best ones don’t approach what I’m hoping for. 

I’d like to ask my personal AI chatbot things like “How might I recast two book proposals I’ve written that didn’t sell?” Or  “Show me the notes and transcripts of interviews I did for a Google story I wrote in 2002.” Or “Generate the names and latest email addresses, sorted by story, of every person I’ve quoted in anything I’ve written. What did they say in those stories? Or “Describe my method for writing and reporting stories for a journalism school class. Or “Show me videos I took of my kids before they turned five.”

We’re not there yet. But Google’s NotebookLM is close. I used it for this piece, uploading a 75 minute interview I did with its editorial director, Steven Johnson. I’d guess it reduced the story production time by 25 percent.  I asked it to write a 1000 word story and a 2000 word edited Q&A based on the interview. Neither were bad first drafts. And they helped me draft this version much faster. 

It’s even more useful If you’re working on a bigger project. You can upload all your notes and research, and it will instantly become your AI assistant for that specific project. Tech companies have been promising something like this for generations. All that’s done has been to generate laugh lines for late night TV. Remember Clippy from the 1990s? 

But NotebookLM is slick, and it works almost as advertised.  If you’re an author or a magazine journalist, it’s like having someone who can instantly transcribe every interview, parse contemporaneous notes and research, and then create indexed interview summaries and timelines. In the past three months half a dozen journalists I know have talked about how transformative it has been for them. 

If you’re a teacher or a student it can instantly create study guides for course materials. If you’re an executive readying yourself for a meeting,  you can have someone upload everything your appointment has said over their career. It will instantly create a briefing document along with suggested questions. If you’re a scientist and need to do a literature review, it can do all the reading for you and spit one out for you complete with footnotes to specific passages in every article. 

I could see how teachers might chaff at its ability to take an entire course syllabus and spit out briefing documents, FAQs and study guides without students reading a page. I could also see how it would enable those teachers to leverage that technology and create much deeper comprehensive offerings where students might be responsible for two or three times what would typically be assigned. 

Technically, NotebookLM is just a fancy front end to Google’s massive Gemini AI operation, which has been improving rapidly like most AI operations out there. But Johnson and his team, which is now about 30 people, have worked hard to make sure that it’s both secure, controlled, and easy to use. It won’t use any information other than the information you upload. It won’t search the public internet, for example. And none of that information or even what the AI learns from processing your material is available to anyone else, he said. They’ve also spent a huge amount enabling it to suggest questions based on your material as often just thinking of the right question to ask is a stumbling block for users. (Three on the team just said they were leaving for a stealth startup)     

“It's a very literary/journalistic product,” Johnson said. “There's a disproportionate number of writers on the team because a lot of what we do is decide “What should the responses sound like? What is the tone? How do we elicit that from the model? How do we write prompts in plain English that bring out that behavior from the model?” All those things are really editorial questions, not technological questions.” 

Johnson says that the product is about to get vastly more powerful too. Google is hugely expanding the amount of information NotebookLM can parse in one gulp in the coming weeks from 32,000 tokens (about 24,000 words) to 1 million tokens (about 750,000 words.) The more tokens a system can process at once, the more accurate its answers will be. (For more on this, search “Tell me about the technology driving all this?” in the Q&A below.)

This boost is distinct from the storage size limits for each notebook. Users can upload and store a maximum of 50 sources per notebook, with a maximum of 500,000 words for each source and a maximum overall of 25 million words per notebook. The maximum number of notebooks is now 100. A new premium version NotebookLM Plus gets you bigger storage limits, among other things.

NotebookLM presents you with three windows.  The left window is where you upload and store source material. This is where you run into some of its limitations. It will process pdfs, Google docs and slides, and most, but not all audio and video. You can feed it links to websites and public YouTube videos and cut and paste as much text as you want. But any Microsoft Office files have to be converted to pdfs. The new, invite-only, enterprise version does support Word and Powerpoint, with much higher storage limits. Both versions work better with words than spreadsheets.

The right pane contains a list of various documents you’ve asked NotebookLM to create and offers option buttons that would create an FAQ, a study guide, a briefing document, and a timeline. The center pane is where you do most of your querying. It’s where it presents results, all with citations to the exact location in your sources. Click on a citation and it shows you in the left pane exactly where that information appears. 

Johnson said that Google’s approach to content rights is simple: If you have the right to have it on your computer or cloud drive to read it, then you have the right to upload it to NotebookLM. “Over time we want to explore the idea of actually having some rights management in it. I would love to be able to purchase an ebook and read it inside NotebookLM. But we would want to make sure that we were respecting the DRM behind ebooks to do that,” he said. 

Google publicly launched Notebook LM at Google I/O in mid 2023 as Project Tailwind. “By December 2023, it really started to come together,” Johnson said. “One of the things that we did was we built a lot of the tools, features, and design up front even though we knew the AI model wasn't quite ready to do the things we wanted.  But we knew the trend lines. So we were just waiting for the AI to catch up to our ambitions for it.  And that really happened in May of 2024 when we switched over to Gemini Pro 1.5.That was the point where we said “Oh, it's finally working.” 

But what really catapulted NotebookLM was its podcast feature that came out last fall. You can upload your resume,  audio from last night’s school board meeting, a day of the Federal Register - almost anything -  and it will generate a podcast-like conversation about the most interesting portions of those documents.  You can ask the software to create a podcast that focuses on any subset of the material you want. 

What’s created so much attention is just how real the conversation sounds. Google found a man and a woman voice model to train its systems. The chemistry between them is so good that it’s impossible to distinguish their conversation from the real thing - even though it’s entirely AI generated. I tried it out with my Johnson interview. It’s astonishing.

“Notebook LM is a tool for understanding complex things,” Johnson said. “That was why we were so excited about the AI podcast feature. We thought “What if one of the ways you want to understand something is to listen to it?” People often learn by listening to engaging conversations.” 

Johnson’s journey with NotebookLM began three years ago in the middle of 2022. He’d penned a long New York Times Magazine piece about the future of a new kind of artificial intelligence called Generative Pre-Trained Transformer 3, now known to all of us as Chat GPT. He raved about its next level intelligence but also worried about what it might do to society. He wondered whether we should even be building something so powerful at all. 

Back then very few had any experience with ChatGPT. It wasn't widely available. That wouldn’t happen until the end of that year, 2022. And those that did have experience with it found it theoretically interesting but not accurate enough to be super useful. In fact when Johnson's piece was published, he said many readers accused him of not being critical enough - "that I'd bought into the hype about these fake AI things," he said. 

But two executives at Google Labs read the piece and thought “More please.” They wondered if they could convince Johnson to work at Google Labs part time and help them build something specifically for writers and researchers - something that would enable them to have more control over their material. 

Johnson didn’t know the executives (Clay Bavor, who had helped start it and Josh Woodward, who now runs it) But they’d been reading his work for years. Johnson has written 13 books that all touch on various aspects of how science and technology affect us and our society. His Substack says “If you’re interested in how innovations drive change in society, and how we can learn to better align our broader interests with the march of science and technology, this is the place for you.” 

Bavor and Woodward also knew that Johnson was obsessed with writing tools - how and why some worked and others did not. Johnson wrote a 2800-word article in 2017 about how he used Scrivner and how it had sped up the time it took him to get from hunch to narrative. Another one in 2021 “How Do you Capture your Hunches” provided  details on eight other tools he uses and why.

“So I started (at Google Labs) part time in July of 2022 thinking we might build a prototype of something.  I'd meet some interesting people. I'd learn a lot. (And then it would be over.),” he said .”Google must have like 6,000 prototypes in development at any given point. I thought this will probably never see the light of day. There already was a small  project called Talk to a Small Corpus of Text. We baked that into a real prototype where you could upload parts of my books and have a grounded chat about those books. It was pretty cool. There was nothing else like it out there. 

“And then the ChatGPT moment happened (in November 2022). And here we were sitting on the beginnings of a new app that had been designed from the ground up for the age of language models. And so suddenly (Google executives) said “Oh, that's interesting.” And so we got a disproportionate amount of attention internally given that there were only about four of us working on it.” 

One of the most talked about stories in Silicon Valley over the past two years has been how OpenAI took the lead in AI with ChatGPT even though most of the technology had been developed inside Google. The 2017 paper “Attention is all you Need” by eight Google engineers is considered one of the seminal moments in the history of AI, maybe on par with the creation of the World Wide Web in the early 1990s.

“There was definitely a sense of  “We've got a lot of great  foundational (AI) technology. But we need to start putting it out there in the world. What do we have that we can put out there?” Johnson said. 

All of this has turned out to be good news for Johnson who decided fairly quickly that he needed to become a full time Google employee. “I realized quickly that I was thinking about this 130 percent of the time - in the middle of the night, when I woke up, every minute really, “ he said. 

It’s also turning out to be good news for Google because it helps Google make the case that despite being caught flat footed by ChatGPT, they are catching up fast. Johnson says that already 10 percent of Googlers internally are using NotebookLM including Demis Hassabis, who runs Google DeepMind and just shared the Nobel Prize in chemistry.

“We want to be the place you go if you are interested in trying the latest and greatest cool things from Google. A couple other places you could go are to Gemini, or AI Studio,” Johnson said.  “Ultimately I hope that we can be one of the defining pieces of software for this - where 10 or 15 years from now you can look back and say that was one of the programs that figured out how to do AI in the right way.”

Here are edited excerpts from our conversation: 

Fred Vogelstein:  You've spent a lot of time writing books. How did you end up doing this?

Steven Johnson:  I've always been interested in using tools for writing - programs like Devonthink and Scrivener. I've written at various points about the tools that I'm using. And then in the spring of 2022, I wrote this long piece for the (New York) Times Magazine (about ChatGPT3) It was annoyingly controversial. Critics said “Oh, he bought the hype about these fake AI things.” 

But there were two people at Google Labs, Clay Bavor, who had helped start it, and Josh Woodward, who now runs it, who had been long standing readers of my stuff. They read this piece in the Times magazine and were like, “I wonder if we could get Steven to come to Google Labs  part time and actually build a new kind of AI native tool for thought - basically a research and writing tool - since he's so obsessed with these tools.  

So they sent me a cold email that said “We have a proposal for you. At minimum you could come and give us a motivational speech every quarter and be an advisor. But we think it'd be more fun if you were there in the room 50% of your time. We've got a small team that can help you build whatever you want.”

FV: When was this? 

SJ: This is June. The first correspondence was May of 2022. So I started in July of 2022 thinking we might build a prototype of something.  I'd meet some interesting people. I'd learn a lot. But I also thought Google must have like 6,000 prototypes in development at any given point. So probably this will never see the light of day.

And so we built something really fast. We built a kind of internal prototype that from the very beginning, was not just that you were going to have an open ended chat with the model. You were going to be having a grounded chat with the model - grounded in the sources that you provided. It was all about, “Give it your documents, and that's what the model becomes an expert in.” 

And then the ChatGPT moment happened. And here we were sitting on the beginnings of a new app that had been designed from the ground up for the age of language models. And so suddenly (Google executives) said  “Oh, that's interesting.” 

FV: How did you come up with the idea for the podcast feature? Why has it been so important to NotebookLM’s success?

SJ: One of the things that we’ve come to understand over the last year is how to talk about NotebookLM. It’s a tool for understanding things. Just as Photoshop is a tool for moving pixels around, and Microsoft Word is a tool for creating documents, Notebook is a tool to help you understand complex things. 

The way we do that is to say “Give us whatever material you want and we’ll convert it into different formats to help you better understand it. So you can convert it into a briefing doc, or an FAQ or you can ask it questions, whatever you want. “

That was why we were so excited about the AI podcast feature. We thought “What if one of the ways you want to understand something is to listen to it?” People often learn by listening to engaging conversations. 

So we scrambled all summer to merge this other prototype into Notebook. Then we released that in mid September (2024) . 

That’s when the popularity of NotebookLM went insane. Up until that point, the best way to appreciate the product was to load up some very complicated sources, ask a very complicated nuanced question, get a very rich answer from the model with citations, and follow those citations back to the original text to read about further reading. That’s very powerful. But it’s hard for an experience like that to go viral.  

With the podcast it's just not something you can share and say “Look at this beautiful answer I got.” You could go “Listen to these two people talking about my resume.”  and share that on TikTok or whatever. 

The prompt is basically encouraging the host to find the most interesting things from the material that they've been given. It’s what journalists do . How do I make this complicated thing  intelligible and interesting to a general reader? And so one of the things that I often do when I'm exploring new material is I'll load it into Notebook, and ask it “What are the most interesting things here?” And just listen to what it comes up with.  That was never a search query you could do. 

FV: Tell me about the technology driving all this? 

SJ: There were a couple of things that came together. The underlying AI model is Gemini Flash 2.0. We are currently testing a 1 million token context window versus like 32K or something. And we'll probably switch over to that 1 million token model sometime in the next couple weeks.

FV: What are tokens? What is the context window? And why is going from 32k to 1 million tokens important? 

SJ: Tokens are just the word they use to describe the units of information that are fed into the model. (One token is ¾ of a word. So 32,000 tokens is about 24,000 words. 1 million tokens is about 750,000 words) 

The context window is the short term (working) memory of the model. So the bigger the context window, the better (the answer) particularly for the kinds of stuff that we do. If you've got 500,000 words worth of research material, and you could put that all in the context window,  then you're getting really great answers from the model because it sees every single word of your material. 

If your sources exceed the context window, particularly by a lot, the quality isn't quite as good. We might not grab the exact right passage, or we're eliminating things, or we're summarizing things, and there's a little blurriness there. But if you can fit everything into the context, then the accuracy rate goes up a lot. 

One thing that's important about this is that the Gemini team spent a lot of time training these models to be good at source grounding. “The user is going to give you these documents. You need to answer factually based on these documents. And stick to the facts in these documents. Do not kind of go outside that border. 

And Gemini is just very good. I would argue it's the best source grounded model out there because of the training the Notebook LM team was giving it. 

There's also a host of technologies in the audio overview side of things that make the voice models just sound incredibly good. They're trained on people in conversation, which is really important. It's not just two different voice models talking to each other. It's like a conversational model and some very beautiful prompting on that side. 

I didn't do it so I can, I can praise it effusively. To generate the script for the audio overviews, the system has a whole (blazingly fast) edit cycle to it where it generates an outline,  revises it and then creates a detailed script and then revises that. 

One really big important thing to mention is we are never training the model on any information that is not  uploaded as sources. That's why the context window is really important as well. All we're doing, the model's been trained separately on a huge amount of information. That is a separate deal that Google has done. 

We are simply taking your information and putting it in the short term memory of the model and asking questions. And the second you end your conversation, it is wiped from the model's history. There's no way for it to stick around in the model's general knowledge. There's no way for that information to be passed on to some other person. 

FV: Does the Gemini team limit how you use it or put conditions on it?

SJ: I would say it's the opposite, particularly in the last couple of months where we've had such a nice reception. The Gemini folks who are developing the core technology see Notebook as a really good place to showcase the new things that they develop. 

FV: What’s features are on the horizon? 

SJ: Over time we want to explore the idea of actually having some rights management in it. I would love to be able to purchase an ebook and read it inside NotebookLM. But we would want to make sure that we were respecting the DRM behind ebooks to do that. 

And that's something I'm hoping to do this year. I think there's also a market to have it curate the most interesting essays on a particular topic and, and then sell that bundle, that Notebook for something like $9.95. You upload all this material and you get back this little collection of knowledge about a given topic. 

FV: Thanks for your time.

---