Behind the Scenes of My Interview Research Process — Andy Matuschak Crashes My Crib (#162)

This episode is a little different: I’m the one being interviewed—and my interlocutor is Andy Matuschak, an independent applied researcher focused on "tools for thought" (ways to augment human intelligence). Andy founded and led Khan Academy’s Research and Development Lab, and prior to that, he was a senior engineer at Apple where he helped build iOS. I first discovered Andy’s work in 2021, and it was a game changer for me and the podcast. We spoke on the show in 2022.

In 2024, I recorded some podcast interviews in the US, and had the pleasure of hanging out with Andy while I was in San Francisco.

In October, Andy dropped by my place in SF to go behind the scenes of my podcast research process and interview me while I prepared for a conversation with Larry Summers. This is an unvarnished, unfiltered look at my tech stack and how I prepare for my interviews. I'm grateful to Andy for suggesting the idea and for so thoughtfully drawing out my current strategies, tactics and tools.

I support Andy's research. If you'd like to do so too, go here.

If you'd like to access my interview research notes for podcast interviews, you can support to access here:

• Richard Butler
• Larry Summers

Video

Sponsors

• This episode is sponsored by Math Academy, a fully-automated online learning math platform. For a limited time, new customers can get their first month free. Go to mathacademy.com. Use the code "JOEWALKER".
• This episode is sponsored by Vanta, which helps businesses automate security and compliance needs. For a limited time, get one thousand dollars off Vanta at vanta.com/joe. Use the discount code "JOE".

Transcript

JOSEPH WALKER: So I'm here with my friend and key intellectual influence, Andy Matuschak. Andy is a former guest of the podcast. He was an iOS engineer at Apple. He led research and development at Khan Academy, and he's now an independent applied researcher thinking about tools for thought.

And we've had some really fabulous conversations in San Francisco over the past couple of months. And today, Andy is going to be looking under the hood of my preparation process for my podcast. So, Andy, welcome. And is there any context or introduction you'd like to add? 

ANDY MATUSCHAK:  Yeah for sure. I mean, so the reason why I'm here is that you have an extremely unusual process for preparing for your podcast. In some way, you are a kind of athlete of learning, along some really unusual axis. And because my research is all about learning, I am very curious about your process. And I want to ask you many, many questions about how it is that you go about learning. But just so your viewers have some idea of what we're getting at here, tell me, at a high-level, what preparing for the Taleb interview was like for you?

WALKER: That one was sui generis in that I also did about two months of work with a private tutor, where we are working through the Blitzstein and Hwang Introduction to Probability textbook and some of the Wasserman All of Statistics textbook.

I also did Nassim’s two-week, Real World Risk Institute course. So that was, I think like five hours every night for two weeks, ten business days. So that had an enormous amount of background or lead-up prep before what I would call the intensive prep process. 

MATUSCHAK: I just want to highlight this: 50 hours just for the last course. That was not “intensive”. Plus the tutoring—not “intensive”. So what's the intensive prep, Joe?

WALKER: Yeah. Before we do that, let me try and add up how long the non-intensive prep would be. So 50 hours for the Real World Risk Institute course. And then tutoring every Saturday we might do like five hours. We maybe did that eight times. So we’re already at about 90 hours of work. There's also a sense in which I've been preparing for that podcast for the last eight years. Reading his work and being an interested reader of his. 

But then the intensive part is 1 to 3 weeks before the interview where 8 to 14 hours a day you're: reading as much of their stuff or stuff as relevant to them as possible; note-taking practice; spaced-repetition memory prompts—so every morning I'm doing prompts on the previous day's material. 

And for Taleb that intensive process would have been—because I'd just come off the back of the interview with Boyd and Richerson in San Francisco, I didn't have too much turnaround time—maybe nine days of intensive work for Taleb. That's every day.

MATUSCHAK: So call it another hundred hours.

WALKER: Another hundred hours.

MATUSCHAK: So this is what I'm so interested in here. This is a kind of “extreme athletic learning:. There's like, 200 hours of very focused learning happening for this particular interview. And then, of course, your prep for the interview, which is tied in the second hundred hours, where you're preparing your questions and the structure.

WALKER: Right. 

MATUSCHAK: That's part of the learning process. And it's also part of what makes the learning interesting, because there are some other super learners that we could talk to who are just very attracted to the idea of: I want to check all the boxes. I want to know probability and I want to know statistics and I want to know economics. And so I'm going to work through all the textbooks. But the learning that you did is motivated by this particular conversation. And, you have this extremely concrete thing that you were trying to be able to do with great excellence. 

So I'm really interested in digging into what those 200 hours look like. And of course, it’s so interesting that then, you finish the Taleb interview and then you have Fukuyama, then you have Larry Summers. And so these things just continue and compound. And so I'm also very interested in the relationships between these various sprints. 

So to begin, tell me about how you orient? So you've just booked the interview, perhaps you've booked the interview a while ago. And now that particular person, Summers [let’s] say, is the next one up. And you're getting started for your first hundred hours. There's an enormous pile of material that you could read. There are many textbooks that you could brush up on. So you have to do a lot of satisficing. You only have 200 hours to spend learning this stuff.

So how do you think about orienting to it? 

WALKER: Right. So one of the reasons I was very excited to speak with you today was I haven't articulated a lot of this stuff. And even though I'm kind of living it, I haven't codified or documented this process. So [I’m] kind of thinking through answers to these questions or articulating them for the first time in this session.

So I think there are a couple of things. One is if it's an episode like the Taleb episode or to an extent, the Boyd and Richerson episode, where I'm not only trying to understand the person's work, but I'm also, somewhat as a prerequisite, trying to master a sufficient amount of the field that that work sits within. Then I'll firstly be speaking to people who might be able to advise me about a syllabus. So I did that with Taleb. I spoke with a friend who's a professor of statistics. I spoke with some other friends who work in finance and have studied statistics and probability. 

MATUSCHAK: Do you tell them that you're trying to master the subject? Do you tell them that “I'm interviewing Taleb and I want to be competent?” What do you ask them?

WALKER: Both. 

MATUSCHAK: Okay. Do you give them some sense of the constraints? 

WALKER: Yes, the time constraints. 

MATUSCHAK: What do you tell them? 

WALKER: I can even pull up my messages, but it would be something like: so I've got two months, and I want to get to a pretty decent undergraduate understanding of probability and statistics. What should I focus on? 

MATUSCHAK: And you want that, so that you can understand the primary works? 

WALKER: Yeah. I'm trying to furnish myself with context for the interview and the guest's body of work. 

MATUSCHAK: Right. So the concern is that  if I were to try to be in your shoes and prepare for this interview with Taleb with, like two years of undergraduate mathematics, from 14 years ago, then reading his books, I would somehow be deficient. I wouldn't understand the theories deeply enough. There's something where you really need to be able to engage, you know, to ask the good questions, to participate with Taleb in this interview the way that you want to show up. 

WALKER: Yeah, exactly.

MATUSCHAK: Great. So, you ask these people. They give you a syllabus. They give you a reading list.  They send you textbooks —You're not going to read the whole textbook, right? So how do you decide what to read?

WALKER: Yeah. So again, you're usually consulting the same people and asking questions like which are the most important parts of this textbook? Which are the most important chapters? Which are the most important topics to focus on? Is there anything that you think I can kind of leave out or omit in my work, given these time constraints? 

MATUSCHAK: Okay. And with Taleb you were working with a tutor? So how does the tutor relationship relate to the syllabus that’s provided by these people? 

WALKER: Yeah. So, a shout out to Tom. Tom has a physics background and we belong to this math group in Sydney. There's a Signal chat. And I posted a message in the group asking whether anyone was competent in probability and statistics, and whether I could pay them to be a private tutor for 1 to 2 months. And I got really lucky with Tom because Tom was looking independently to learn statistics anyway.

So we just worked through the two textbooks together. 

With that context about Tom, it was somewhat more of a peer relationship than a mentor-mentee relationship. But he still knew a lot more than me. And I leaned on him more. I don't think he leaned on me at all, but I was leaning on him a lot. 

MATUSCHAK: But this is not a tutor relationship where he's presenting the material. There's a textbook. The textbook is the primary source of explication. You're both reading the textbook independently, and you're coming together. 

WALKER: Yes. So then we're coming together and we're solving basically the practice problems at the end of chapters on a whiteboard, at a university, on the weekend. 

MATUSCHAK: Okay. Have you done problem solving separately? 

WALKER: Yeah. I mean, if you were good and did your homework.

MATUSCHAK: Okay. And then you're additionally doing it together. Are you working on the same problem at the same time, or are you working on the same problem individually and then comparing? 

WALKER: A little bit of both. So sometimes we would jointly work on the same problem on the whiteboard. And you kind of alternate if you're coming up against an obstacle in the problem or something, maybe the other person takes over, takes the whiteboard marker, and they try for a few minutes. Other times we would separately work on them and then compare notes. So it was quite fluid. 

MATUSCHAK: Was there a time where you found some of the material very difficult and you were having trouble making any progress, and you really had to lean on Tom. Can you tell me about that time?

WALKER: This happened often. I guess I always had an aptitude for maths, but I haven't had to use that skill much in the last few years. I think there was just a lot of basic mathematical stuff that got exposed in the process. I mean, it was always pretty apparent that I was lacking some kind of prerequisite if the problem was just not making sense. And I think Tom and I were both good at calling that out and then going back to, I guess what you might call “prerequisite training”, where it's like: okay, I'm missing some foundational piece of knowledge here. Let's go back. Does it make sense? Oh, it still doesn't make sense. I'm missing an even more foundational base of knowledge.

MATUSCHAK: Right. And those are probably often pieces of knowledge that are not in the textbook because they're like, oh, you need to know something about geometric series that you're supposed to remember from Algebra II. 

WALKER: Yeah, exactly. 

MATUSCHAK: So then Tom is doing a mini lesson on the spot?

WALKER: Exactly. Which is one of the beauties of being able to work with someone who's so knowledgeable. Because if you're on your own, then you're kind of floundering around on the internet or using LLMs.

MATUSCHAK: Right. So, in these sessions, not only are you building some problem solving fluency by doing practice problems together, but you're also acquiring net-new information, because you're getting some mini lessons or Tom is revealing some things to you.

So how do you go about absorbing that? Are you taking notes while you're in these sessions with Tom, or are you doing a brain dump when you get home? 

WALKER: So a couple of things. I'm taking handwritten notes on a notepad. I've actually got some of my math tutoring notepads in the other room, which I can grab if we think it's worthwhile. But then when I get home or sometimes in the tutoring session—but more often when I get home—I might add some memory prompts as well.

I should actually say — this is another part of the Taleb story — I worked with a second tutor, which was my aunt, who is a really good high school maths teacher. And so I was kind of coming at it from both directions. Tom and I were doing some of the more advanced stuff, and then I was rebuilding from the foundations with my aunt.

MATUSCHAK: Oh very cool. 

For the moment, I'm going to jump ahead. 

So you did all this preparatory work that’s not about economics, that’s not about Taleb’s work. And then you had a textbook reading in economics as well. Did you have a textbook reading in economics? Is that what you said as well? 

WALKER: A textbook reading in economics? 

MATUSCHAK: Was there part of your syllabus in economics, or was it just probability and statistics? 

WALKER: Oh, just the probability and statistics. 

MATUSCHAK: Do you have a background otherwise in economics? 

WALKER: No. 

MATUSCHAK: Okay. Because there’s a lot of discussion about options trading and stuff like this. I mean it kind of lost me. I don't really know about this stuff… 

So, now let's jump forward. You're engaging with Taleb’s books and you run into this stuff about, like, futures trading and why obviously you'd want to do this in this circumstance. And he often doesn't really explain this stuff very thoroughly. So are you doing kind of just-in-time learning while you're reading this book? And what does that look like? 

WALKER:  Yeah, I do a lot of just-in-time learning. So, I think partly that looks like just googling stuff or using LLMs to, for example, quickly look up definitions or clarify some kind of concept. I also have the benefit of [having] a bunch of friends and people I could message and pester with different questions.

MATUSCHAK: How do you decide when to do that versus using LLM? 

WALKER: I think in the first instance, you should always look it up. So you’re not wasting someone's time. But if you're still not getting it or you feel like maybe the question you're asking yourself is ill-posed or not even wrong and you're not sure why, then that's a good time to message a friend. 

MATUSCHAK: So you have this fixed amount of time and these 200-ish hours, you have this kind of preparatory learning to do in probability and statistics, but also some earlier mathematical topics. And then also you want to engage with Taleb's primary works as well as probably a bunch of primary works that surround that, like perhaps criticisms or other things that are affiliated. So how do you compose this reading list? And then I'll be curious to learn about how you structure your time attacking it. You're constantly prioritizing, right? 

WALKER: Right. I don't know how I compose [the reading list]. I mean, I guess maybe this isn't a satisfying answer, but I feel like you just need to have a lot of taste about what are the most important topics, what are the topics that I need to spend the most time on in my prep because maybe they're not as important, but they're more difficult to understand or comprehend. What has he not spoken about much before so I can create some kind of counterfactual value by focusing on these particular questions? 

And then my reading list evolves constantly on a day-by-day basis, as I’m in that kind of intensive research sprint, because every day and every hour you're getting more context and that is leading you to reconfigure what you think the priorities are. 

MATUSCHAK: Yeah, you can't make a serious plan in advance. 

WALKER: No. It has to unfold. 

MATUSCHAK: So I'm very curious what it looks like to engage with these books for you. It's purposeful reading. You're generally interested in Taleb’s ideas, but also there's this very specific thing that you have to do at a very specific time. 

So you're opening the first book. You've studied probability and statistics, but you don't yet have a lot of background for Taleb. Book One - are you going in with a lot of questions? Are you trying to get a lay of the land? How do you approach this? 

WALKER: My approach to reading books could be a lot better.

MATUSCHAK: Okay, but tell me what you do. 

WALKER: Yeah, okay. So it's not super systematic. I guess there are always questions in one's head that are kind of floating around. 

MATUSCHAK: Can you think of a question you had when you were opening Book One of Taleb? 

WALKER: Yeah, okay. Because I'd read his stuff a lot over the years—so what was interesting about this prep process was that for the Incerto, his popular books, it was more me returning to those and trying to refresh my memory —And so maybe one question was: here's a category of thing that I often think. 

So I go into the intensive research sprint with these kinds of preconceptions and I'll have a note where I'm just dumping thoughts and vague ideas for questions, links to different interviews the person's given, or critiques of their work or other sources I think might be interesting. And I mean, for some guests, I've been accumulating these notes files for years. 

MATUSCHAK: Right. And probably for guests that are wishlist guests. 

WALKER: Exactly. 

MATUSCHAK: Do you have one of these notes for Taleb? 

WALKER: For Taleb?

MATUSCHAK: Yeah. If you’ve got it. If it's better to do some other guest that's fine too. I just want to orient to what this super messy “getting started” [process] is like. 

WALKER: Okay.

[Joe shares his screen and opens his Apple Notes and Bear Notes.]

MATUSCHAK: This is the third note [for Taleb] that you're opening right now. And I feel like there's something very true about this. 

WALKER: There's something very true about this. It's also a testament to the fact that he's been on the wish list since I started the podcast. Back when I was using Apple Notes. 

MATUSCHAK: You have strata… 

WALKER: Exactly, exactly. 

MATUSCHAK: [Andy skims one of the note titles.] 

Taleb questions. Yeah, this is a long note. Wow. [Andy laughs.]

WALKER:  This note is years old. 

MATUSCHAK: [Andy marvels at the screenshots.] 

Oh, screenshots. 

WALKER: [Joe scrolls through Taleb notes.] 

I mean, tell me if you want me to slow down. 

MATUSCHAK: No, this is great... Stop. I'm just going to read aloud a little bit. 

WALKER: People should know I'm probably going to be embarrassed by about 95% of the stuff in here because it's just like the ramblings of a mad man. 

MATUSCHAK: Yeah, yeah. Let me pause and express gratitude for opening up your creative process. This takes vulnerability. And I think if I can brag on you for a minute... I hope that you feel confident in showing me this “behind the scenes” vulnerable stuff because the interview with Taleb was so competent. That's the finished work. This is the messy work that went into it that we're seeing. 

[Andy reads from the note.] 

Okay. “What do you think of Eugene Fama’s arguments on bubbles?”

That seems like a very representative example of just like a very scrappy thing. I assume that you ran into Eugene Fama’s argument on bubbles at some point and then what, you just add this to this note? 

WALKER: Yep. And I would have added that six years ago. At the time, I would have thought that was a good question, but that is not the kind of question I would ask in an interview today. 

MATUSCHAK: Why is that a bad question? 

WALKER: It's too... a few reasons. Firstly, I don't think the question is well-phrased. It's a bit too... I mean, “what are Eugene Fama’s arguments?” That should be elaborated on in the preface to the question. Secondly, it's just no longer interesting to me. 

MATUSCHAK: Yeah, fair enough. Okay. So you're collecting these questions for like six years or whatever it is and you're coming to these books. Do you refresh yourself on the questions? You just kind of hope they're in the back of your mind. 

WALKER: So I will consult the questions every other day or when I get to the tail end of the intensive research sprint and I'm trying now to compose the sheet of interview questions. 

MATUSCHAK: So consulting the questions every other day? That's really interesting. So what does that look like? You're scrolling through this? 

WALKER: Yeah, I'm skimming through it and I'm not going through it in detail, but it's a more serendipitous process. So I'm just letting stuff kind of jump out. 

MATUSCHAK: Yeah, yeah. That makes a lot of sense. 

Okay. So you open book one. What does this look like? Are you reading linearly like front to back? Are you jumping around? You've read this book before, but maybe it's been a few years. What is book one, by the way? 

WALKER: So, for Taleb, book one is Fooled by Randomness. Let me step back and give some more context for the Taleb interview. So I had read his work over the course of several years. And what I wanted out of the interview was to focus on some of the stuff that I thought was more civilizationally important, but not well understood. 

There's a real, I guess, like “self-development” kind of cult that's embraced Taleb’s work, especially Antifragile, his third popular book, but I didn't really want to go for that angle. I wanted to do an interview that was a bit more technical. And so the best book for that was his Technical Incerto, which is called Statistical Consequences of Fat Tails. And he actually gave me a copy of the book [when] we caught up in April in New York prior to recording the interview in August. And I mean, one of the things that was really helpful about that meeting was he was directing me to what he thought was the most important stuff in his work. 

So firstly, the non technical chapter in that book or the chapter that I think is the best distillation of his entire body of work is chapter three or four. It's kind of a more fulsome version of a lecture he gave at Darwin College or something like that. And that chapter was the first thing I started with in that book. 

MATUSCHAK: So how did you know that that chapter was going to be the most useful thing? 

WALKER: Because Nassim Talib told me. 

MATUSCHAK: He told you. Okay, great. 

WALKER: You don't get that privilege for every interview. 

MATUSCHAK: Sure. But you have various signals. Is there another document like this document that's keeping a cue of what you should be reading next, and then it's at the top? 

WALKER: I do all of those on paper. 

MATUSCHAK: On paper. Okay, great. Love it. Love it. 

WALKER: Sorry. I'll also do them in notes. 

[Joe points to a section in Bear Notes.] 

This might be like a reading list. It's an absolute mess. 

MATUSCHAK: Sure. But this is real .

WALKER: But I do a lot of my reading lists on paper. 

[Joe hands Andy a piece of paper.]

MATUSCHAK: So you're capturing this, it's a little hard to rearrange stuff on paper. So I'm guessing these notes were like this, the numbers read in order 1, 7, 6, 2, 3, 4, 5. And that's because you are prioritizing. 

WALKER: Yeah. 

MATUSCHAK: Okay. And I notice that only one of these is checked off and that's because this is very live for you right now and you're just getting started diving into it. 

Okay. When did you write this list? 

WALKER: Sometime last week. 

MATUSCHAK: Cool. Coming back to Taleb for a minute. While you're reading that third chapter, what are you doing? Are you reading digitally? You're reading physically? 

WALKER: Physically. 

MATUSCHAK: Okay. Are you holding pens? Are you holding post-it notes? Is there a notebook next to you? 

[Joe picks up Post-its / book tags and shows them to Andy.]

MATUSCHAK: Okay. Cool!

WALKER: Book tags. Critical. 

MATUSCHAK: So how do you use them? 

WALKER: Should I grab the book? 

MATUSCHAK: Yes. 

WALKER: Okay. I'll be back. 

--------

All right. So I've just retrieved some things. 

MATUSCHAK: [Andy points to the books.] 

Yeah, I really enjoy the talismanic pile. I love that Antifragile has this toilet paper in it. 

WALKER: Uh, that is kitchen paper towels. I read this book in San Francisco in late 2016 and this will be San Franciscan tissue paper. So it's come full circle. 

MATUSCHAK: So you began with this one? 

[Andy points to the book, Statistical Consequences of Fat Tails.]

WALKER: Yeah. So these are all things I've kind of read over the years, but I've sort of forgotten things and I didn't have a very good, you know, memory practice back then. 

And so I know I'm needing to return to them. And one of the points I was going to make earlier was you go in with a lot of preconceptions. So a lot of this stuff is just going to be, not even wrong. And so part of one of the questions I'm asking when I'm coming back to these books or opening them for the first time is: What are all the ways I'm just obviously wrong about questions that I currently think are really good. And they're the kind of questions where you go into the interview and you look like a fool. 

Some of the questions I already have that I think I'm excited about—maybe these are my juicy questions—I'm looking to refute them. 

MATUSCHAK: Looking to falsify. Not to falsify what you think the answer is, but to falsify the hypothesis that that question is good. 

WALKER: Exactly. 

MATUSCHAK: And I find it really interesting that what your preparation is for here is that there's really two things that are related but distinct. You talk about looking like an idiot, probably to your audience — you wouldn't look like an idiot— but you would look like an idiot to Taleb. And that would ruin the interview. 

WALKER: Exactly. 

MATUSCHAK: He would no longer take you seriously. 

WALKER: I mean with the audience, there is also maybe a Gell-Mann Amnesia that you’re protected by. But you know, even with the audience, you don't want that.  That is the important thing, because if you lose the guest’s respect by asking stuff that's just silly or a waste of time, that means that the rest of the interview isn't going to go as well. 

MATUSCHAK: Right. And the second thing, of course, is preparing so that it's interesting for your audience and you understand well enough that you can be like a kind of a translator, you know when you need to supply this extra information, while you're also kind of playing the foil, you're sort of acting as the non-expert insert that can interpret and communicate the person's ideas. 

I guess the converse of not looking like an idiot with your questions is ideally, if you're asking really penetrating questions that they haven't asked before, then not only do you not lose the guest’s respect, but, but in fact, they kind of lean forward and they're like, okay, this is real. We're doing this. 

WALKER: Absolutely. And to add to that, they give you better answers. And maybe even some of the best answers they could give. Things that they haven't shared publicly before, things that are truly valuable to elicit. You have to morally deserve those answers by putting in the work.

MATUSCHAK: Yeah. I remember when you interviewed me, you asked me some incredibly deep-cut question about something that I'd said in some… I don't even know...It was like some tertiary material, maybe about Deutsch and the beginning of infinity and how it influenced me. 

And it had been years since I said that, but you dug it up and you asked about it. And that was the moment for me when I was like, okay, I'm switching into a higher gear. Like, this person has done their homework. I'm engaging at a higher level. 

Okay. So back to the post-it stickies. I do want to see what you've done. 

WALKER: So I also found this. 

[Joe picks up a piece of paper.]

This was a reading list for the Taleb intensive prep, but this is the second one. So I had one piece of paper and it got so messy and crowded that I started a fresh one. So this is at the very end. And actually what's interesting about this is how much is incomplete. And I never feel satisfied with how much I've been able to prepare for. 

MATUSCHAK: Right. This is something you've doomed yourself to. Because you're doing fixed-time, flexible scope [and] always things will fall off the list. 

WALKER: Yeah. It's a sad, sad, painful truth. 

Okay. [Joe shows Andy his book tags from his copy of Statistical Consequences of Fat Tails.] 

So these are the remnants of some book tags. So we're kind of like returning to the battlefield. 

MATUSCHAK: Show me how you use them. 

WALKER: Okay.  

[Joe flicks to chapter three.]

For anyone interested in Nassim’s work who wants a somewhat more technical without being overly technical distillation of just what I think is the most important part: Chapter 3: A non-technical overview: the College Darwin Lecture.

MATUSCHAK: Yeah. It says non-technical, but I'm seeing hyperboloid level plots and stuff. 

[Andy laughs.] 

I don't know…

WALKER: It's quite technical. It's the most non-technical chapter in this book. 

Okay. So, how I will read this: So the book's on the desk in front of me. I'm at the computer. Ideally also, I have a digital version of the book open in some kind of LLM or LLM wrapper simultaneously. So I've got two versions. The wrapper I use at the moment is called AI Drive. There's probably 200 of these. I mean, you could even use Claude Projects. 

Does OpenAI have an equivalent now?

MATUSCHAK: Canvas I think, but I think Canvas is more like an Artifacts analog. I’m not sure they do. I think it'll be more like you would attach the PDF and talk. But I could be wrong. It's very hard to keep up. 

WALKER: So I use AI drive. I settled on this for three reasons. Firstly, as opposed to say Claude Projects where you're uploading text files, here I'm uploading PDFs. A lot of the files I deal with are PDFs in the first instance. So it's just a more frictionless process. I'm uploading a paper or a version of the book or a PDF I've downloaded from libgen or whatever. 

MATUSCHAK: You can't upload PDFs into Claude Projects? 

WALKER: I thought it was only text files. 

MATUSCHAK: Yeah. I think you can upload PDFs. 

WALKER: Okay. The second reason I like AI Drive is that I can organize things with folders. And the third reason is I can select any model that I want to use to interrogate the file. And that's important because when I was uploading books. A lot of those files are too large for either Claude or ChatGPT. And so I'm using Gemini Pro. And this gives me all of those options. 

So you're not going to, unfortunately, be able to see my chat history because they only introduced that feature a few weeks ago. So it may not exist for Taleb. 

So here I've uploaded all of the PDFs. 

[Joe refers to AI Drive.]

MATUSCHAK: Right. So you have the physical book here, you have PDFs here. And so you're encountering some technical explanation that you don't understand. And so you ask a question.

WALKER: Exactly. 

MATUSCHAK: How do you localize the question, by the way? I mean, you presumably can't say like on page “n”? Probably the PDF page doesn't correspond. 

WALKER: I mean, it depends on the PDF, whether it's a photocopy or not. Actually, you know what? I remember when I was doing this, I think I checked to see if they correspond and they do. So I might say something like: “On page 22, where he mentions that in Mediocristan the probability of sampling higher than X twice in a row is greater than sampling higher than 2X once. I don't understand what this means. Can you…” And then I'd put in some kind of prompt. 

MATUSCHAK: Yeah, right. So you get an explication. That's really interesting. 

WALKER: So I've got a kind of study buddy with me as I read. 

MATUSCHAK: So I noticed this requires sophisticated reading on your part. This requires reading comprehension practices. You are monitoring your understanding. Is that just natural for you? Is there a systematic way that you do it over kinds of questions that you ask yourself as you read? 

WALKER: It's pretty natural and pretty chaotic. 

MATUSCHAK: So, chaotic in the sense that probably you do that sometimes, but not all the time. And sometimes you don't understand something and you just let it go…

WALKER: Right. And it's like: okay, I'm not going to be able to understand everything in the constraints I have. Does this seem important? Not really. I'll let this slip. 

MATUSCHAK: Right. So sometimes the way that some people will achieve this is by writing like a precis or something of each section. In order to check their comprehension, you're kind of doing it in your head. 

WALKER: Yeah. So I will write some kind of note at the end of [a chapter], so usually when I'm working with a book like this, the unit or increment I'm dealing with is a chapter. So as I'm reading, I'm asking for help with things or looking things up I don't understand. Mostly using these post-its and these are for things that I want to turn into memory prompts at the end of the chapter. I don't want to do it on the spot because I don't want to interrupt my reading. 

MATUSCHAK: It's distracting. 

WALKER: Yes. It becomes too much of a chore. 

MATUSCHAK: Do the colors mean anything? 

WALKER: No.

MATUSCHAK: So it's only for markers for needing a memory prompt. Do you have some way to mark: I don't understand this, I need to follow up? 

WALKER: Do I do that? So my habit is, if I think it's important, I'll look it up on the spot. Because I'm always afraid that if I don't, it's going to affect how well I understand the subsequent material. 

MATUSCHAK: Cool. Are you writing questions into notes while you're reading [such as] interview questions or questions for yourself, questions that need further investigation? And so you're typing into your notes file while you're reading? 

WALKER: Yeah. I'm jotting things that jut out as particularly important. I'm kind of jotting them down. 

MATUSCHAK: And does that go into the Nassim Taleb research mega note? 

WALKER: No, that goes into Obsidian. 

[Joe opens his Obsidian Vault and shows it to Andy.]

MATUSCHAK: [Andy laughs.] 

That's [note-taking] program number three. Great. Love it. This is very true. 

Okay. We have “per book” notes. So this is kind of a commentary - I see excerpts.. Everything that I'm seeing on the screen right now is an excerpt. 

WALKER: Yeah. So this is probably a bad example… 

MATUSCHAK: Scroll up.

[Andy reads Joe’s notes in Obsidian on Chapter 16 of Taleb’s Statistical Consequences of Fat Tails.]  

For this chapter, your note for 16: “for this chapter, it's fine to focus on the actual, not rescaled estimates of death as…”

WALKER: That's kind of a note to myself. 

MATUSCHAK: Okay. Those are your words. Interesting.

[Andy continues to read from the same note.] 

“I'm confused as to which is the correct inter-arrival time for wars generating deaths of more than 10 million?”. So this question is buried in this massive note and a pervasive problem I think that many of us have who try to do this kind of process is like: then what? You know, you have this many thousand word note, which is merely one of several dozen And there's a question buried in the middle of all the excerpts. What happens to that question? 

WALKER: So as I'm reading the chapter, I'm jotting down things like that. This question, obviously, wasn't properly resolved or maybe… maybe was it? It kind of was... 

MATUSCHAK: Is that an active decision not to resolve it? Or is it more like it just kind of falls through the cracks?

WALKER: I think for this one, it's a little bit of both. I kind of got an answer that I was happy with, but I was only like 85% sure. So I'm happy leaving this in because I might come back to it and it'll force me to get to 100%. 

MATUSCHAK: Okay. And your pages look pretty clean here. So I take it you're not writing in the book with a pen. 

WALKER: No, I usually don't.

MATUSCHAK: So what's the function of these excerpts? You have a lot of things like pull quotes.

WALKER: So these are a bit lazy. I think the context here is I'm under time constraints. Normally, I would try to synthesize and express things in terms of my understanding, which some of the other notes will demonstrate. But these excerpts are things that I think are very neat distillations of an important idea. 

MATUSCHAK: Cool. So it's almost like a resource for you. You can easily access these distillations now that they're in this mega note. 

WALKER: There's another way I use these. So I finish a chapter and it's kind of peppered with these book tags. 

MATUSCHAK: Now you're writing memory prompts. 

WALKER: Yeah. And the reason I like these [the book tags] is firstly, as we discussed, I don't interrupt my flow as I'm reading. But also things that at the beginning of a chapter, things that seem like important memory prompts are superseded by maybe even more atomic ideas further into the chapter. 

MATUSCHAK: It's easy to remove. 

WALKER: Exactly. Or maybe they're expressed in a later part of the chapter in a much clearer way. And so I can just kind of… 

[Joe gestures at removing a tag.]

When I go back through, maybe only 70% of the tags will actually become prompts.

 So I've got to the end of the chapter. And firstly, I'll put in a lot of the sections [into Obsidian] that I think are important or that I want to turn into prompts. 

MATUSCHAK: You'll begin by extracting excerpts that correspond to where you put stickies next to it. Okay, before you write the prompts, you'll just extract them verbatim.

WALKER: And then also just for posterity, I might extract other excerpts that I think are generally important but that I don't intend to turn into prompts. 

MATUSCHAK: Are you doing that while you read or how do you know which ones to extract? 

WALKER: I guess I just use the tags. 

MATUSCHAK: So you're also marking things that you want to extract but which don't necessarily want to become prompts. 

WALKER: Yes. And I can remember which are which because I've just read the chapter. 

MATUSCHAK: This chapter, do you read it in one sitting? 

WALKER: No, this one might have taken a couple of sittings. I think it's about 40 plus pages. 

MATUSCHAK: Right. I'm just kind of picturing you engaging with this over the course of maybe two days or something. 

[Joe shifts to his Obsidian Vault.]

WALKER: So at this point,  I open two windows of the same note and at the bottom of the note, I have my Mochi prompts. And so on the left, I'm looking at the content that corresponds to the prompt I want to write. And then I'm writing it here. 

MATUSCHAK: Ah! Right. So you're kind of just moving through it. 

WALKER: And then I do that chapter by chapter. 

MATUSCHAK: And you put the Mochi prompts at the bottom of the note as opposed to interleaved because…? 

WALKER: Because, if you look how many…

[Joe scrolls through many Mochi prompts.]

MATUSCHAK: Oh yeah. This is great. Look at you. 

WALKER: A lot of this is all Andy's doing. You did this to me. 

So I have a lot of Mochi prompts and I think it's just kind of unwieldy if I intersperse them through the main notes.. 

MATUSCHAK: Honestly, it's bad both ways. You know, the challenge with this way is,you have the original excerpt plus possible commentary up there and you have the Mochi prompt down here and sometimes you want to see them together and it's hard to reconcile but then as you say, if you intersperse them, it's dissatisfying for the other reason. 

WALKER: Yeah. And then it's like, do you want maybe two columns with the notes on one and the prompts on the other?.

MATUSCHAK: It's like a translation. 

WALKER: But that doesn't always work because sometimes the prompts connect to multiple pieces of …

MATUSCHAK: They're synthetic… 

WALKER: Yeah. They're synthetic.

MATUSCHAK: Yeah. I feel you. 

What is this? So you're using some kind of Obsidian plugin that extracts these Mochi prompts—

WALKER: Which you recommended to me!

MATUSCHAK: So I recommended it to you without having used it just because I saw that it kind of works—

WALKER: Well, I should say it's crazy how this plugin made the whole experience disproportionately enjoyable. 

MATUSCHAK: Yes. So let me just narrate for your viewers because it's very non-obvious. A thesis that I staked in the ground like five years ago was that I think it's just really a bad idea to be jumping back and forth between the place where you're writing long form prose and about a thing you're studying and the place where you're writing memory prompts about the thing you're studying. 

It just adds a lot of friction and dissociation. And so I made a system that lets you interleave those things. Other people have also made such systems. You're using one that I've not used, which I think I suggested to you. 

WALKER: It's great, by the way. 

MATUSCHAK: That's great to hear. And so this format… those are some kind of markdown heading— 

WALKER: Yep. So you can set it however you want. I've chosen H6 [“######”]. Just to visually differentiate it from the main headings in my text. 

MATUSCHAK: Got it. And then the following block becomes the answer. And are there multiple blocks? How does this work? 

[Andy points to the screen.]

WALKER: Okay. So the plugin doesn't work for images. So if an image is what I do, I separately place the image in Obsidian and then I have to go and do it in Mochi. 

MATUSCHAK: So I think one of the questions I wanted to ask you is that I think it's pretty non-obvious why memory prompts should be a big part of your process. Like you're sort of cramming. 

You're doing this two weeks before the big thing. And memory prompts, they're big advantages. Naively one might see them come over months and years. But they seem to really help you with your process. And I want to understand, that I think it's really interesting and non-obvious, how even just in the span of like 10 days or something the memory prompts are really helping you. So tell me about how. 

WALKER: Yeah. I can answer that question in two ways. The first is I want to be in a place where my knowledge is compounding and prompts that I've created for interviews years ago, I'm now leveraging for future interviews. 

MATUSCHAK: So it makes your study more rewarding because it doesn't feel like a one-off thing. It feels like something where you're really growing yourself in a durable way that's compounding over time. 

WALKER: Absolutely. 

MATUSCHAK: So even if it didn't help with just that one interview. It would still be worth doing because it would make you more enthusiastic about studying and help in the future. 

WALKER: Yep. And as I'm writing them, I'm writing them with an eye to the future. So when I was doing the Boyd & Richerson interview. I know a lot of that whole deck, it's like four to five hundred prompts, I'm going to be using for a future interview I do with someone in their field. 

But I think in these kinds of intensive sprints the prompts help in two ways. And this is the second way to answer your question. And the two ways they help with the intensive sprints are firstly, because I'm retaining more information, it makes the next day of the sprint even easier. So I'm compounding the information. 

MATUSCHAK: It's really interesting and non-obvious that you'd notice that day-over-day. And the thing is, I wasn't convinced that that would be true. And I noticed it actually depends a lot on what I'm reading whether it is true. When I'm in a situation like the one I think you're in where it's just it's difficult materials new to me. It's outside of my field. That's what tends to make it most likely to be true. And the thing that makes it most obvious to me that it's true is I like to—when I'm doing a session like this—write the prompts and then at the end of the day do the prompts like the same day at the end of the session even. And it's kind of humbling how often I will not be able to summon the answer to a prompt that I just wrote like an hour or two ago. 

WALKER: I have that feeling sometimes, often I should say. 

MATUSCHAK: When that happens it increases my confidence that this practice is helpful. It's very humbling. It is hard to write good and effective memory prompts. 

WALKER: But actually, I'm not sure if we were coming back to this, but I should say the second reason I find the prompts super helpful is when you're doing — and you would be able to empathize with this—but when you're doing solo research you don't get a lot of feedback, and for me the final output is the feedback [from] firstly the guest themselves in the interview and how they respond and then ultimately the audience after the interview is published. 

But it can be quite a tedious, lonely, difficult process doing this like one to three weeks of long days of research by yourself. And having a little ritual every morning where I'm reviewing the prompts from the previous day and you know getting 90% plus of them correct or whatever is incredibly rewarding. So it adds some kind of like shorter feedback cycle to my prep process so it's been incredibly motivating. 

MATUSCHAK: Right you can also see you're producing something. It's like a pile that's growing. There's an output. 

WALKER: Yeah. I'm making progress. 

MATUSCHAK: Right. Whereas, this book retains its shape as you work your way through. I guess it gets these little stickies on the side. 

Yeah I really resonate with that. 

I want to ask you about prompt writing. This has been an interesting question in my research and it's just a thing that a lot of people deal with. So it's hard to write prompts about this general material as opposed to memorizing vocabulary words or whatever. So tell me about your challenges with that or if you even have them. 

WALKER: Yeah. So I mean firstly I should say everything I've learned about prompt writing I've learned from you. Your writings online. So I think for me personally the way I would articulate the most important overarching meta skill is to have empathy for your future self and to know what kind of prompt is going to be trivially easy to guess, what kind of prompt is going to be too ambiguous or confuse you. And what kind of prompts are going to be most durably useful. 

MATUSCHAK: And sometimes it's probably hard to write a prompt for a particular thing.

WALKER: Yeah. So I think a few things that help are… Firstly—and again I'm articulating a lot of these principles for the first time here—but one rule of thumb I have is I won't write a prompt on something that I don't understand. 

MATUSCHAK: Yeah. Actually, I learned a lot from Piotr Wozniak's 20 rules of effective space repetition flashcard writing and his number one suggestion is: understand before you memorize. 

WALKER: Right. Having said that maybe there's one little exception to that though which is for definitional things. I might write prompts even if I don't fully understand all of the contours of the underlying concept. And it's still helpful because even if I'm slowly, stochastically learning this concept, I'm still picking up the language and it's making the eventual understanding easier. 

MATUSCHAK: It also creates some feedback, right? If you have this review come up and you can parrot the answer, but you can feel that you're parroting and that you don't know what these particular words in the middle mean. It's like it's an extra signal for you. It's like, okay, I better track down those words. 

WALKER: I mean, should we review some of the prompts I've been doing recently for Larry Summers? 

MATUSCHAK: Yeah, let's just flip through a few. 

[Andy reads from Joe’s list of Mochi prompts in Obsidium.] 

“What are the two basic stories of catch-up growth?” When this prompt is imported into Mochi with the kind of plug-in that you're using from Obsidian, can you name that plug-in? Probably people are going to ask. 

What is it called? 

WALKER: Mochi Cards Pro. 

MATUSCHAK: Great. Love it. 

When this is imported into Mochi, do you get some kind of context about: what are the two basic stories of catch-up growth? My concern as a prompt-writing coach might be like, whose two basic stories? There's probably lots of basic stories of catch-up growth. Do you get context like this is what Halperin said? 

WALKER: I do often use that as a caveat. And the reason that caveats like that are important is you don't want to assume that this is correct or consensus. So it's important to associate certain ideas with certain people. I think here I just felt sufficiently confident that this is a kind of basis. 

MATUSCHAK: That it is universal. 

So is your answer that the plugin in fact does not supply any kind of built-in…

WALKER: Oh, I see what you're asking. No. Should I quickly demonstrate how the plugin works or how I use it? 

MATUSCHAK: Well, let's just go look at Mochi. What does it look like to review these in Mochi? 

[Joe opens up Mochi.]

WALKER: So firstly in Mochi, I'll create a deck which becomes a subfolder in Larry Summers. There's a ‘Basil chats’ folder. Basil who I've been speaking with about this. And then I write a new prompt. And then I will push it through to Mochi.

This is so nice.

MATUSCHAK: Yeah, I see. But you do get context. So open that up again. 

[Andy refers to the recently added Mochi prompt.]

So I see at the bottom here “Larry Summers”, “Basil chats”. Do you see that when you review? 

WALKER: Yeah. But it doesn't help me. Because I know I'm reviewing for Larry Summers. 

MATUSCHAK: Is that because when you sit down to review, you're choosing what you review. You're not doing interleave review. 

WALKER: Yeah. So each morning I'm reviewing... Say I'm in the intensive sprint for Larry as I am at the moment. Each morning I'm really just reviewing Larry. 

MATUSCHAK: Yeah, right. You're not reviewing your Taleb prompts? 

WALKER: No. 

MATUSCHAK: Right. 

WALKER: But I mean, it would be like that where I'm preparing for several interviews concurrently. 

MATUSCHAK: Like, you're preparing right now for Summers simultaneously with Fukuyama. 

WALKER: So some mornings last week when I sat down, I'm getting a mixture of prompts for both interviews. 

MATUSCHAK: Okay. But although you write the prompts because you want the long-term memory benefit, you aren't necessarily always doing the long-term review of all of the prompts. 

WALKER: No. 

MATUSCHAK: Right. 

So no judgment, I promise. I notice you have 1,200 cards due today. 

[Andy laughs.]

 Can you tell us a little bit about... how do you relate to that emotionally? 

WALKER: It's meaningless to me. 

MATUSCHAK: Meaningless to you. Okay, cool. 

WALKER: Because I'm using Mochi in a very specific way and also I haven't —this is bad—but I haven't taken the time to kind of go through and learn how to change these settings and optimize. I probably should get your advice at some point and how to use it better. 

MATUSCHAK: Okay, but it doesn't bother you. 

WALKER: No, I just ignore it. 

MATUSCHAK: This bothers a lot of people. It's a big problem for a lot of people. People feel overwhelmed. I mean, how are you going to review it with 1,200 cards? And you know, the answer is 50 at a time for a few weeks. 

Okay, so I am curious how you reconcile this desire to have the long-term accumulation of knowledge on the one hand with on the other hand the practice of not actually doing the reviews for older cards or when you're between projects, do you do them? 

WALKER: Yes. 

MATUSCHAK: Okay. So you switch into a different mode. 

WALKER: Exactly. 

MATUSCHAK: But you've been in intensive mode for like a couple of months straight now. 

WALKER: Yeah, exactly. It's been just like back-to-back sprints because of this trip. So when I get back to Sydney, it'll be a morning practice where I'm getting a kind of mixture of different prompts. 

MATUSCHAK: Cool. 

What rate do you find yourself… like a card comes up and you're like, this is just a bad card and you suspect you could get rid of it? 

WALKER: Interesting question. Pretty rarely. The more common thing that happens is: so maybe like five to ten percent of cards after I’ve first written them, I'll be like, I need to revise that, [the] phrasing is convoluted or unclear and maybe one percent I'm like, this just isn't necessary or helpful altogether. Delete.

MATUSCHAK: And when you have that observation about revising, do you do it on the spot?

WALKER: Yeah. I have to. I have a zero inbox policy with that because otherwise you forget.

So one of the benefits of this plugin is in theory, if I edit the original prompt in here—and by the way, the reason, which I think I may have got from you originally,the reason I like having the prompts in here [in Obsidian] is that they're embedded with the kind of original knowledge. So the underlying knowledge changes, then I can edit the prompts as well. 

And the way this plugin works is if I edit one of the prompts and push it through again, it should change that prompt rather than adding a new one. It's a little bit finicky. 

MATUSCHAK: Really? How? 

WALKER: Well, we can test. 

MATUSCHAK: I'm not sure I believe you, but it doesn't seem like the information is there that would allow that to be possible. I would expect it to be inconsistent, I guess. 

WALKER: Yeah. Sometimes it works. Sometimes it doesn’t. 

So this is ‘Basil Chat’. So let's put in a test here and see if it works… No. It’s created a new set.

MATUSCHAK: Yeah. It didn't remove the old one, interestingly. So it added a new card for the modification that you made. It didn't duplicate all the others. Actually, no, it did. Sorry. You’re going to have to clean this up now. That's unfortunate. That's a feature that I really, really value. 

WALKER: Me too. And it is purportedly a feature of this, but sometimes it works and sometimes it doesn't. 

MATUSCHAK: It needs to do something else. 

WALKER: I had this same conversation with Jason Benn a few weeks ago and I was like, oh, it never works. It's really, it's really finicky. And he was like, “okay, show me”.,. And it worked perfectly. Like what we did just then, worked. 

MATUSCHAK: Yeah. I mean, I can see how they could have some heuristics that would make it work sometimes. But you know, having designed one of these plugins, there's limits to that. It really needs to add some kind of identifier somehow to the question that allows it to track the identity of that question over time. It shouldn't just duplicate everything though. That's silly and avoidable and I'm willing to believe it's just a bug. I really value—my system doesn't have a manual push through operation. It just goes in the background. And so I really like being able to just jump into an old note and delete three prompts and add two new ones or whatever. Just have it show up. But yeah, it's difficult to arrange. 

Okay. So you have these prompts. You're reading, what is that notebook for? The Rodeo notebook? 

[Andy points to a physical notebook on Joe’s desk.]

WALKER: This might have been some Taleb math stuff. 

MATUSCHAK: And you're doing math in the notebook rather than on the computer because you want to be able to draw. 

[Joe nods.]

You don't use an iPad?

WALKER: No. I probably should. I've just never been introduced to it. 

MATUSCHAK: Now kind of moving through this process. You've done some of this initial study with the tutor. You're reading the books, the book list is kind of ever growing as we've seen. It does not actually get fully consumed. At some point here, you are starting to accumulate like real questions for the guest. I've seen references to a Trello board. 

WALKER: Kanbun.

MATUSCHAK: Can you share some of that? How do the questions start getting accumulated and structured? 

WALKER: So this is also a recommendation from you. Which again, it's been fantastic. So these are the two plugins I use, both recommended by Andy are the Mochi plugin and the Obsidian Kanban plugin. 

So I think the software is a constraint here, but ideally I would have questions embedded in [my notes]. [For example,] I've got a note on this paper that I'm intending to [read], maybe it's giving me some context for the Larry interview. And I can add a comment or something that connects to a question and then aggregate all the questions. 

MATUSCHAK: Your Logseq and Roam and Tana users are foaming at the mouth right now to tell you about the many ways that this is possible. If only you were to see the light and use yet a fourth note taking software. But we're not doing that. 

WALKER: So, what I'll do is as a question occurs to me, I'll come into this Kanban board. 

[Joe shows the plugin in Obsidian.]

The plugin is ‘Kanban’. 

MATUSCHAK: All right. So markdown-backed too! Interesting. Yeah. I'm curious about that format. 

WALKER: Oh, yes. So, why that's useful is I'll get the markdown version and then I will copy and paste into Google docs at the very end of the process from markdown. Is that what that means? 

MATUSCHAK: What it means is that—the naive way to do a Kanban board plugin would be that when you make a Kanban board, it's just some proprietary format. It's like a JSON file or something. And you wouldn't be able to just copy it anywhere. I mean, it would be something that only that plug in knows how to read. But Obsidian’s ethos is, these plain-text, human-readable, interchangeable files on disk - that they're just plain text files. And so it is lovely that this Kanban plugin is following that ethos. 

So you have this Kanban board. You're throwing, clearly, tons and tons of stuff into this Kanban board more than you can possibly ask. How, how do you use it? 

WALKER: So I have columns based on the different topics I want to ask about. And then I'll have a “To Categorize” column. 

MATUSCHAK: [Andy reads from the Kanban board.]

Where do these columns come from? How do you know that you want to ask about “Precautionary Principle & AI”? 

WALKER: Leading into the interview, during the prep process, I'm forming ideas about one of the most important kinds of things to ask the person and organizing those into themes and topics. 

MATUSCHAK: So a different process that it sounds like you don't use would be: you come up with lots of questions that seem interesting and then looking at the questions you say, well, these questions all seem to have the same theme. I'm going to label that theme. 

WALKER: Yeah. I do that too. That's why this column is crucial: “To Categorize”. So most things I just dump in there. 

MATUSCHAK: It's interesting to contrast the ‘in process’ one to the finished one.

WALKER: So for Larry Summers, here, that “To Categorize” column is biggest. 

MATUSCHAK: And you have this challenge now where it doesn't fit on the screen. It's actually several screen heights. 

WALKER: I can actually change the width of these columns. The reason I haven't is because I want to see my categories. 

MATUSCHAK: It's just a limitation of the digital canvas.

WALKER: Yeah. It's not ideal. 

MATUSCHAK: Okay. So this right now is mostly a dumping ground.

WALKER: And these won't be perfectly formed questions. Sometimes there'll be like two or three versions of the same question. And I've forgotten that I'd put it in, and I've put it in in slightly different words or I ultimately realized the questions can be merged or collapsed. So at this point it's very messy. 

MATUSCHAK: Got it. You start organizing it more aggressively while you're still in the studying process probably?

WALKER: Yeah. But most aggressively, like, 80% of the sort of work on this Kanban board would happen a day or two before the interview. 

MATUSCHAK: That makes sense. You kind of want to save up until you've kind of accumulated all the material you're going to have and then you're prioritizing and arranging. 

WALKER: The thing I really like about doing this in a Kanban board, is for an interview  the sequencing of the questions is really important and the kind of narrative arc or, I don't know, maybe you need to establish a certain premise before you can ask another question so that the audience can follow along. 

And the Kanban board enables me to reorganize questions very easily. And then I can change it to Markdown and then I'll copy and paste it into Google Docs. At that point, when it's in Google Docs, I may share it with people who I'm sharing the questions with, and to be like, do you think these are good questions? Am I missing anything? 

But also in Google Docs, I'll tidy up the formatting and then print it out and take in physical questions for the interview. So I could actually, if you're interested, I could show you what the formatting looks like for a final interview. 

MATUSCHAK: I want to stick for the moment to the studying and planning phase. I noticed that there's this prompt near the top that says: “TO PRACTICE: loading more context into the preface of a question to get a better answer.” And so this is a note to yourself?

WALKER: This is a note to myself. 

MATUSCHAK: And you're keeping it in the ”To Categorize” [column], so it's always visible and near the top. You have it highlighted. So it's really standing out for you. These are sort of pole heads and they're standing out. 

WALKER: Yeah. So there's two meta things and that's just because they're relevant to the second meta thing (which is in bold) about how many questions I think I'll be able to ask in this interview. 

MATUSCHAK: This is only a 50 minute interview. It's tight. 

WALKER: Much tighter than I'm used to. And so these things obviously very strongly pertain to the kinds of questions I'll be crafting, how many and their sequence. And so I want to be reminded of them as I'm going into the place where the questions are. And the only way I can do that is making them cards on the Kanban board themselves. 

MATUSCHAK: What's interesting about this to me is that you're not only learning about the domain, you're also learning about the art of interviewing, and you're weaving those two learning processes together. 

WALKER: I have decks in Mochi for—

MATUSCHAK: Oh, you were telling me about this. Show me Mochi decks for the art of interviewing, your videography ones of course. Maybe we can see a little bit of interview technique. You can show the secrets. 

[Andy looks at a prompt on screen.] 

“Don't talk unless you can ____”. What are the three words? 

WALKER: “Improve upon silence”. 

MATUSCHAK: Ohoo, “Improve the silence”. Yeah, that's great! 

[Andy looks at another prompt on screen.] 

“What is a very easy way to transform potentially any question into a more interesting one?” 

WALKER: “Flip the sign”. So instead of asking what would you do in X situation? What wouldn't you do in X situation? I need a better example there—

MATUSCHAK: So this is a challenging kind of prompt to write because what is a very easy way? You have an answer in mind, but there are many very easy ways, potentially. So as you get better and better at interviewing, you will probably learn more easy ways to transform any question into a more interesting one. 

WALKER: Absolutely. This is a very flawed prompt. 

MATUSCHAK: Yeah. It's very hard to write a prompt like this. 

WALKER: Exactly. And as I was writing it, I was conscious of all the prompt writing best practices that I was violating, but I was also cognizant of the fact that at least for the next few months, I'm going to know what the answer is and it's going to serve me. 

MATUSCHAK: I was just thinking, as a mild prompt writing coach, what would I do with this? 

WALKER: What would you do? 

MATUSCHAK: Really, really difficult. So, one thing that I might do is—so you want the prompt to have an unambiguous right answer, or for it to be a generative thing. 

So a different way to phrase this prompt is “name three easy ways to transform any question into a more interesting one”. And it could be a different answer every time. Or often when that answer comes up, flip the sign. Or it's an epiphany that occurs at a particular place and time in a particular context in response to something. And so, if I'm having some very general epiphany, but it was at a dinner with my friend Laura and we were talking about X and her suggestion about how to handle X was this specific piece of advice.

WALKER: I have some like that. 

[Joe reads from the card.]

So “When running a loop around the Berkeley campus…”—

MATUSCHAK: Exactly. That's exactly how I would handle it. It's a cue, but it's not a cue that gives away what the answer is. The other thing I would do is practice applying it, that is transforming questions in that way. And just thinking to yourself about how it makes the question feel. 

WALKER: Okay. Just so I'm clear, practice writing better prompts? 

MATUSCHAK: No. So given the negation insight, a prompt, which says: “get a more interesting question by a negating tactic”. So, flipping the sign in order to improve. Or you could list a few to apply it to. 

WALKER: Oh, that's a nice one. I like that. I like that as general advice, that practice. The reason it might not be so applicable here is because it's trivially easy to do because you're literally just flipping the sign. But it does make it more salient. 

MATUSCHAK: That's right. And salience is something that we don't understand well enough and it's something I'd really like to understand better. So it has to not just be the task, but it also has to be a reflection. So, what is the effect of flipping the sign on this question? Evaluate it - Do you like it better? Is it worse? How does it feel? And then you're kind of engaging with it. 

WALKER: And maybe do it for a question where it really does transform it into a more interesting question and one where it doesn't so much. 

MATUSCHAK: Yeah. A more challenging version of that would be: “come up with a question which benefits enormously from flipping the sign to make it more interesting. And you must provide one that you haven't provided before”. 

WALKER: Yeah, that's a great one. That's a nice one. I should write that down. 

[Joe types a new Mochi prompt.] 

“Come up with a question that benefits enormously from flipping the sign. Make it one you haven't created before.”

MATUSCHAK: And this, by the way, is a general technique. So we listed a few things. There's recall, like what is the technique—the insight I had at that particular moment. There's application—practice, applying the technique. And then there's, if it's a technique that wants to get applied in a particular situation or it doesn't always work, you want queuing—you want to practice about the queuing of the technique.  You can create a question like this that interrogates or supports the trigger or queue for the salience. This pattern works generally.

[Andy refers back to the Mochi Board.]

We were looking at this because we were talking about the meta layer that you're learning about interviewing. I can see all the super interesting questions in here. I'd love to spend a lot of time with [this] - honestly, I'd love to benefit from your learnings here.

And you're doing that simultaneously with learning about the material. At some point the time runs out and you're interviewing the guest. One question I have is, has there been a moment when you were in an interview and you felt some kind of a failure? Like you didn't study adequately the thing that you needed to study.

WALKER: So I can think of two examples, but they're somewhat different. With the interview I did with Katalin Karikó, that's when I was still working full time alongside the podcast. And I did that on a very short trip with three other interviews, including the Danny Kahneman one. I mean, another thing I've been learning is how to, how to travel effectively — have no jet lag, scheduling all of the logistics that go behind it. And that one I felt like I was really disappointed because I felt like it was an incredibly important interview and I didn't feel like I'd done enough prep for that one. 

MATUSCHAK: Interesting. I love that interview. Tell me about what disappointed you. 

WALKER: I just didn't feel like I understood her field in any deep sense. I mean, I understood things at a high-level. Like I could talk about how DNA goes to RNA, goes to proteins. And that process is kind of uni-directional. And various sorts of things. I didn't feel like I put in the hours that would have made me feel better. 

MATUSCHAK: Okay. So a naive person could ask here: well, Joe, Andy liked this interview. So you're doing this incredibly intensive study in prep. Can your viewers tell when you fail? Can your viewers tell? 

WALKER: I don't know. I mean, I think some viewers can. Particularly discerning viewers probably could. I guess there's a question about what failure means here. 

MATUSCHAK: So the bar here is kind of interesting because when you did that Karikó interview, it was so significant because she'd not done other interviews. And so it was the first longform interview that anyone got to hear from her about it. And that alone has value kind of irrespective of anything else. 

I guess one thing I'm wrestling with here is: Joe, of course you don't have expertise in her field. Like even if you did have the full 200 hours to devote to it, which sounds extraordinary. It sounds like a huge amount of work to prepare for a podcast guest. It is obviously inadequate for understanding a deep niche expertise like synthetic biology. People go to PhD programs for that. So how is it possible to understand the field well enough? Even with the extraordinary 200 hours? How is it ever possible?

WALKER: Right. Well, I mean, I think that's part of the project of the podcast to work out the extent to which those kinds of endeavors are possible. 

MATUSCHAK: There's some kind of “satisficing” thing where it's like you get far enough. I can see this in your interview with Taleb: I can see a difference in your preparation. And maybe it's because you've been thinking about those questions for so many years, where it's almost like you get to pose as not a peer, but like a really interested, rising senior undergrad or something who's like talking to the professor and can kind of hold up the conversation. As opposed to somebody who showed up for the podcast, “I have some questions for you”. And the way that manifests for me as a listener is in the unprepared stuff, where like he says something that you don't expect and you come back immediately with an improvised follow-up or reply that requires that you understand his response thoroughly as it relates to the domain. And that you can generate a thoughtful or curious follow-up, it does indicate a different degree of expertise. And so how did it manifest when you were talking with Karikó, the “not-preparation”? 

WALKER: So I knew ahead of time that I hadn't done enough prep. So the strategy I evolved going into that interview—maybe a day before or whatever to deal with that— was to focus more on her story. And it was a justifiable strategy on the grounds —

MATUSCHAK: It's super interesting! 

WALKER: Super interesting. And it was the first long form podcast interview. So I could comfortably say that that was going to be the focus for those reasons, not because I hadn't done the prep. I just remember feeling a deep sense of guilt and shame that I hadn't been able to, with this precious opportunity to put in as much work as I had wanted to. 

There was a moment in the Stephen Wolfram interview where towards the end, I asked him about physics and I asked him a question that I didn't really understand myself, but it had kind of been something I thought of and someone else thought it was a good idea. And I hadn't even, I mean, there was no work behind the question. And it was probably one of the few questions where his reaction was genuinely disdain. 

[Andy laughs.] 

And I remember feeling a sense of, like ick with myself. And I think—one of the rules I have is—if you don't understand the question, don't ask it. But that was one little moment in an otherwise very good interview. 

MATUSCHAK: Good thing it was near the end. 

WALKER: Yeah. And I hadn't done the work to even have the right to ask that question. I was kind of just parroting the words. 

It didn't feel right. But I mean, you can still get interesting answers that other people can appraise, even if you're not really appraising it in real time. But I think as a general heuristic, I have a rule of not asking anything that I'm not genuinely interested to hear the answer for. 

MATUSCHAK: And able to comprehend the answer. 

WALKER: I mean, those two are very highly correlated. 

MATUSCHAK: If you can't comprehend the answer, then you can't be in real conversation with it. All you can do is nod. Thanks for that answer. Hopefully some of them will benefit and then ask your next question. You can't act to it. You can't follow ups. You can't dig in at all. 

WALKER: Yeah. So you have to be able to comprehend the answer. Otherwise it's fundamentally no longer a conversation.

MATUSCHAK: Right - You can't dig in. You can't react. That makes sense. 

So I notice that in your note system and in your Mochi, everything is sorted into folders by the interview guest. You also have this aspiration that you've expressed to build your understanding of these various topics over time and have these conversations, I imagine, to also build on each other so that what you're learning from one leads to another. I'm curious, have there already been instances of breaking down the lines between these folders, the strict hierarchical structure where these guests are kind of in conversation with each other?

WALKER: Kinda… not yet. I mean, this system is reasonably new. 

MATUSCHAK: Well, I don't necessarily mean it has to be in the system, but in your process of doing the podcast. 

WALKER: So this folder is actually called Cultural evolution. And I already know that I'm going to be building on this—so I used this for Boyd and Richerson. I know I'm going to be using it for Cecilia Hayes, Cultural Gadgets.

MATUSCHAK: Isn't it relevant for Fukuyama as well?

WALKER: Absolutely. 

MATUSCHAK: Well, now you'll have to evolve your structure somehow to support these kinds of cross cutting and learnings. 

Do you have a sense of how you want to do that? 

WALKER: No, I don't. I was expecting to cross that bridge as I came to it. I'll be thinking about that more and more. 

Do you have any kind of tips or suggestions in mind? 

MATUSCHAK: I think it's an open research question. Obviously I've written a lot about [it], and one way to do that is to break everything down into like really small pieces and that way they can easily be reused in multiple places and discover the structure that wants to exist before you know what it is. 

WALKER: The evergreen notes are atomic. 

MATUSCHAK: Sure. Or Zettelkasten’s stuff. There's lots of downsides to that. There's lots of ways in which that would work poorly with your process. I think it's not at all clear. That's why I asked 

[Andy laughs.]

So we've looked at your Kanban board. You also showed me earlier a photo of a physical version of one of these preparation boards. Can you pull that up? 

WALKER: [Joe pulls-up a photo.]

So this is my hotel room before the Wolfram [episode]. 

MATUSCHAK: So my instinctive reaction is this looks like a million times better to work with than the obsidian version. I'm curious how you take that reaction. 

WALKER: I liked this. I liked this a lot because I like operating in three dimensions. It's superior in many ways.  The limitation here is I can't easily transport this stuff into Mochi. 

MATUSCHAK: Yeah. But these are questions for the guest, right? 

WALKER: [Joe zooms into image on screen.]

 So the yellow cards (post-its) are ideas or points. And then the pink post-its are questions for the guest. 

MATUSCHAK: Interesting. And I can see there's some correspondence between [them], the yellow card has a point and the pink card, the question that follows from it. And then the red sections I see -  NKS, “New kind of science”, probably. 

WALKER: Yeah.

MATUSCHAK: So the book notes and the questions for the guest here are interleaved. How does prioritization and ordering and structure happen of the questions for the guest? Do you take all the pink notes off and put them somewhere else?

WALKER: So these ones on this mirror [are the] questions. And this is almost like a Kanban board. These are the categories —

MATUSCHAK: So are you rewriting the pink questions onto the yellow stickies there? 

[Andy points to an area of the photo that displays a second coding system.]

WALKER: Yes. And the yellow ones here. So now the color coding system has changed. The yellow ones here are my kind of final version of the draft question. 

MATUSCHAK: Pink is the category. 

WALKER: So there's like one, two, three, four, five, six, seven categories. 

MATUSCHAK: Do you need to get better at this process, somehow? And if so, why? 

Like the Taleb interview seemed good enough. As a listener, it seemed pretty well informed. I don't have a strong sense of: well, it would be 10 times better if you were 10 times more informed. It's unclear to me. Is there something you're satisfied with in your learning? 

WALKER: That's a fantastic question. So one of the realizations I've come to on this trip and in large part, thanks to conversations with you, is there's this growing bifurcation between the prep process and the podcast itself. And you almost overextend yourself in the prep process. And then people would just say the tip of the iceberg in the interview. 

MATUSCHAK: Right. That's why this is so interesting! 

WALKER: But also —I don't mean that in a self-congratulatory sense, or not only in a self-congratulatory sense — but I also mean it in the sense that a lot of the stuff I do in prep isn't actually ultimately useful in the interview. 

MATUSCHAK: You can't know in advance, right? 

WALKER: You can't know in advance. Maybe you could argue that it helps you avoid asking dumb questions or the wrong questions. So it is actually helpful, but just not in a way that's visible. But I do genuinely think there's a large portion of prep that doesn't end up being useful. So, the prep becomes a thing in and of itself. And the interview is just like a forcing function for doing the prep. And part of the reason why I’m wanting to share elements of the prep process as well, is because I think if I'm benefiting so much from those and the value isn't always cashed out in the podcast, other people should be able to get that value by looking at the prep process. 

So one answer to your question is, is I can keep doing more prep and better prep because the prep is valuable in and of itself apart from the podcast. I think another answer to your question is that the more efficient I can be with prep and the more I can compound knowledge between interviews. The more interviews I can do. 

MATUSCHAK: Right. If you can do 200 hours worth of studying in one hundred hours, you could do more interviews. I see. 

WALKER: Have a bigger impact, reach more people. 

MATUSCHAK: Yeah, that makes sense. Because I was thinking about what are the archetypes of interviewers, right? There's the low effort podcast interviewers, right? And you've left them behind and you know, Karikó doesn't want to talk to them. And now we have the Joe tier. And then I was thinking, well, what would it mean to be more prepared? What would it mean to understand the topics way better? Is there an archetype like that? 

And one easy example is to look at research disciplines, where most professional societies have memoir-style interviews that they will conduct. The association of physicists, something like this—I don't remember its name—and the mathematician association, both of them publish these interviews when a mathematician or physicist turns 70. Someone from the society will sit down and interview them, but they are a physicist or mathematician themselves. So it's a colleague doing the interview.

And that really does have a different character because they can go into the paper and probe it deeply. They can ask: Where did that insight come from? Did that come from this person? They know things that you couldn't possibly ever know.

So I'm wondering about what's essentially a continuum hypothesis: Is there a spot between the level of expertise that you acquire now with your current level of prep, and the far end of the spectrum, where a colleague physicist deep in the domain, who has published a hundred papers, is interviewing their more senior colleague? Is there something meaningfully different from what you're doing now between those points?

WALKER: That is an interesting question. I think so. I mean, I'm not sure what you’re contemplating by “meaningfully different”. And not every person who needs to be interviewed has the benefit of one of those colleagues to interview them. 

MATUSCHAK: That's true... 

So over dinner, we were talking about this idea of making accessible interviewees who are not served by that kind of a society. [It] is a very interesting question. 

WALKER: Because there are so many people who are not going to take the time to write a book. So there's value to digging the information out of them. 

MATUSCHAK: It doesn't really say much about what it would mean to be more knowledgeable as the interviewer of those people. Because you're still not going to be their colleague. 

But one model is that there are some people who go [to] some podcast hosts, often because they once worked in the field, [and] their podcast is focused on one topic. So maybe they worked in finance and so they're interviewing a bunch of people who do finance things and they do some prep for each interview as they learn more about finance. And there's an accumulation that really caches out over time.

And maybe they're not going to be an economist, but they are extremely expert for a non-economist. And so they get to kind of a different strata and you've pursued a breadth-y approach. It seems to make it harder to do that. 

WALKER: It means that the knowledge isn't going to compound [as quickly]. 

MATUSCHAK: I mean, clearly we saw some ways in which it does—

WALKER: Yeah. It will but it'll take a much longer time until the exponential curve starts to take off. 

MATUSCHAK: Yeah. That's super interesting. 

And you mentioned this aspiration to share more of the prep process. We last talked about this, I don't know, six weeks ago or something. Any thoughts? Or maybe you can just articulate what that aspiration is as it sits in your head right now. What do you want to do? 

WALKER: So I think I'll publish at least three elements: One is the [Mochi] prompts. So, I'll publish my Mochi deck for the Larry Summers interview; I'll publish my Mochi deck for the Fukuyama interview. There's only 80 prompts in [Larry Summer’s Mochi Deck] at the moment. 

So I'll publish —I'd be interested to hear what you think. I'm not sure how useful they'll be to people, but people might enjoy peeking under the hood of the process — I'll publish my Obsidian notes. And then I'll publish video recordings of me working and preparing. So to give people like a fly on the wall kind of point of view of what that looks like, I've already recorded some of those for both Fukuyama and Larry Summers. And another kind of video that I'll publish is calls with private tutors. So I've done a few of those again, both for Fukuyama and Summers, which are interesting. 

One kind of like mini challenge I'm contemplating there is —so obviously before I record one of those calls, I'd ask the person if it's okay, we record it and this is how I intend to use it — but making sure that they're not thinking about the future potential audience and that they're still just focused on providing like a private tutoring call. It can sometimes change the incentives a little bit, in a way that maybe cuts across the like efficacy of the tutoring  call. So I'm going to have to think about all these interesting ways in which I balance those things. 

I mean, the other interesting thing is there are a lot of people who, say with the Larry Summers preparation, I've had the privilege of speaking to a bunch of people who would want to remain confidential. So there's always going to be stuff I'm not able to publish, but I want to get people with different glimpses of the process. 

I've asked people what they would be interested in seeing and I was kind of surprised that a lot of the feedback I got was more about how I think about interviewing and preparing for the interview itself as opposed to the kind of background purpose. 

MATUSCHAK: They don't know what your prep is. 

And that's what's so interesting about this. Like you go to a podcast app and there's a bunch of podcasts that are interviews with scientists or academics or whatever. And they kind of look the same on the surface. Like, here's Joe talking to some academics and here's this other person talking to some academics. But there's this totally different character underneath and the nature of the prep. 

And they both just look like podcasts with lists of episodes. It's understandable why people have that belief. 

You know, you asked like, what might the use be of sharing the Mochi deck? My immediate, high level reaction is that I think there's just a ton of value in the meta of this, of people having some sense of what it looks like for one person to seriously study and prepare in this topic where they're unfamiliar. Because the nature of knowledge work and of studying, it's so invisible. Like it's people sitting alone with books and the things happening in their head mostly. 

So I think that fact leads people to these beliefs about like, well, this person's just really talented or well, this person's just really brilliant. And maybe there's some elements of that, but I think those beliefs persist at the strength that they have because it's invisible. 

So when I did, and we talked about this, this session with Dwarkesh where we sat down and we read this quantum mechanics book together. He had in the moment this great shock of like, wow, this is what it looks like to read a book. And to ensure that one understands as one is reading, what the things they're saying, to really interrogate the book, to be in conversation with the book. 

He found that really surprising and he reports post facto that it really changed the way he prepares. And I liked Dwarkesh, but I don't actually care that much about that one influence. I care way more about the fact that that video is very popular. And it's strange because it's a video of like two guys reading a book together for like two hours or something, but there's tons of comments on the video saying things like, “oh, I wish they'd taught this kind of thing in school”. And to me, that's kind of sad. This is so clearly not the right format for conveying this kind of practice or knowledge. It's two hours of just raw energy. 

WALKER: Well, your old live stream videos, I think there might be three of them on YouTube. Those were revolutionary when I found them. They deeply informed some stuff I did at Forage in my last [job]. 

MATUSCHAK: And again, what's so interesting about that to me is that it's so low effort. Like those were not special sessions. I did not think about what could be especially clarifying to show. They're totally unedited. They're three hours, so they're very hard to consume. And so it's easy to imagine improving upon this, that's why I'm excited about you doing this. 

Okay. I have one last question for you, Joe. You have these very charismatic sheets of paper on the desk that say, NO SEMANTICS and CONCRETISE. Why are these here? What are these doing for you? 

WALKER: These are reminders for the Fukuyama podcast. I mean, I could have just done these as prompts, but I think I have— 

MATUSCHAK: Reminders for you? 

WALKER: Reminders for me. So I've listened to a bunch of his interviews over the last few weeks. And I think two of the main failure modes is that interviewers tend to ask him about really abstract concepts, like liberal democracy or whatever, without putting it in terms of real world examples. 

And the second failure mode is a lot of the questions are of the sort: ‘What does this word mean to you? Or what's your definition of XYZ?’ And I find questions of that nature of limited value. I think for a couple of reasons. One is like a philosophical reason. It kind of presupposes this like essentialism, that definitions are somehow important. And secondly, almost by definition, I guess pun intended, you're not really creating any new knowledge. You're just asking him how he defined some word in a book and it's not a very useful question. 

So I mean, I probably don't need to remind myself. I feel like I'm good enough as an interviewer that I'm not really going to fall into those traps. But I think, it's just really important to me that I don't walk into those traps and that the interview is different and that we talk about real stuff and specific stuff rather than just: what's happening with liberal democracy? What do you mean by liberal democracy? 

MATUSCHAK: I'm reminded of [Robert] Caro's “is there desperation on this page?” He has this wonderful practice of pinning index cards up next to his writing session that are kind of thematic reminders like this. 

And one of the ones that's most charismatic is about LBJ: “is there desperation on this page?” He wants desperation on every page. 

Well, thank you so much. So this is super, super interesting, at least for me and hopefully for other people as well. 

WALKER: Yeah. This has been a lot of fun. Thank you for coaxing these questions out of me and these answers out of me and helping me to articulate all of this. And really, all of this, this whole tech stack and these various practices have been influenced by you. So I'm very, very glad I discovered you and your work. It's been a game changer for the podcast. 

MATUSCHAK: Cool. Great. 

WALKER: Thanks, Andy.