Building entropically, Gemini, Prompt engineerings revenge, and Superpowered

0:00 Nabeel Hyatt: technology tends to go through ages of entropy and de entropy. We all love, especially as engineers, we love de entropy, we love simplifying everything, cleaning it up getting the signal from noise bringing it all down into something that works. Things that are trying to make a promise of de entropy too quickly when all of these LLMs are so new Just feel incongruous to me when the goal is solve the problem reliably and, and we're still not at reliable solution. Fraser Kelton: Boy, we wrestled with this one, but that one feels really right. It's going to get more complicated in every direction because we are not at the reliability required for consistent value in many use cases.

0:44 And like, why bother adding abstractions of simplicity if you say it's still not going to be good enough? Nabeel Hyatt: Yeah, exactly. Fraser Kelton: lot easier for you to get something than broken into production. Nabeel Hyatt: That's what, that's the headline. Why are we making it easier to get broken things into production? Fraser Kelton: Oh, but what a teaser. Nabeel Hyatt: I know, I know. Hello everybody. Welcome to Hallway Chat. I'm Nabeel. Fraser Kelton: Fraser. Nabeel Hyatt: And we are here to talk about what we've been talking about in the world of AI mostly. I, I didn't really know we were signing up for this every week when we signed up. They come fast, Fraser. I felt like I just talked to you on the hallway chat last week about the launch of ChatGPT and all of your stories around that.

1:26 But at the same time, there's like a million things to also talk about. So it is both, feels like these C shows are coming all the time, but also too much to talk about. Fraser Kelton: I was thinking after recording the last one, how nice it is to be able to talk in depth with you about these topics and just laugh and explore. And so I'm good with it all. The, you know, here, here's something last week, I think I said the line, where's Google. And we had an answer. Kind of, right? We kind of had an answer. Nabeel Hyatt: Yeah. Oh, I loved, we had our AI dinner this week, and we had somebody from the Gemini team sitting at the dinner, all night long, mouth shut, and I am like spouting off and spitting all kinds of stuff about Google.

2:08 And I don't know that I picked up the smug look on his face. Fraser Kelton: I Nabeel Hyatt: yep, tomorrow morning you're gonna see that Google's got a little bit of a comeback. Although I think it's a little bit of a comeback, right? So... what is Gemini, Fraser? Fraser Kelton: Gemini gets announced in summer by Google, where they say, we're training, A large language model that's going to be amazing. And has now become a little bit of a meme because they have talked and talked about what they're about to do and the rumors. Nabeel Hyatt: their answer to OpenAI and Anthropic and the others. Yep. Fraser Kelton: Yep, that's right. And we're at that dinner, as you say, and it's become a little bit of a joke as like, where are they?

2:53 They've been talking about this for months and nothing's here. And then, boom, we wake up the next morning. I get a text from my friend that says, surprise, and they've shipped some of Gemini. They haven't shipped the most capable model and they shipped a lot of. Demo videos which we'll come back to and talk about a little bit, but they've announced a something called Gemini Ultra, which is, you can think of it as the equivalent of GPT 4,, then they've shipped Gemini, I don't know, the terminology is crazy, Pro, Gemini Pro, and Gemini Ultra is not available, Gemini Pro is available as of the day of the launch, and that is comparable in performance, at least on the evals, the evaluations, to GPT 3.

3:38 5, and then they have What I think is probably maybe one of the more interesting things, they've then distilled it all down to something called Nano, which can run on, and is running on the Pixel, which is a pretty awesome thing for them to have done. Worth calling out, Gemini Pro, the mid tier model that's equivalent to 3. 5, is now live and integrated into BARD, their ChatGPT like product. And it's, you know, one year after the fact that that's been rolled out broadly by OpenAI, and so quite a Nabeel Hyatt: I have to admit, Fraser, If this is nothing else, it reminded me that BARD exists.

4:23 It was the first time we've even mentioned BARD. Like that, that, how useful it is in at least our workflows, right? Fraser Kelton: it has reminded me that it's a thing. It still didn't encourage me to go use it. Like, I'm not sure what this is an answer to in terms of ChatGPT. So, they now have a parity with the model? Okay. Right. Okay. Sure. Nabeel Hyatt: We were texting and I've got some WhatsApp groups that were fiddling around and talking about this when it launched in a Discord group of AI engineers and so on. And I got to say the evals of course went from oh my god and then the demo videos, oh my god, to pretty quickly, hey, is this a bunch of BS? Fraser Kelton: Yeah, that's right.

5:04 I think the first thing to call out is I think the general level of performance across Some eval benchmarks give you a sense of where relative performance of the model might be, right? So GPT 4, far outperforms basically anything up until the release of Ultra, and you could then have probably high confidence that it's going to perform in a very different class once you get it into production. I think the thing that we can say is without having actually played with it, the evals suggest that Ultra, the large one is directionally equivalent to GPT 4. And That, that's something, right? I think that the fact that we might now have two GPT 4 type equivalent models in, in market proves that somebody else other than OpenAI can do something on this magnitude and, Nabeel Hyatt: not willing say that yet.

5:53 That's I'm not absolutely crap. Like I'm willing to say that soon and, and, but, but, Fraser Kelton: Yeah, that's fair. , Nabeel Hyatt: the MMLU benchmarks are comparing a five shot reported GPT 4 benchmark to a 32 shot, I think it was. If I remember correctly, UltraReport. It's just not, those are not comparable at all. Frankly, the stuff that everybody would probably normally out of the box use this product for, the average consumer, those all, at least the evals we can see, seem comparable and seem fine. So let's not overstate some kind of eval problem where there isn't one, least in this specific case.

6:35 But, the kind of like general math cases in particular seemed a little cooked and unfortunate. And I think, frankly, when they're speaking to a highly technical audience, I'm not sure why they were doing that. Like, I, I don't, I don't Fraser Kelton: Yeah, yeah, that, Nabeel Hyatt: it's not them. It just felt like a hiding the ball when clearly even with code, like the code generation at a Gemini is quite good. I just don't understand why they even did it. Fraser Kelton: I think if you strip it away and you look at a comparable measure of five prompt, five examples in the prompt, GPT 4 outperforms it, but it only outperforms it by a couple of percentage points. But again, like, I think that once we get our hands on it, my guess is that we will see that this is directionally similar ish.

7:17 Nabeel Hyatt: let's finish on this rant and then I actually want to talk about the thing I really loved about Gemini. The part that I think was unfortunate and I hope no startup takes away from is that everybody gets excited because there's an announcement from Google that there's finally Gemini out and within a few hours, it just dawns on everybody that, okay, Gemini is not really here because it's just pro, we don't have an exact release date still, this, this graph of evals is, Cooked in a couple places and the rest of them are still comparing to GPT 4 back in March eval tests and only beat them by three five percent and since then GPT 4 has gotten a lot better and By the way, evals don't really matter.

7:58 So like that's the negative side the positive side is So many companies absolutely fail to show their product in action in unique and novel ways that pull the heartstrings. And I think they, if you haven't, if you guys are listening to this on the podcast and you haven't yet watched the Gemini, Demo videos go on YouTube, you should take a look there's some wonderful craft work that is not too pretentious and not too overblown, but just, in fact, it's very clean and simple, except for the YouTuber Roper, Roper video, that's kind of overblown, but the rest of it is very simple, good demos of showing various ways that this product can be put into use that an average consumer, which might be the aim of this announcement, is much more the average consumer You know, or Google stockholder then it is aimed at engineers.

8:51 I mean, didn't you love those demo videos? You watch them, right? Fraser Kelton: Oh, yeah, but I, I don't know if You're getting me riled up here, man. I had shared the one demo video where they show it, the three cup technique where there's one ball underneath the cup and then they shuffle the cups around and it tells them which cup it's under. Because this, It's a multi modal model that has been trained from the start for multi modality. So, it's accounting for text and image and, and video and, and audio right from the, the start of pre training steps, rather than, you know, kind of bridging that in after the fact. And this, this demo video is one of the best demo videos I've ever seen.

9:35 Nabeel Hyatt: Yep. Fraser Kelton: And then, and then it comes out that it's all fake! Nabeel Hyatt: Right. Fraser Kelton: So in the demo video, go watch it. It's, it's remarkable. There's a a set of hands and some cups and a ball and the demo says, okay, now I'm going to put the cup under here and move it around. Where is it? And in, in real time, the Gemini voice comes back and says, the cups, I don't know, under the left side, and the man lifts up the cup on the left and sure enough, there it is. Nabeel Hyatt: Right. Fraser Kelton: And then, People have discovered shortly thereafter the fact that this is basically the equivalent of like a simulated scene where they had to prompt Engineer along the way that said, just turns out to be they fixed it in post.

10:19 Nabeel Hyatt: Yeah, that's really Fraser Kelton: Now, I will compare that, I will, yeah, I will compare that to Greg's demo of GPT 4, which was all live. without any editing and in real time. And that is the, I think that is the way that you introduce your products. It's brave. It's brave, right? You're doing it live. It could fail. And you, you're owning it because you have so much confidence in what you've built. Nabeel Hyatt: uh, I don't, I can't entirely disagree. You know, it is as much as I love the product demo, what I would have loved was a demo like that and then a how to behind it. You know, I think it's okay to make things that are somewhat polished and beautiful, but it would be great if it if it turned out that they reveal the covers.

11:04 And by the way, that's no implication on Fraser Kelton: think, I think, I think Paula should be. Nabeel Hyatt: separate Fraser Kelton: Yeah, yeah. Polished and beautiful is good, but I think it has to be grounded in reality. This is a case where they edited across two different dimensions, and people came away with a dramatically different perspective of what it is that's actually happening. And so I think it's unfair to not allow anybody to use the product, and then introduce it with a demo video that basically obfuscates the truth from two different perspectives. That's just weird. I think that this is An excellent moment for everybody at Google because they've shipped, or at least they've partially shipped and I think that you, they've taken the first step.

11:49 No, they've taken the first step. No, no, no, no, no. Like, listen, this, the race was set off a year ago. They, they did this in a year. foR a company of their size, this is, this is not to be, scoffed at. Think about all of the complexities. They had to live through smashing together brain and deep mind. They had to go and find, like, the path through all the bureaucracy and politics to get an aggregate amount of compute required to be able to do this. They had to solve all of the different challenges, both technically and politically, within the organization to do this. And it's out. And I think that, that itself is something that should be respected. And, we can squabble over the evals and stuff, and the proof will be when we actually get to use it.

12:30 But it looks, it looks directionally good. And that's something. You know, I, I feel like I also did a good job playing your role. You usually are the one who's clairvoyant in, in many respects. And at that dinner, My guess was that Google was going to come roaring back. Nabeel Hyatt: your quote. Fraser Kelton: that. Yep. Because they are The best at, at the technical pieces that have to come together for training a model like this. And if you look at some of the stats, I forget what the stats called, but for basically the measure of efficiency when they were training Ultra, I think they reached some level of like 90%, 97 percent efficiency in the utilization of their hardware when training Ultra, which is just a remarkable achievement.

13:17 And this is the area where we should expect them to be great. And I think they have shown that they can be great, if not, you know, on a year's delay. And then I think the real challenge for them is going to be how they bring the great technical piece into their two products that are now the two front war. BARD, as you laughed earlier, is a thing, and then the second one is they're going to have to find the right way to integrate these technologies into search. And that's going to be an excruciatingly hard challenge because it's orthogonal to the business model that is search historically. Nabeel Hyatt: Look, I, I, I, I say if I were, I'm not to speak for Demi or Eli or anybody else over the Google team and what they're doing, I'm sure they know a lot more about how to do this than we do, but, but I do think their role or their way to fit if I were trying to navigate this space and I was Google was to take almost an Apple approach to this given their scale and size.

14:15 And what I mean by that is I, I always joke people think of Apple as innovative, and I think of Apple as a last mover advantage, not first advantage company. They have had a few moments in their life where they have been very early, but in many ways it's taking the things that are already out there, that are already somewhat proven. And then making them so polished and so well thought through that you just, they feel like they fit into your life immediately. And, you know, they were not the first to release little notification widgets on a smartphone that was Android. They're not the first to do wireless charging that was Android. Go way back, they, they, they took a lot of their early ideas from Xerox PARC. And so if Google wants to the game of being last, because it's really gonna work and work reliably, there is a game to be played there.

15:01 Because I don't think OpenAI wants to play that game, to be honest, and you can't play both. I think right now, in many ways, OpenAI Playing closer to the Android or Samsung, if we're going to use smartphone analogy model where they are riding the front edge of development. It drives them crazy if somebody else gets something out new ahead of them and they want to play the front edge of the game. I think both can be successful strategies as long as the thing that Google eventually releases As you get to Ultra is worth the time and energy. That's the, you know, the, like, it's worth weight that that's the thing that will be left to find out. Fraser Kelton: We shall see. We shall see. Nabeel Hyatt: look, not, it's hard.

15:42 Of course, the cup example is tough. You know, these, these prompts are hard to shape. It's hard to get the little alien inside my computer to understand that I'm playing a Cup game. Fraser Kelton: Issue with the cup thing is that they imply that they lead the viewer to believe that there's zero prompting. It's not that prompting's hard. The way that the video is presented suggests that there's zero prompting and that there's this real time multimodal model watching you and, and the cups and inferring with real reasoning and there's somewhat complex prompting happening at each step behind the scenes, which is what I think has, has caused everybody to be really disappointed in, in a decision to do that.

16:29 Nabeel Hyatt: The last thing I'd say on Gemini, is that, is a lot of this consternation would have been solved. If they would have released APIs for developers to build with at the same time. And I think, I think we've, supposedly gonna come out December 13th. I don't know if Ultra's gonna be involved in that. But in a world of AI movement that's five, seven days from now, I mean op open AI fires a CEO and goes through a, a coup attempt then gets back a CEO in that time period. Like a lot, a lot happens in five days. Fraser Kelton: lot happens in five days. Nabeel Hyatt: and so, like, I'm sure this was PR oriented, they wanted people to watch a Mark Roper video and so on and so forth before developers had control of the narrative.

17:16 But it's really unfortunate on, because I think it creates a sense of doubt when it shouldn't be. It should just be high fives, hand clapping, and playground. And I think that was a little bit of a PR mishap, but we'll, we'll see what happens in, in seven days. Fraser Kelton: Yep. Yep. And anyway, prompt, prompting is hard. We, we talked last time about efforts in ChatGPT to simplify the complexity and ambiguity of prompting specifically with DALI, where they want to take the, the three words that somebody who's unfamiliar or lazy with their, their directions wants to do and how, if you're a power user such as yourself, it's just suboptimal.

18:02 Nabeel Hyatt: Yeah, it's, Fraser Kelton: What, what Nabeel Hyatt: I saw my great example of that this week because I'm going to keep banging the drum that I think Prompt Engineering is a real skill and will be a career for quite some time and that actually Prompt Engineering is going to become more of a language. Before it eventually gets abstracted out, but our ability to totally abstract it out while we're still trying to figure out what these non deterministic models can actually do is very, is very far away, maybe years and years away before we can build these systems on top of them. I got handed a wonderful example of this today, I sent it your way which is Anthropic.

18:42 Which we are investors in by disc full disclosure to everybody. Anthropic has a competitive model to OpenAI and Gemini called Claude. And there is a well known research problem and execution problem in these long context windows. of AI where I'm asking it to, for instance, look at an entire PDF or look at a long chat and find some phrase or find some word inside of that, did Sam talk about the beach or not, or what's the best cooking technique? And it turns out that if it's mentioned in the beginning of a doc, Or at the end of a doc every model, all of these LLMs show that they can find information at the beginning of a doc and at the end of the doc faster and more reliably than in the middle of the doc.

19:26 The middle, it's the missing middle it just sometimes misses stuff. Well, this has been a quote unquote known thing, of which people have been trying to do all kinds of different engineering techniques chunking the data into smaller bits, and then there's like comparison evals against different models at different times, and how they perform on the missing middle, and so on and so forth. And then it turns out that Claude releases a paper today called Claude 2. 1 Prompting, that says, well, what did it say, Fraser? What's the crazy, deep Engineering technique that, that scientists have figured out in order to finally unlock moving from 23 percent missing middle accuracy up to 97 percent missing middle accuracy.

20:10 Fraser Kelton: mean, they add a line to the prompt that says here is the most relevant sentence in the context, which basically nudges the prompt to go and pull out the relevant sentence for the question at hand. And that's the bump, Nabeel Hyatt: Yeah. I mean, that's insane. This afternoon, I'm going to do some work and figure out whether this works in GPT 4 as well. I didn't have a chance to come up, but, but it'd be really interesting. If both of the results, don't you think, Fraser, would be interesting? Like, if that phrasing does work in GPT 4 as well then it's like, oh, you just figured out a new incantation, kind of like we found out that if you, you tell a model you're going to tip it to do something, I'll give you 20 if you answer this question.

20:58 They, it tends to perform better in that question, even though, of course, you're not giving the model 20. Another crazy incantation. And then if, so one, if it worked, that's interesting, and it tells us more, a little tip into the language of how to use these models for large context windows, which is particularly valuable for Claude, because it has such a large context window. You can just put lots and lots of text in there. If it doesn't work in other models, that's even more interesting, right? Now, for all these companies that are trying to say, don't worry, I'm building middleware dev tools that let you switch in and out models arbitrarily, with the, like, like they're all the same. That they're not. Fraser Kelton: I would be so surprised if they're the same today.

21:40 And the difference is only going to grow over time. There's a whole bunch of different things going on here. This is a quote unquote eval called Needle in the Haystack. And I think that, yet again, this is a situation where the eval doesn't measure anything proximately close to what happens in, in production for people who are building products, right, is if you, if you insert into the middle of some financial set of documents, a single sentence that says Dolores Park is the best place to have a drink in San Francisco and then the model can't find it, right. I'm not sure that that is reflective of any real world problem that people are trying to solve with this.

22:21 The other thing that is so interesting here is the Anthropic model they hypothesized performed poorly when people were running it through that, that eval because they've trained their models to cut down on inaccuracies specifically for these types of use cases. Right. And so they basically have trained the model to say, okay, if something feels completely orthogonal from the rest of the documents, it's probably not something that's, that's important and or accurate. It's probably not even accurate. So, so just ignore that. Right. And then the eval is basically testing for the model performance to do exactly that. Nabeel Hyatt: Yeah, but I want to get back to the point that I wanted to make, which this is six words that you put into a prompt if you were trying to do long text retrieval, text from a long context window that, that does boost performance and.

23:19 I don't know, it just, it tells me how naive we are collectively about how to use these models. Um, Emma, who's an AI hacker in residence for us, she did a benchmark on some internal tools that she was using on Glaive versus GPT and she found that without prompt engineering, Glaive did better than GPT 4 probably because it's trained only on highly quality synthetic data and so on and so forth, but that if you added the sentence, You're a well known historian to the prompt for both Glaive and GPT that then 4 suddenly did better. And it's just another good testament to you just need to find the magic five incantation words to suddenly make your business be able to move into prod.

24:09 That's Fraser Kelton: that is so crazy. They could just even try to internalize that in the brittleness of these models. You're going to have, you are, you're a well known historian and then it finally outperforms it. I don't know how this gets solved, like, other than at the system level. Nabeel Hyatt: But, in the very early days of video games, you worked at the assembly level to make a video game. In the early days of computer graphics, before we got to engines, we had to work in code. And we will eventually get to lots of GUIs and engines. And we've actually talked before about how prompt engineering is not how every average user to use these products and people are bad at English. But at the same time, if you need performance, you need to be at bare metal as close to the model as possible for probably a little while until everything really, really works.

24:59 And at that point, when it's automatic when we've made our 50th first person shooter, That's in production and making hundreds of millions of dollars a year. Then we can talk about making an engine for making first person shooters. And when we get in game parlance, you get Unreal and and you get Unity and so on and so forth. But it feels like we are still in, you know, how was Pac Man built? Fraser Kelton: I don't, I don't want to open up this can of worms, but don't you think that is a measure of. The model's capabilities not being strong enough. Nabeel Hyatt: We know that. Just like in, like, early programming days, you were wrangling with the amount of memory on computer. You've got a thousand twenty four bytes of memory, and you're just trying to make a spreadsheet work in this tiny little bit of memory, and you need every little squeezing bit of thing just to make it operate, right?

25:51 And isn't about speed the way it often was back then but it's still about whether the job can be done well or not. And, and yeah, we'll need to be very close to bare metal, until all these things run perfectly all the time, and be about efficiency and cost and abstraction Fraser Kelton: what Yep Nabeel Hyatt: the rest of that stuff. If five words for your specific use case going to increase performance, then I don't know if I'm Fidelity or Procter Gamble or Figma, or InstaWork or another startup , like, I don't know that I'm willing to take the future of my business's effectiveness in AI, which could twist and turn on five words, Fraser Kelton: right.

26:37 Yeah. Nabeel Hyatt: And who's going to figure out those five words for your specific business? It's certainly not going to be some random middleware company. It's going to be you because you care about your company and you've hacked away at it or had a prompt engineer who's hacking away at it. You've really worked it to try and figure out how to wrangle this alien to do the work that you want it to do. Fraser Kelton: The point here is that the brittleness of these models today across different use cases suggests that you're going to want to have people, quote unquote, like, working at the metal. Yep. Nabeel Hyatt: analogy I would use is, in the really early days of the web, there was almost immediately A bunch of WYSIWYG web page developer software companies.

27:22 There were 30 startups that were like, you don't have to learn CSS and HTML just use our little product, and you can get your web page out without tweaking it at all. And, you know, if we fast forward 10 years, of course, there's many of those companies. Today, there's Squarespace and webflow, a bunch of these companies that are helping everybody from a restaurant up the street all the way to complex enterprise websites. But in the early days, As a good example, prior to CSS, the way that you laid things out on a webpage, so the way I got something to show up on the right hand side of a webpage versus the left hand side of a webpage, was to use a kludge which is to build a table, kind of like a spreadsheet on that webpage in HTML, and then, in one of the cells on the right hand put my logo so it's on the right, and then make the cells of that spreadsheet And it's a, for me, it feels like we are way more in that land than we are in, in WYSIWYG abstraction land.

28:29 And so the whole first wave, the whole first couple of years. Of WYSIWYG website builder companies all went out of business very, very quickly. What, what happened there? If we're going to use that analogy, what happened there? WhAt would be the business, if you wanted to help a million companies build their first LLM applications. And the contention is that it's not the time to build the square space of the space, uh, which I'm not, by the way, you know, this is us just chatting on a podcast . A founder could walk in tomorrow. And pitch and pitch the most beautiful wonder idea for Squarespace for AI and, and just prove you totally wrong.

29:11 And that's the joy of this process, Fraser Kelton: That's, that's what this rule so fun. Nabeel Hyatt: Yeah, exactly. So strong, strong convictions, really loosely held. But, actually, do you believe in my analogy? Do you think that's an apt analogy or do you think I'm full of it? Fraser Kelton: No, I don't think you're full of it. So, if I understand what's happened in the Anthropic case, it is The way that they have tried to nudge the model to improve performance has then resulted in some wonky behavior that you can then nudge it over that hurdle with five magic words. And what does that say to me? That, that says to me that there's probably a solution that happens at the system level.

29:54 If you think about how this may mature why would they want their customers to ever have to think about that? They'll, they'll find ways to absorb the solution or abstract the solution for use cases where it makes sense. Nabeel Hyatt: Yeah, but I don't have time for that. I'm a founder that wants first mover advantage. Or, My boss me that I need to have an AI strategy and I need to, I need to launch next month and it can't, it's got to get out of demo land cause I've got an earnings report next quarter. Fraser Kelton: This, this is why that that person is having. Random success. Sometimes they're succeeding, sometimes they're failing. And sometimes they come back to the drawing board with an entirely new approach one month later. We've seen that a lot. Nabeel Hyatt: That's very true.

30:35 I do wonder. If Procter and Gamble and, and Fidelity and JPMorgan and every other company is trying to figure out how to use AI. If I just think about the web, the web analogy for a second, and you don't want to overstretch any analogy of course, but the really effective companies in that first wave for helping to bring everybody onto the web were kind of a mixture of tools companies slash consulting companies. Fraser Kelton: Yeah, Nabeel Hyatt: It was scient and viant and Razorfish that, you go pay them hundreds of thousands of dollars and they would build time magazine. com for the first time, these kind of mixture of design agency, software engineering, and then they ended up with internal tool stacks that they knew how to use.

31:22 I think there's an analogy to Fraser Kelton: Oh, hell, yeah, I mean, yeah. There's a reason why OpenAI has, I forget, I'm going to get the names wrong here, but has a Keystone partnership with Bain and Anthropic has a Keystone partnership with BCG. Is these are footsie things to bring into the enterprise, as we've seen. Five words makes the difference between something that looks horrible and something that would be delightful in production, and there has to be people who can help you navigate that, uh, as the world is changing underneath your feet. Three months. No. Nabeel Hyatt: Well, the contention is right that, Razorfish and Scient and Vine were net new org, yes, they were consulting organizations that rhyme with Bain in the way that they actually but Bain is old school.

32:12 Are there really great AI implementation engineers waiting at Bain to take you out to market? Absolutely not, I would guess. I, I suspected it's a net, that there's an opportunity for a net new company to be filled with people who like to implement, who will help take these tools, which seem maybe very easy to stand up very quickly. I can just go to a prompt and type things in, but I think are probably more complicated and people will find are more complicated than they think. To actually implement and get live. And that's why I like the HTML analogy. It's incredibly simple to build your first HTML page. But then, and it feels like anyone can do it.

32:54 But actually trying to run the NewYorkTimes. com you know, is another whole order of magnitude more difficult. And especially in the early days where people didn't really know web and how to do web development. You needed a set of people that were your launch team and stood up the internet, you know, website by website. I think there's a little bit of that that probably goes on and I just don't think it's going to be McKinsey or Bain or the folks that have, Really very little of this specific type of DNA, but I could be wrong. Fraser Kelton: Yeah. People who did it back in the day for transitioning people onto the internet. Did they do it through just specialized know how, or did they build tools and platforms that allowed them to, to simplify the task for others?

33:41 Nabeel Hyatt: Like anything you start out making a thing and then you're like, once you've done it two or three times, engineers can't help themselves. And so you start to build efficient Fraser Kelton: is it, but that a, so, but are we back to this is, there actually is a middleware company, like a tool that's going to start from a consultancy type perspective and then get built out? And then is your, your issue with the tool startups? Just the fact that they're not going to market appropriately. Nabeel Hyatt: That's a good pushback. It might be. I mean, we'll, none of us know, we'll see how this all plays out, but yeah, maybe the right way, it's not what VCs want. Hey, why don't you hire more implementation engineers? It's not what VC on a panel would be.

34:22 They'd be like, no, no humans. The AI should write itself. But for where we are on the technology side it might be that the right answer for the next 12 to 18 months is. You have a whole bunch of implementation engineers that are script monkey that know all of the unique folklore about how to wrangle these models in the right direction. So you're still selling your tool set, but you're selling your tool set along with a handful of implementation engineers and a maintenance contract. And I know that that, that breaks a lot of the purity software that we would all love for engineering to be, but it might be the right thing for, for this particular stage that we're in.

35:06 Fraser Kelton: Could be . You know, going back to the start of the API, there were two people, a guy named Boris and a guy named Andrew at OpenAI who were prompt wizards, like they just knew how to, to construct and orchestrate these things in a way. And that's what, that's what they did. They ran around to the implementations that seemed most interesting and then helped them sand off the rough edges to see if it was a path to production. And in many cases, They could nudge them there, whereas as few, few people could. Nabeel Hyatt: Boris is a great name for a startup. Fraser Kelton: Yeah, he is remarkable. He himself could be a startup. So you don't think that these things get abstracted the other way, where they get pulled down into the actual model level, and that people aren't interacting with any of this above that.

35:52 And, and it kind of ties back to the Gemini thing, right? Nabeel Hyatt: Oh, I think that's a very good point. Very likely that that happens in parallel. And, technology tends to go through ages of entropy and de entropy. We all love, especially as engineers, we love de entropy, we love simplifying everything, cleaning it up getting rid of the noise from signal bringing it all down into something that works. But when things are not working fully you can't jump three steps ahead, you have to go through a phase of entropy. It's why I don't get nervous about One more model launching or one more startup launching. We need as many shots on goal and bets to move this technology forward as quickly as possible.

36:37 Things that are trying to make a promise of de entropy too quickly Just feel incongruous to me when the goal is solve the problem reliably and, and we're still not at reliable solution. And so my is that reliable solution is going to get way more complicated before it's going to get easier. Fraser Kelton: Boy, we wrestled with this one, but that one feels really right. It's going to get more complicated in every direction because we are not at the reliability required for consistent value in many use cases. And like, why bother adding abstractions of simplicity if you say it's still not going to be good enough? Nabeel Hyatt: Yeah, exactly. Fraser Kelton: lot easier for you to get something than broken into production.

37:19 Nabeel Hyatt: That's what, that's the headline. Why are we making it easier to get broken things into production? or, we could just fix it with marketing Frazier. Fraser Kelton: No, man, this is like, I'm listening to you talk about Gemini and then like nudge me and I don't, they, they misrepresented what the product is. Nabeel Hyatt: you, you're not reacting to the evals. You're reacting to the demo video. Fraser Kelton: That's right. Actually, I don't even care that much about the evals. I think it's more interesting to consider that all of these models are going to have different tricks ranging from those five words that Anthropic had to do all the way up to like Q star with test time compute type stuff. The thing that bothers me is video. And I just thought about what the equivalent is.

38:01 Remember like a decade ago Apple started marketing their new cameras by showing you the output of the iPhone camera when they announced it. And then I don't know whether it was Samsung or LG. And when they announced it, they shared photos from DSLRs and they silently just wanted people to infer that that was the image quality that was coming from the phone, and then people discovered within an hour that it was a digital, like, SLR that took the photo. That feels exactly what happened here. And I'm sure that the demo with a little bit of rough edges that they would have had if they had shown us the prompt steps in between and the wait for an inference to occur still would have been a magical moment and people would have lost their minds, but because we feel misled, it erodes our trust and we feel betrayed, which is a very funny thing to say.

38:53 This reminds me of A moment that has surprised me, and there's a lesson here broadly for founders, and it's not just, you know, be honest in your marketing material. I knew when I was a founder that the common wisdom was just be completely upfront with VCs because they have seen so many pitches that they can sniff out when something doesn't sound correct. I will tell you in a pitch a couple of months ago you may not remember it. There was one moment where you paused, you raised an eyebrow, you asked one question. And it was, it was not an aggressive question, but it, it pulled the first thread that got to the truth.

39:34 And my sense is, in that case, if he had just been up front We reach a slightly different outcome versus having to pull that thread and discover that there was a little bit of, of deception in how he was presenting things. Nabeel Hyatt: Oh, well, to be clear, he was trying to put a gloss on everything. And I do remember exactly that meeting the company had gone through a pivot, some founder breakup y stuff, you know, just a lot of change. And I think. And had been around for a little bit, all things we didn't know when that founder came in to present, but we take first meetings all week long, like, if we're not good at reading people and figuring out what's really happened, you can't do this job.

40:17 And meanwhile, the most important part of this job is Establishing whether the other person across the hall is authentic and you can trust them, because you're going to be on a long journey together. And so, before it's a good business model or it's an amazing product or it's somebody you want to work with because you love the intellectual banter and you think they're going to be a great leader or whatever else is going to get you excited about this startup, you can't do it if you don't think they're all being authentic and real and honest with you. And so, Yeah, that was a founder who had clearly gone through some stuff, Fraser Kelton: we don't care if they've gone through stuff, right? Of course. that's part of the Nabeel Hyatt: we would love if gone through stuff, like, you learn some lessons, just own it, and tell the story about how your, you started thinking it was this other thing and you were just wrong, or, Bye.

41:07 You moved to this town and it was the wrong town because there was just a bunch of fly by nights. You took this founder on board, but they were just a ne'er do well, so you had to get rid of them or just whatever it happens to be that, that you went through. You just want your learned insights. And I think way too often people want to tell a glossy story about how everything's up and to the right and you got to get on board right now. Because this round is, oh, the other trick is like this round is closing in two days. This stuff, like create senses of urgency. None of that stuff, all of that stuff just hurts. If you have a slow fundraising process, first of all, people probably already know. Just say, I think they're all dumb. This is the reason I think you are going to be smarter than all of them.

41:49 You can try and go to their ego. So I'm not saying you don't try to storytell. I'm just saying you have to know how to do it with. Being yourself and being authentic to the journey that you've been on. Fraser Kelton: Yep. How many times have we seen somebody, oh, okay, the deal's coming together in two days, you have to move quickly, and then we're like, okay, well then this is not the deal for us. And then all of a sudden you see them try to backtrack fairly quickly. Well listen, we really like you, so maybe we can give you a couple extra days. And you're like, alright Nabeel Hyatt: I had the exact opposite thing happen to me last week where I had a founder email in and I passed over email and I wrote up, but I wrote like a good little paragraph about why, like this is the thing that these are the reasons that I'm not sure you're going to be there.

42:32 And I've gotten, mostly you get crickets, they're going to move on, which is totally understandable, right? The second thing you get is defensive, angry feedback that I'm just dumb. Which I'm not sure exactly what that sales tactic is, but whatever. I got back from that founder you might be right. Here are the things that I think I've worked through. To try and prove what you're saying wrong, and then gave a couple of little notes of the other things they tried that don't come out, of course, in the one paragraph pitch of the other versions of that business over time, the struggles they've had, and so forth. I mean, I got on a Zoom on that person, like, three hours later. Fraser Kelton: Yep. Nabeel Hyatt: I was like, Oh, you're Fraser Kelton: get it.

43:12 I get it. Nabeel Hyatt: this business. And you're authentically trying to engage with me on it. And you're not combative about it. You're just having a conversation about it. Like, awesome. And I, you know, didn't turn into an investment that day. It may in the future we'll see, but I certainly hold that founder in really high regard. Fraser Kelton: I get it. It was amazing to see, some depth of experience such that you just, you knew based on two sentences that something wasn't right. And it just reinforced what I had been told when I was a founder. Don't bother, right? Nabeel Hyatt: It's similar when you're pitching and you don't, you're trying to gloss over the particular risks or problem with your startup, the old real estate trick.

43:54 That Realtors use is when they show you a house they list all the wonderful things and then, and then they're doing the walkthrough. They talk about the one thing that's the problem with this house and what they're trying to do focus your time and energy on the one thing so you don't think of the 30 other things. That's very different from authentically having a conversation with an investor about your business. But similarly, these are early stage startups. There's no way nothing is wrong with your business. Fraser Kelton: Yeah. Nabeel Hyatt: And so you might as well talk about the things that you think are really risky or are broken or that you haven't figured out yet. Because the right investor is going to be the person that's going to be like, I don't think those are real risks, or I'm willing to take on that risk, or like, I think you can solve that risk, and that's, that's the right way to have the conversation about the path.

44:37 Nobody expects these things to be totally finished and that's a very, Fraser Kelton: sure. Yeah. Somebody internally here said that the quickest way to a no is when there is no risk, right? Because that's not a venture business. That's, that's not for us. Nabeel Hyatt: When a founder feels like They know the problems they think they've kind of solved and the areas where they're self reflective and self aware enough to realize they've got a lot of work to do. And you can have an open and honest conversation about doing that work.. Fraser Kelton: Mm hmm. I also think the challenge here is that every firm is different, right? And so, whenever anybody has shown us a demo, you see almost everybody in the room lean forward. And when people have had stilted presentation pitch mode, everybody's, you know, kind of in lean back.

45:24 Nabeel Hyatt: At Spark, we like demos, we like talking product, and that's, you know, but you're right, that's not how it is at every shop. That's not that's not how lots of investors operate. Fraser Kelton: that would probably be the challenge here is everybody operates differently. That's where you have an opportunity in these moments to find the person that you want to be with for a long time. Right? So there, there will, there are different founders who appreciate different types of techniques too. Nabeel Hyatt: It can feel from the fundraising side, and I certainly felt it as a founder, like, I just want somebody to give me a first term sheet. I'm just trying to raise capital, whoever it can be. But that's a little bit like in today's age, like applying to college by just saying, I, I really love your school because it's a great school for learning.

46:06 And it's, that's not a great way to get into college. I had an admissions person at NYU tell me that they, the admissions people there, they always do a thing where they cover up, why do you want to go to NYU? And if the answer is you could put in Columbia instead of NYU the answer, then that's not the person for NYU, right? Fraser Kelton: Yep. Yep. Nabeel Hyatt: know, it's, if it's, I love the opportunity for internships in the dynamic city and, you know, stuff like that. It's like, that's not really about NYU. That's about New York. so I think similarly when fundraising The thing I got to in the latter half of my third, fourth startup in fundraising was, um, I'm going to pitch the way I want to pitch, not the way my founder friends tell me to pitch.

46:47 And I'm going pitch in a way that is authentically me and the way that I want to talk about how I want to raise, run this company, the culture I want to build, the problems my startup has. I'm just going to lay it on the table authentically. And then the job isn't to find. 50 term sheets. The job is to find one or two term sheets. If I, if I can pitch the way I want to pitch my business I'll get lots of strong no's, but one strong yes. And lots of strong no's, but one or two or three strong yes's, it is ten times more valuable than a bunch of meh. This seemed okay, because those don't lead to board seats and checks and people who are going to join your cause for the next ten years.

47:29 Fraser Kelton: So having listened to that and then having a moment to reflect. The thing that I would do differently that I think would have a material impact. is to have a very authentic opening as to why I was excited to have this conversation with this specific person in this specific firm. You and I had the joy of sitting with that founder a couple of months ago now, who said, I'm excited at the prospect of working with Spark because you have a history of supporting founders doing brave things. And I know that that worked. On you and I because we independently said it with other people after the fact, Nabeel Hyatt: good Fraser Kelton: And it was gr it was great sales. It, but it was, well, just like any great sales, it was authentic and it, and it resonated, right?

48:13 Nabeel Hyatt: and you could feel it in the tone of when they were saying that, it was something they were really feeling. What do you do when you're pitching, X, Y, Z, fund? That you don't really know why, why you're talking to them. You just, you can't figure out the most amazing, there's no obvious amazing reason why you're talking to them in the first place. Fraser Kelton: Why are you wasting either of your party's time? Is the first one. If you can't put in 15 minutes of thought and research and come up with one reason, then why are you talking to that person? Nabeel Hyatt: There are people that like, there are CEOs that like to build very long spreadsheets of the 40 people that they're going to go through and talk to. And look, there are times where fundraising is really CEOs who It was the 38th person in the Excel spreadsheet that you got to that raised the round.

49:04 Personally, I have had rounds where I have had to do that in the past. But I think that is different from being casual about it. And I think that's what talking to. Long term relationships, these are big decisions for the person on the other side and they can feel it when when the work hasn't been put in. And so, I know for a lot of founders raising money can feel like a quote unquote distraction and I want to get back to quote unquote work. And I've always really hated that phrasing because You know, getting rid of a board member is like 10 times harder than getting divorced. Like, you're, you're recruiting somebody that is going to be you for a really long time.

49:49 Like, you should put the time and effort in the same way that you would to recruit a CTO or anybody else. Fraser Kelton: Oh, for sure, right? It is, it's a byproduct of COVID maybe where it became speed dating and the whole community went crazy, but the idea that you would sign up for this level of intense. Camaraderie without having a an investment seems rather silly. You know, I got good news for you. I got good news for you. Nabeel Hyatt: What's up? Fraser Kelton: I googled superpowered. Nabeel Hyatt: Mm hmm. Fraser Kelton: And I'm, I'm reading an article and we'll come back to the product, but I wanna, I want to, I wanna live with you. The company says they are not shutting down the initial product, Nabeel Hyatt: Yes!

50:34 Fraser Kelton: All right. Let's take a step back. Tell us about superpowered and why you're over the moon. Mm, Nabeel Hyatt: so, this is a great segue into product of the week I have been trying to record most of my life on a daily basis more and more of my life, and try and summarize it and make it searchable and so forth the super powered started out as kind of like meeting bot helper company actually prior to GPT and then post ChatGPT, they turn it into an AI note taker for your Zoom meetings or your Google Meet meetings and so on and so forth. Now if that sounds like 30 other startups, that is because there are like 30 other startups that are also aI note taking startups, folks like Fireflys, and I think Gong does this for salespeople.

51:26 And you could just go open the Zoom app store and take a look through. And by the way, Zoom itself has natively launched summarization as well to take notes while you're inside of your meetings. And so the question is, why am I excited about superpowered not dying when all these things exist? Everyone's going to launch a version of a product, and they're all going to be noisy, but the real question is, who's done it right? And at least in my personal view, I've tried all of these products. And none of them are good enough that I would ever use them week after week after week, except for SuperPowered. Fraser Kelton: What, what is super powered? Nabeel Hyatt: You mean, what does the product do?

52:07 Fraser Kelton: You just said that it's all of the small things that they've done right that make it stand out from a different AI, like transcription service. Like, isn't it, don't you just want it to do reliable transcription? Nabeel Hyatt: No. First of all, nobody wants to look at the transcription of any meeting. thEre is no way That I want to actually look through all of the ridiculous things that I talk about every single day, word by word. What you really want is summarization, and what you really want is action items. And the execution on that summarization and the execution on those action items is what matters. And it just turns out that there's actually wildly variant execution on that job.

52:51 The particularly two problems that I have With most of the other products that do summarization are first, they run inside of Zoom as an app. And I don't want Zoom having control over it. I want my desktop to have control over it. And so, SuperPowered is a desktop app, not a Zoom app. That's the first that matters a lot. Fraser Kelton: that's a big difference. Nabeel Hyatt: and it allows them, in particularly, to add new interfaces, new Chrome. They can iterate on it, like, 50 times faster. than trying to be one button on the toolbar on the bottom of, of Zoom. It also means a user doesn't have to ask corporate overlords whether they will approve this, this app to run their infrastructure.

53:34 Which I think is a real thing we've got to think about in AI. A quick aside, I was talking to my friend who works at Amazon and he said, you know, we talk in these podcasts of all these wonderful products and he goes, he just, he's like, just remind you what's going on in real life. Every time he even goes to use ChatGPT, Amazon internally puts up this big prompt that yells at him and says, listen, just so you know, if you put any confidential information into this product, we will come and kill you. He can't have things scraping his email to summarize them properly. You can't have products that are helping arrange meetings through AI, Amazon's not getting that stuff happen. They're on lockdown. Oh. And anyway, so, but super powered runs on your desktop, that's the first thing, and so I have control over its use not my corporate overlords.

54:20 And then the second thing and again, it kind of maybe goes back to this previous conversation about just understanding where we are on the entropy curve, is that they let you edit the prompts. So they have, they have meeting types. So for instance, if I'm meeting a new company, I have a meeting type called New Company. And then when I meet with the founders we work with, that's called Founders, right? And, and the notes I to take away, the takeaway from each of these types of meetings is remarkably different. And of course have prompts that they put in there that are starter prompts for the noob who doesn't know what they're doing. But inevitably, I probably for 90 percent of people today, Like, something's wrong about that. It says something in there that I want that's not right for me, and it lets you open up the prompt, edit the prompt, and get what you want out of it, which is the difference between something that is kind of like, meh and okay, and it like gave me a couple of interesting summarization topics and titles, versus I feel like an active participant in making this thing work.

55:20 Fraser Kelton: Don't you think that this is also maybe why they're pivoting away from it in the sense that This is a really hard problem, right? I was just thinking that the diversity of meetings that people have and then the preferences of workflows across those different types of meetings means that there's like an explosion in Quote unquote, getting this to work well. Nabeel Hyatt: I think there's two points there worth touching on. The first of which is that, look, of course it's a problem. It's a problem. That there are lots of different use cases in meetings, and it's a problem that this is a really busy market with lots of competition, so it's hard to stick out.

56:02 iF a startup doesn't want to solve problems, then what are they doing? Like, I, I do worry sometimes that we, we try and avoid all of the risk in our startups when problems existing out in the world is why startups have a chance to exist in the first place. So you have to pick your proper problems, but, Fraser Kelton: Huh. Nabeel Hyatt: but yeah, let's, but let's spend a little time figuring out if I'm a startup, how to solve this problem, because if we can solve it, then it's perfectly obvious for the 18 or 20 other AI meeting note companies that they have not solved this problem yet. So if I have a Fraser Kelton: Mm Nabeel Hyatt: then suddenly I can explain that breakthrough very clearly to my customers, and I now have an advantage in the market.

56:43 Fraser Kelton: Hmm. Nabeel Hyatt: So, Fraser Kelton: Good, good point. That's fair. Nabeel Hyatt: and then my second point is somewhat related, but I think it's also back to the entropy, de entropy thing. If you, if you honestly think that we are still at the point where we're trying to make all these AI products work, then just accept the idea that this is AI products are early adopter products for right now, and they will not be early adopter products, only idiots, crazy people, only crazy people like you or me might try and play with. An early adopter across every single vertical and horizontal, every single week, to see how all of these tools are developing. But early adopter doesn't just mean nerd in, in It means that for somebody, this problem is so acute Fraser Kelton: Mm hmm.

57:32 Nabeel Hyatt: will be an early adopter to try and figure out the solution. And that early adopter customer will help you find the solution with you if you give them the tools to work on it. And as an, as an example, you know, Adept, which is where an investor in, is an action, they create an action transformer model, and they've released a workflow tool for building your own little webpage navigator to take actions on a webpage and do little workflows. Now, the model itself is not the large model they'll be launching relatively soon, so it's an earlier model, and the workflow tool itself is, let's be honest, like, kind of hard to use, clearly an R& D product and definitely not a late adopter product that I would give my mother or father, right?

58:24 But Fraser Kelton: mm hmm, Nabeel Hyatt: for the people who, which Those workflows are really, really acute problems in their lives. They are going to trudge through it. And then you will learn with your customer versus in some R& D lab somewhere where your assumptions about your customer are wrong, which is the right way to build when you're early in a market. Fraser Kelton: mm hmm, the challenges that exist today are, you know, normal in terms of trying to figure out how to solve a large, meaningful problem and that it's a shame, uh, if that's the reason why a group who had a little bit of an edge on it is, is likely to not be investing too actively into trying to solve it.

59:08 Nabeel Hyatt: And look, we don't know the superpowered AI founders. I don't know where they are in funding or their traction or their progress or what excites them and gets them up in the morning. I'm just a consumer of the product . But all I know is that everybody here should go to SuperPowered AI and give them money so that they stay in business, so that I can keep using product. Fraser Kelton: You know, I just skimmed the article. It says that it's hard to differentiate in this type of a market to have sustained growth. thEy are profitable, and so they hope to find somebody who will just continue to run it. But they're pivoting to become an API provider for anybody to create a natural sounding voice based AI assistant.

59:50 Nabeel Hyatt: That is also a busy space of course, but, you know, it's also very possible that's just a problem that they are more excited about solving. And will, they will through the hard difficulties of that particular problem with more verve and with more passion than they have for meeting notes, which is fine. Fraser Kelton: you know, yeah, people don't often talk about that, right? Is you go through a pivot and you're doing it for a lot of You know, logical reasons, but you might either pivot into or away from an idea that you actually care deeply about. Nabeel Hyatt: Yeah, it's not a short road.

1:00:30 Yeah let's be done for today. I Fraser Kelton: do it Nabeel Hyatt: think we went through some good stuff. Go download SuperPowered. Give it a try. We'd love to hear from you on that product. I'll also add there's a couple other products that have launched that are allowing you to do live AI drawing if you wanna try one. Leonardo AI launched a pretty good live canvas feature where you can draw on one side through a prompt and it redraws it on the right hand side. If you want to give it a shot and then we will see you all. doing one next week, Fraser? Fraser Kelton: I don't know. Let's see. We'll see in the future. We, we're not sure. Nabeel Hyatt: We'll see you all in the future. Bye Fraser Kelton: See ya.