Create an AI Voice Character app only using AI in less than 40 min (Cursor, Next.js, Daily Bots)

0:00people are using AI to create voice character apps that are printing money if you are technical or you aren't today's episode is a mustat characters are the new apps and no one's talking about it we literally created a weatherman weatherman a fully working weatherman character app in this episode and you could do the same thing it might look like a toy but all the best startups look like toys enjoy this [Music] episode all right I got my main man Eric here he's been building a bunch of these voice chat apps think about it as like an audio chat gbt and I dragged him on here to give people a how to tutorial on

1:00how to do it yourself uh Eric thanks for coming on and what do you think people will learn if they make it through this video yeah thanks for having me Greg um I think first of all you find out just how awesome it is that we have this sort of these tools available and how quickly you can spin up something um that will kind of wow both yourself and and your friends I remember when I first started playing around with this stuff um to build like voice agents and voice voice Ai and all that it like you experienced chat BT for the first time and you're

1:36like holy this is cool but then you talk to chat gbt and and it's almost like this 10x Improvement in experience um I'd say I'd say yeah I'd say you just kind of learn just how exciting all this stuff is and how easy it is and I I'll say that you know um if you don't know anything about coding you know there are things that you have to kind of pick up along the way like what is GitHub what does cloning a repo mean what does it mean to deploy something but uh what I want to focus on for this kind of session is just like the cool

2:14stuff how do we get this spot running uh how do we configure its personality um all that so I have everything kind of set up beforehand but beautiful uh I'm sure we can provide links and resources to kind of help along with the actual like yeah setup of things let's let's let's dive in let's get right into it sounds good I'll just share my screen and by the way while you're pulling that up I think my goal for everyone is build your own ideas like you're going to watch this and you're going to get a bunch of ideas around what sort of voice sort of apps could I

2:53create and Eric has been creating a few of these they've been going viral on social so if he can do it you can do it too too so let's let's dive in awesome we're going in all right uh so first things first I think I should give a quick overview of like just the tools what I'm what I'm using what I'm familiar with um I'm sure most people watching are familiar with kind of the AI landscape in terms of things like cursor so this is my code editor cursor hooked up with you know all the cool um llms to help with coding um and then the main kind of star

3:36of the show here that is going to be really doing all the heavy lifting in terms of the voice in terms of the back end um is this company called daily and Daily's really cool I actually just first heard of them I think this year when they announced um not daily Bots actually daily Bots was something that they announc a bit after um but it was through this other Library called pip cats and pip cat was a library more focused on devs but just kind of make it a lot easier to interface with AI and allow you to kind of um build voice assistance but what

4:16daily Bots when they came out with this I was like oh this is going to be kind of gamechanging because now you don't even have to handle all the llm calls and stuff with Pip cat daily is doing that themselves and so all you do is you know sign up you go on to this dashboard and you get this API key and then there's this repo here where they have a demo up it just runs in the browser um and I have that set up over here I just clone the repo uh which basically just means you download it to your system open it up and you get this you get this demo

4:56screen uh now what you have to do from here is connect the demo to the actual daily backend over here with this API key this is sort of your your key to uh unlock the daily Services um so you put it in this m. local file here um I'm realizing that this is a uh key that I'm showing uh so I'll have to you know regenerate it later but it's okay um but once that is all set up realize that right now like no code has been written all all that's been done is just you're just downloading this repository you're signing up for this service you're getting a key you're

5:42pasting it in this file uh and then you're running it with uh one command over here just yarn Dev that basically just starts up the server you're heading to this little local post URL and then bam you're over here um shh don't tell anyone but I've got 30 plus startup ideas that could make you millions and I'm giving them away for free these aren't just random guesses they're validated Concepts from entrepreneurs who built $100 million plus businesses I've compiled them into one simple database compiled from hundreds of conversations I've had on my podcast but the main thing is most of these ideas don't need a single investor some cost nothing to

6:42start I'm pretty much handing you a cheat sheet the idea bank is your startup shortcut just click below to get access your next cash flowing business is waiting for you so let's try it out I wonder how this is going to go with the whole like audio thing but we can just start with the defaults and see what happens just testing if it works um take some time

7:22okay so let's see um I'm sure you saw the like audio wave form appearing Greg but did you you didn't hear anything right because I heard it through my airpods yeah I didn't hear anything okay so let's try let's try this hello I am example but your assistant I can help answer questions and provide information on a wide range of topics what would you like to talk about do you hear that yeah I heard that I heard that should I answer it no I just ended the call I just wanted to test the oh wow that was okay so what did I just listen to okay

8:04awesome cool we got audio working with the okay awesome you you just listen to the bot The Voice bot now um that was an llm response so daily on their back end was calling um the llm that we specified which is over here uh together AI they're provider a bunch of AI models um the one that we're using is llama 3.1 70b and the place that it's getting you know the how to respond kind of information from is this configuration file so the whole point of this daily bot thing is to take away all the complexity of like you know the actual hard AI stuff and

8:52then just put in front of you like super easy um configuration tools to change like how you want the bot to behave um and then the voice right voice is really important it would kind of suck if you couldn't change the voice um along with along with this is really cool um function calling tools that you can give to the bot to actually do stuff um and by stuff I mean like system stuff like computer stuff calling apis getting data um making an Instagram post whatever computers do already on the internet you can get an AI voice but to do um and even better to respond like to commands that you just

9:44tell it like hey you know what's the weather and then give you the weather now we're going to kind of get into how all that is set up you know later on but just kind of like putting it out there um this is the configuration file you can see over here there's a bunch of characters that they kind of put in uh by default with these prompts and whenever I'm working with AI stuff I'm always like the first thing that I think to myself is like show me the prompt um I want to know what exactly is going into the llm and so the default is this it's like you are an

10:19assistant called example poot you can ask me anything and then some instructions on how to speak so that the audio comes out corre correctly cuz what's happening uh on the back end is essentially the daily bot system is taking in the audio from the microphone and then it is converting that audio into text using something called a stt or speech to text provider and then that text is being piped into the llm the AI model the AI model does its thing it outputs its response as text and then daily takes that text and then pipes it into a TTS so a text to speech provider uh and then with that speech that it

11:09gets it's piping it out to your browser and so it's this whole system that it's doing um and there's a lot of complexity to that there's a lot of you know things to be aware of in terms of performance in terms of like you know how to handle errors all those things um which would be nightmare if you are developing that yourself but of course you know we're in the age of AI now and good tools and DA's handling all that and so we can just sort of focus on hey how what should our assistant do is it even assistant at all maybe it's uh you

11:46know they've got some pretty cool characters here theyve got a jenzy middle schooler they got a skateboard meme guy um and I think what I kind of did beforehand is configure the function calling the tool calling um just a simple weather man so I configured the function call the tool to get the the weather um and uh display it you know talk about it Etc so these preset characters so these are pre-built characters if you wanted to build your own character could you do that oh yeah no you just edit the file you just you just add just your character in so you can see here um the character selection

12:33screen over here is a is a direct one to one with this file so as soon as we edit like for example I can demonstrate it we have default we can just name this like Eric right and then refresh the default character should be name Eric so that's proof oh cool it's just it's just the file yeah right and that's why this gets really interesting and and for people listening it's I think that if you think of characters as the new apps then there's a huge opportunity and not many people are talking about this around just ideating on characters figuring out what's going to work and

13:18launching them yeah that's actually really good point I had even really thought that too deeply but I think that kind of idea is bouncing around a lot um um on the internet especially one thing that I can think of um Nikita Nikita beer which I'm sure most people are aware of at this point that are into like the tech scene he recently tweeted out like you know social apps are kind of dead after iOS what like 18 or yeah iOS 18 the contact sync is um very much restricted and what Nikita said in that tweet was that uh initially he thought like AI character chat apps were kind of

14:02a fad but after seeing like a bunch of the retention graphs he's like okay this is not going away and people really like talking to these characters um and that the kind of future of Highly retentive engaging consumer apps and you know non-consumer apps too will probably have something to do with you know creating a character that does something acts in a certain way totally yeah and I think it's B2B and B Toc I think it's just going to be the way that we're going to interact with content and this is just a new form of content where where else do you want to take this so let me demonstrate the function

14:45calling and kind of Step through that because I was um that's what the prep this morning was was about uh before the call I kind of just wanted to make sure that because I haven't actually I hadn't actually implemented function calling into um actually at all I haven't really made use of it but especially in the case of like the voice agent um that's something I never done before so I want to kind of Step through myself and figure it out um ran into some snags that I'm I'm glad I caught myself on before the before the call I can just kind of streamline through how this all

15:17works um I'll kind of Step through like the actual documentation so maybe viewers can kind of get a sense of like oh you know you're never going to be able to figure out everything on the first go you're going to want to develop this habit this practice of being able to read documentation um understand it and sort of apply what it's what it's you know saying sure but also now in the age of AI it's like if you don't understand something ask chbt ask Claude um and so here we are on the daily Bots documentation um you can check it out yourself just about you know general

15:56information on how it's all working we're interested in function calling so what exactly is function calling you know people talk about it a lot um basically you have an llm you have an AI that takes in text and outputs text how the hell can it do anything except output text well short answer is it can't do anything other than output text you need to build the system around it you know to actually take the text that it outputs with and do stuff with it now what llms can do if they're sufficiently smart enough is follow instructions really well um and so function calling is essentially you having an llm you having

16:39an AI model that is smart enough to understand what a function is and what a function call is and like what a response is and kind of follow all those instructions to Output an answer that your system around it can then take and do something with it so the LM isn't actually doing anything the LM is always just outputting text but the system you build around it can take that output and do whatever you want um that's a kind of core Insight here I I think it can get confusing when people talk about AI um AI can't you know AI is often referred to as like talked about this like

17:15blackbox system thing and a lot of it is a blackbox but I think we can gain a lot of insight in s of understanding like how things actually work so this kind of provides some information about you know what goes on in terms of function calling and this was what I was kind of focused on is adding it to the Daily part and so my code changes here uh very minimal are basically just adding this weather tool so over here in config I have um set up just a little bit of code to Define um the tool to the llm name is really important we'll kind

17:57of get to that later but we want to make sure that our names are align and we're sure of what we're naming the tool um and then let's see we want to obviously describe you know what kind of parameters we give to the tool so parameters aren't something that is necessary um but probably you want because you know if you're going to do like to get if if you want to get the weather you want to get the weather for a given location right and so um specifying to the AI that oh you know you can pass in a location is pretty important um and then all this stuff

18:39that I've done is basically um following the instructions here so that the AI model on the other hand uh on the other end knows what's going on with all these functions so the AI model is sort of like okay I have access to these functions um this is the format that I need to respond with and this is really important this is really important that the AI does respond with this format because what daily is doing on the other end is waiting for a message that you know starts with angle bracket function and ends with that and and that's how daily understands oh okay the AI model

19:21is trying to get the weather and so over here in page. TSX I'm essentially defining you know what actually happens when the AI calls AKA outputs you know I want to call get weather and this is where actually the magic happens in terms of the function calling we can see get uncore weather is aligned is a same as get underscore weather it's very important um if they're not online you can run into issues where the AI is calling a function that doesn't exist you know and so this is kind of the alignment point and then this is what we're actually doing you know the system the server is doing in terms of um like

20:05the code it's it's calling this route uh API weather so where the hell is that well you know there's a folder called API here um I guess before we started uh we didn't even specify that we're working in nextjs but you know nextjs is essentially this framework that allows you to set up a server that runs um react but also allows you to specify Roots uh pages that uh run code and so over here you can see there's a directory called API there's a directory called weather um the names lining up with here are not a coincidence over here in the route. TS file uh we are

20:52specifying what actually happens when um we're fetching that route and so I'll just put it up here now if you read through this closely you'll see that um we're not actually getting the weather we're we're specifying like the function liter called get nonsense weather was specifying a bunch of these things uh the reason why I I did that was one because didn't have a lot of time to to prep but also I think it's going to be pretty funny um to actually get some kind of you know you I'm gonna read I'm gonna read a couple of these you know conditions and stuff like that just cuz

21:31some people won't be able to see it on on their screen so Sunny you know you got Sunny rainy cloudy peculiar Whimsical flying pigs flying pigs good place so this is actually kind of good that we specifying these kinds of more wacky sayy things because a potential thing that could happen right is what are we telling the AI to do we're tell them to be a weather man right and so let's say like you know the function calling didn't work and the AI doesn't actually like do the code like execute the code that we tell her to well we're still telling them to be a weather man and so most likely what the

22:14AI is going to do is hallucinate some kind of you know weather report and so what this actually in hindsight um I realize is like oh because we're specifying like nonsense phenomena like flying pigs cats and dogs if the AI says those things we can be sure that it's it's working um we can also be sure that it's working by looking at the logs here and just making sure that um nextjs is telling us that the server is accessing the API weather route um but you know that's bit a bit more boring than the AI saying flying pigs and realizing oh yeah it's actually getting that that thing

22:55there's another reason why it's important to specify which is you know I think about it as like Tik Tock moments it's like how do you reverse engineer someone screen capturing recording your product and if someone uses this product and it says it's rain you know raining pigs or singing rainbows like that is that could go viral your character could go viral so as you're designing your character and thinking about you know the Ultimate Experience that you want to create I think it is important to add constraints and try to think about what's the type of thing that will get you free distribution oh yeah and you're you're

23:40the king of that so I know you know that I mean I want to say King I'm working on I'm working on a Tik Tok I promise you um okay so let's let's let's get some zy weather reports shall we let's do it let's do it I can't believe we're making a weather man this is like insane and by the way it's like a mid hurricane right here in Miami so this is like perfect it's not it's not looking good well it's not really looking good here in in New York either it's bit bit cloudy but at least there there's no hurricane um hope you're staying

24:13safe okay so I have it loaded up on my phone and um I guess it will be I think I can like show my phone on the camera see and we'll zoom in yeah quick ad break let me tell you about a business I invested in it's called boring marketing.com so a few years ago I met this group of people that were some of the best SEO experts in the world they were behind getting some of the biggest companies found on Google and the secret sauce is they've got a set of technology and AI that could help you outrank your competition so for my own businesses I wanted that I

25:01didn't want to have to rely on Mark Zuckerberg I didn't want to depend on ads to drive customers to my businesses I wanted to rank high in Google that's why I like SEO and that's why I use boring marketing.com and that's why I invested in it they're so confident in their approach that they offer a 30-day Sprint with 100% money back guarantee who does that nowadays so check it out highly recommend boring marketing.com try to get as close to the mic as possible um and so on my phone I've I'm accessing the same page that is running on my computer and so this is um running the

25:39code that is also on my screen right now and so you can see when I go to the character kind of selection screen we have Eric there that I edited and then all the way at the bottom we have the weather man so let's talk to the weather man hey uh I am located in New York City can you give me the weather for New York City folks it's your lucky day I've got the forecast for the Big Apple and it's looking like a real tree we've got Cloudy Skies but don't let that fool you there's a chance of whimsical singing rainbows that's right you heard me right

26:16it's going to be a day of magic and wonder in New York city so grab your umbrella and your best singing voice and get ready to take on the day that is so good how good is that yeah that's awesome validation we got it we got it says singing singing rainbows that's not a that's a viable weather thing but it's tot that's that's working yeah I mean the reality I I honestly thought that it was gonna suck that's the I thought it was going to be like kind of bad and it would it that was really really good yeah it's it's pretty cool so I think uh viewers watching you would have

26:54seen actually on on my screen too that we got the log here um what this log is telling us is that the system the server actually did access API weather and provided this location and so everything is working sort of as as we wanted to which is super exciting it means then we're we're having we have this platform kind of to just like build off of and make tiny changes and I think that's kind of uh thing I want to emphasize in building anything is you want to especially when you're starting out make sure that your changes that you're making to things or when you're building

27:33something are small and like understandable um I I feel like in this new world of AI that that we're in um there's so much like hype around one-shotting things like oh I just sent one prompt to chat gbt and built this whole game like that's cool but you're never going to learn anything by just like telling it to build you a whole entire app and then like you have no idea how old the files kind of like rout together um and so I think it's really helpful to kind of build off of something that already exists which is why I didn't bother trying to integrate daily Bots myself but instead just

28:10literally took the repo that um they put up as a demo and just adding onto it um so yeah important stuff and and you can turn this you know that weatherman app into an IOS app you know it doesn't necessarily need to be web um so that's totally possible um is there anything else before is there anything else you want to talk about here um I think the last thing I want to do is just show them your app because your app is basically like to me it's like a fin like it's what you can actually end up creating that's like the finished product um but

28:48is there anything else in terms of the tutorial that you think it's really important people know I mean I guess we could go into like deployment yeah let's yeah actually getting it live cuz it's it's it's live now I've already I've already deployed but I can go through like how to do that and the platform so um nextjs let's just start with nextjs nextjs is this framework that uses react which is this way to build websites and nextjs was created by a company called versel verel really cool company they do a lot of programming stuff what versel also has done is built this platform that allows you to take your nextjs app which

29:33you can develop for free because it's an open source framework and host it online on the web and so they make it super easy um you can type in one command and your app is going to be live on the internet um I think we can just kind of provide links in the description in terms of like the documentation to actually install um the stuff that you need but essentially uh Vel has this tool the CLI command line interface tool um that allows you to kind of use the command line to um do things on the versel platform and so what I did to get my uh

30:27daily bot demo with my weatherman live is literally just Vel um actually yeah that's it that's literally you just type you just type Vel obviously way just you got to like log in so that it connects to your Vel account and everything but you log in you type Vel um and right deploy yeah learn how to deploy type you type this out and it deploys anyone could do that yeah I've I've already done that so it should I think it's going to just trigger another deployment but essentially it's just going to go through this step and I can show you actually uh what's going on on on CES

31:13and over here deployment building um so I guess it usually takes like a minute and a half um but where they deploy it is this link here and [Music] so yeah I could let me just leave this up I'll just leave this up um yeah so people can check out yeah check we'll put it we'll put it in the YouTube description um so people can check it out yeah but it will be it'll be deployed and oh it actually took much uh much less time um and essentially yeah just going to that uh URL on any device we'll pull it up I can pull it up here and you can see

32:06it's right there so that's the point it's you text it to your friend they can try it out that's cool that's awesome okay and you know can you can you show us what kind of a final version would be which is in my opinion like one of your apps um could you show us maybe a video or or a version of what it looks like I guess I can start with um one of the apps that I've worked on that I'm really excited about that uses all this kind of voice stuff is an app that allows you to basically have a FaceTime call uh with a

32:47vtuber so I don't know if a lot of people are yet familiar with what a VBA is but essentially the people that create content a lot of them stream on twit twitch and YouTube but they don't stream using their real face they use software to essentially um embody an avatar and I'd say 99% of these avatars these YouTubers embody anime characters um because the whole kind of technology and Trend started in Japan um and this just kind of the way the culture has gone people love anime and these vtubers are crazy popular they have like millions of followers they're making millions of dollars a year um from subscriptions

33:39from selling merch Etc and I had this idea of like oh wait I can use all these cool new tools coming out these um voice apis daily Etc to create an app where you can have your own personal face on call with a vtuber because when you're watching a vicho like they're they're an influencer they're they're someone you're watching they're Creator you can't have like a personal call with them and so I I realize that with AI and all this all this cool stuff you can you can build that and so um this why I created um yeah so this is mji in terms of the

34:16kind of going to exactly like did I draw her no I bought this model um on a website called nzima nzima is a Marketplace for these kinds of anime models there's a whole cottage industry of people rigging um illustrating these models for vber streamers to use but I bought one not to stream with like as myself but to put in this app and so what this app is using is um not exactly daily Bots um I'm actually using pip cat that I showed earlier because it gives me more control as a Dev and so that's kind of I guess going into a bit more of like oh you know once you learn a bit

34:58more about how these systems work you are going to want to use tools that allow you to have more control over um how data is passed around and whatnot but I can enter the call right now and we can show people yeah let's talk let's talk to her yeah okay so hopefully this I think this has been pretty popular lately it's called Uh Coco Cloud can you see it it's it's blue wow it looks so pretty yeah can you see what what other color is there it's blue and one one other color can you see I see a little bit of white swirl too yes yes you got it the white SCE

35:44okay so we're going to try it and I'm going to give you some to to try it and I want you to let me know what what you think okay okay I'm so excited all right she excitedly takes a small sip of this morning and her eyes widen she favors the taste before giggling so yummy she exclaimed it's so sweet and creamy thank you for narrating that it's actually pretty good it's actually pretty good I like it you're welcome I like it to how would you rate it out of

36:2410 and that's what you guys can create that's what you can create that's the end result that and and that that's just so cool that is so cool how long how long does well first of all did you build that yourself uh yeah yeah well I mean it's kind of you know did I write every line of code that is running on the system no but in terms of like the actual app yeah and the sort of like the server architecture and all that amazing so that's the thing you know a few years ago well first of all it wasn't this wasn't even possible a few

37:01years ago but you would need a large team to do that raise Venture Capital here you are you know in a room wearing a hoodie uh throwing out apps characters you love to see it um Eric this has been a blast if people like this there's a few things that you got to do one is like it on YouTube so more people see this um like it like like like uh and the second is comment I think I just would love to see uh what you what you thought of this do you have any ideas around this um Eric and I will be in the comments and we will be replying um and

37:39then subscribe you know I'm not going to continue making these videos unless I see that subscribe button go up uh subscribe but number go up so don't forget to do that Eric where could people learn more about you and and the apps that you're building characters You're Building I should say just follow me on on Twitter sx.com the everything app um the link will be I guess in the description but yeah that's that's where I'm posting all the unfiltered takes and and progress updates and stuff love it all right we'll have to catch you catch you around awesome thanks so much for having you Greg of course anytime later