# Building Real-World LLM Products with Fine-Tuning and More with Hamel Husain - 694

Channel: The TWIML AI Podcast with Sam Charrington
Video: https://www.youtube.com/watch?v=sGqEKzJYrNE
Duration: 1 hr 20 min
Language: English
Words: 12986
Transcript page: https://viewrankai.com/tools/youtube-transcript/sGqEKzJYrNE

---

[0:00] so evals I think okay so evals is the most important thing in the Consulting work that I do people call me when their AI doesn't work it's like hey like we've kind of prototyped something we're excited about it but it's like not really working people don't know how to get unstuck and so that's where this notion of evals comes in you want a systematic way to test the efficacy of your system it's really the only way you can build AI so it's it's like integral to the building process of AI products [Music]

[0:36] [Music] all right everyone welcome to another episode of the twiml AI podcast I am of course your host Sam charington and today I'm joined by haml Hussein haml is founder of parlance labs before we get going be sure to take a moment to hit that subscribe button wherever you're listening to Today's Show hemo welcome back to the pod yeah thanks for having me back it's been a while it's great chatting a while likewise likewise uh about two three years end of 21 I think something like that yeah um you were deep into mlops I think we spent a lot of time talking about that we spent a lot

[1:22] of time talking about uh notebooks and literate programming and fast AI tooling and a bunch of interesting topics but I wanted to get you back on the show because you've been quite prolific on llms and the way folks can take advantage of fine tuning and eval um lots of interesting stuff I wanted to dig in with you so yeah looking forward to that were you at GitHub the last time we spoke uh yeah I think I was at GitHub yeah GitHub or just leaving and you went to work with um outer bounds outer bounds for a bit what have you been up to since yeah I

[2:02] decided to do you know my own consulting company just because it's such an exciting time to be working with large language models and it's a really exciting time to like see lots of different use cases with this kind of new technology um and work with lots of different companies and so I've been advising lots of companies and helping them build AI products awesome awesome are there any use cases that jump out at you as novel under the radar you know without kind of divulging your client's you know IP or anything like that like I sometimes think that you know a yes llms are very exciting but a lot of the use cases are very

[2:51] similar and I'm always looking out for kind of Novel use cases where you know it's like surprising that that would actually work does anything jump out at you when you you hear that you know um I wouldn't say there's any novel or use cases that surprise me anymore I think I have gotten over the stage of being surprised okay um but I will say that the things that seem to work best are user interfaces that have humans in the loop in a very thoughtful way so instead of just having let's say a chat bot having some kind of interface that allows folks to interact more richly

[3:35] with the software so for example one of my clients is this real estate CRM company called ReChat and they have a chat interface however whenever you interact with the chat interface it pulls up different widgets so if you try to search for a listing in the chat interface it'll pull up sort of a like a listing you know you say Okay I want to find properties that are you know greater than 2 million or less than $2 million with such and such and the ZIP code or whatever your search is it'll pull up a little widget with the results that you can refine um so it

[4:16] actually brings the user interface to the or it's it's a very thoughtful sort of interaction between the software and a human rather than this like naive let's just push the chat interface um you know to the user um and so like things like that make a lot of sense because it allows people to work with AI when they should be working with AI and not work with AI when it doesn't make sense to work with AI um and like inter you know kind of inject the AI in the workflow in a very thoughtful way and it takes a lot of work to do that from a design

[4:59] perspective lot of people just want chat GPT for their software right but that's kind of the lazy thing to do the thing that takes a lot more work is like hey let me thoughtfully figure out how to integrate it into into the user's workflow so wherever that app yeah I really appreciate that take you know I see as well vendors trying to Rebrand everything as agents and um I don't necessarily want to be forced into uh dialogue box every time I want to interact with a piece of software like we've spent a long time refining the user experence for different types of software and it all doesn't look you

[5:40] know the same way um and I'd hate to think that we have to throw all that away in order to incorporate the intelligence that we can get from an llm uh what your what Your describing reminds me of a I saw I read some blog post a while ago that coined this is generative UI that was trying to like to some degree dynamically um kind of push UI elements into a chat framework have you seen that kind of take root more broadly among the folks that you're you're working with yeah and what I suspect it's like there's there's many purposes of this kind of ux it's like one is okay

[6:24] obviously to help people do their work a lot faster but also like to educate the user so that you know you want to use the same UI elements throughout your product for example so when someone sees a listing finder they they know when next time they see a listing finder in another place even without AI they're like they know how to use it because they already seen it before they kind of been walk like walk through it and people can choose to like do whatever they want like sometimes you don't need AI sometimes you just want to you know look around or whatever so like it's it's an

[7:01] important like onboarding thing um and so there's some deep kind of design thinking that I think you know needs to be done on a lot of these AI products the thing that I wanted to really dig into with you uh or at least a really good starting place is uh fine tuning you recently ran what I think you initially intended to be a course on fine-tuning and it turned into a virtual conference uh with a ton of speakers and uh looked like a lot of really interesting and good information but that course I think grew out of your belief that people think of fine tuning as this thing that's like really complex

[7:49] and hard to do and get right but it actually doesn't necessarily need to be uh all that complex is that right no it doesn't need to be complex and so the thing that I really like about LM fine tuning is that it's it's very easy uh from like the mechanics of it you know it's actually easy to actually apply the fine tuning itself so unlike other kinds of deep learning that has come before it where you kind of have to fiddle a lot with hyperparameters and architectures and stuff like that when you narrow it down to large language models there's a like really good tooling and like it's also fairly easy

[8:31] to F tune from a machine learning perspective like especially if you're doing something like Aura it's kind of hard to screw up the fine tuning you kind of have to try to screw it up if you can just that's probably going to be surprising for a lot of folks but before we dig into the details there we probably should take a step back and like answer you know the bigger question which is like when should you be thinking about fine tuning um because I think you know at some point The Narrative was like everyone wanted to F tune and like the people in the know were like don't

[9:12] do it like you know for sure 99.99% you unless you're Google or Facebook or whatever you shouldn't you probably shouldn't be fine-tuning and you know you're saying now it's uh you know it's easy you know it's not as easy like part of it is easy say the whole thing yeah say the part of it is easy you know so I I think it still leaves this space of like does that mean everyone should be doing it now or like who should be doing it and when and also you know what should the expectation be you know when you fine tune you know versus use other techniques like what are you what is

[9:53] fine tuning really getting you so really I I'm wanting to like take a step back and introspect your you know thinking about fine tuning in its role and place so most people should probably shouldn't fine tune especially when you're starting out you should try to use a model off the shelf if you can use open AI use anthropic and start building stuff um you don't want to get nerd sniped by this whole exercise of fine-tuning because like you need to like have have a reason to fine tune and like you need to have some failure mode that you are experiencing that uh necessitates fine-tuning when you fine tune okay

[10:32] there's many different answers for this so one answer is you want your own model of some kind and that that model is us usually an open weights model let's say you know llama mistal whatever and that open weights model off the shelf doesn't quite do what you want it to do or it's not powerful enough you know to accomplish whatever you want to do on your domain and in the situation where you so like yeah in in that situation maybe fine-tuning could be the right fit for you with a caveat of you know fine-tuning really shines when you have like really narrow use cases like you're

[11:23] trying to have a model do do something very specific or the the LM do something very specific if your LM is just a general purpose chatbot that can do anything for anybody then that's not you know fine-tuning is going to be a lot more difficult and um it's going to be a lot harder to evaluate that and so you know I would say okay like you know if you're like a narrow scoped problem that's doing a very specific thing then fine-tuning is more likely to work the canonical use case that comes up all the time with fine-tuning is like text to SQL exactly that's the best that's

[12:06] one of the best use cases in fact one of my clients is Honeycomb honeycomb has an observability platform and they have a query language called hql a honeycomb query language so it's kind of like it's not SQL it's a domain specific language so they have you know a natural language to hql sort of uh product but that is such a narrow task that fine tuning works really well for that so like there's a lot of Dimensions to like when fine tuning would work for you one is like maybe you want some kind of data privacy you want your own private model and you want to make that model a bit

[12:45] better and if you have like that narrow scoped use case okay that might be like some Sweet Spot another reason why you might think about fine-tuning also has to do with open weights models is if you want to try to have a small model like a very very small very cost effective model that you can deploy in your infrastructure in a flexible way if it's like maybe even if it's small enough you can try to deploy it you know on mobile devices and things like that so um that's another reason where you where you might consider fine tuning is like especially with those like smaller models you might want to try to

[13:27] specialize them to get them to do really well on that task to kind of you know raise their performance level to try to approach the ones of the bigger models that are not fine-tuned on your domain but there's a trade-off like fine tuning isn't free even though fine-tuning is easy from the perspective of like there's a lot of tools out there I mean you can even you know fine-tune open AI um with their API which is which is a really St forward process to go through once you start fine tuning now you have to manage that whole process a whole process that you have to think about of continuously

[14:10] training that model evaluating that model for the purposes of fine-tuning curating data um you know debugging things and once you kind of commit to fine tuning also like you sort of start to specialize your model a bit on that that domain data so like you have to constantly sort of manage that because as your product is is evolving and you know changing over time you have to co-evolve your fine-tuning with that so it's not free it's actually can become very uh expensive in that sense so so that's why the narrow use cases mitigate that like if you have a really narrow use case then like okay if it's not

[14:56] especially if it's stable it's not changing that much like the the honeycomb query example it's like okay honeycomb query language is not really changing at all and you know if they want to tweak it over time it'll be like some incremental tweaks it's not like this insanely shifting product whereas like okay if you want a general purpose chat bot to do anything on Salesforce MH then I don't know because you know just you know maybe Salesforce is like changing and adding new things constantly it's a really big surface area and like all this stuff so so that's kind of some things to think about um of course you can try to in

[15:37] there about uh about powerful enough and and even this idea of uh using a smaller model that without fine-tuning wouldn't have the capability that you need what should expectations there be like are you you know does fine tuning allow you to take take a you know a 3 billion parameter model and get you know gpg 40 level performance like how should someone think about what you mean when you say make a smaller model more capable yeah so you can think of it like this like the honeycomb example where going from natural language to Honey to Honeycomb query you can almost think of it as like that's almost trivial for GPT

[16:27] 4 in a way that's you know um gp4 is like kind of maybe overkill for that in one sense that's like one side of the equation and another side of the equ is like hey it's such a very it's a very very very scoped task and so you might not need all of this like reasoning capability and so you can raise the bar so in this situation we were able to uh like fine-tune a 7 billion parameter model to outperform like GPT 3.5 we didn't Benchmark against GPT 4 but it's pretty good and so that's a that's a big win if you know now for example they can offer that to their

[17:10] Enterprise clients and you know they don't have to go through this like complicated data privacy situation and like sock to compliance and all this stuff where have to like shift data around and all that so you can um approach the quality of bigger models but you have to keep in mind of this like narrow tasks thing and so a lot of people don't real like the narrow tasting is a big sort of elephant in the room because people are not good at scoping narrow tasks right right it's not like you know you know you have this this list of potential use cases for fine-tuning that is narrow domain or you

[17:54] want a smaller model or you want privacy narrow domain is is kind of a precondition that you need to meet in order to like if you don't have a narrow domain then you're probably going to have a problem you know even if you just want a smaller model like is that so so there's a spectrum yeah there's a spectrum so I'm I'm talking about like most people when I say this I'm like generalizing and saying like hey like you should have a narrow domain um you know that's a sweet spot like when you have a narrow domain and you want you want data privacy um and you want a smaller model

[18:35] and you want the freedom to like you know use it wherever then you have like lots of reasons to do fine tuning you have lots of reasons and like one reason may not be defensible enough depending on what happens like if it's just data privacy you're like H like I don't trust open AI um you know it might become a day where you do trust open AI so like you know um now there's a spectrum between like fine-tuning and continued pre-training you know so yeahor on that yeah there are I think they're used interchangeably quite a lot kind of yeah so the idea is like they

[19:14] really like okay pre-training and fine tuning are really the same procedure like you're you're Contin you're continue the training process with more data the idea of like continued pre-training is in addition to like domain specific data you like mix in this more generalized data like from the web and so on and so forth the idea being you don't want the model to lose its like General capabilities you don't want it to over specialize in your domain okay um you want to like retain those skills no want to get rusty just you know on those things got it okay and so there's like really competitive fine tunes out there that people have done

[20:02] exactly this and uh organizations like news research have put out a lot of fine tunes like this where they've trained on lots of different kinds of uh heterogeneous data sets from different domains like at the same time like they mixed it together and they've curated the data sets they cleaned them up to to like raise the performance generally of these models on a b against a battery of different benchmarks so that is a thing um and you may you maybe want to do that but you know from a commercial perspective I would say for the most part you know you want to fine tune on a narrow task I I I think it's

[20:48] maybe important to highlight what you just said from the lens of you know let's say you've you've checked all the boxes in terms of you know your use cases and the applicability for using fine-tuning I think the conventional thinking is you use an off-the-shelf model that someone provides you or you fine tuna yourself but you you also kind of highlighted the idea that there are other people that are out there publishing fine-tune models and so if you can find one of those that meets your needs you might not have to do the fine tuning yourself yeah so I would say like one try to not do work yourself

[21:32] wherever possible I think that is like a good principle in life it's like one use open AI if you can use anthropic if you can um and go as far as you can with prompting you know like don't find tune and try to do things like including examples in the prompt so that's referred to as F shot prompting it's also like a little bit more Advanced Techniques like dynamically including examples in the prompt that are related to the user input so it's kind of like if you think about rag for examples you know things like that like try to push that as far as you can before going to

[22:10] fine-tuning and then also even if you do need to go with an open weights model try to find an open weights model that's already fine-tuned on that domain and then you might want to just start from and if you do want to F tune maybe you start with those models like the one that is as close to your domain as possible you just maybe get a bit of a head start instead of just going from just starting with the blank slate or starting from like the base model and are you just looking on hugging face for those models or is there some other repository that people are using yeah

[22:43] looking on hugging face and like roughly bookmarking things on Twitter and hoping that I remember you know um there's there's a leaderboard there's some leaderboards you have to take all these leaderboards the grain of salt um cuz benchmarks are kind of noisy and people you know there's a lot of data leakage from the benchmarks into the training um so you have to kind of ask around and dig into it to see like what you can trust you kind of have to use it yourself also um but yeah like starting from there and I would say like in a shrinking number of cases it is the case that you know as models get much

[23:26] better um you know like the business case for fine-tuning Keeps like shrinking more and more but it's still there like the you know uh it's still there and you know you can it doesn't need to be complicated so in the easiest way to F tune is like use open AI you can fine-tune open AI models um and you're essentially just giving it a list of examples there yeah you just give it a list of examples and you can start there um it's it's a good way to try it at least like get a sense and you know you can kind of iterate from there and see kind of what works

[24:04] for you um but it is really fun and Beyond fun it is like very valuable to a business if you can you know when you have it's a slam dunk when you have a small Model A narrow use case and that small model really knocks it out of the park and you can just hand it to a company it's like this is your model and they can just deploy it um that that can often unlock lots of value um so it's like you know it's kind of like a spectrum it's not like a good let's say flowchart I have but it's kind of like as many you know when you start

[24:46] to stack up all of these reasons Yeah the more you have then the more it kind of tilts a favor of fine tuning but yeah the data privacy one I suspect I mean you know in the early days of cloud people were very suspect of putting their data on the cloud and now no one really bats an eye on like you know uh about the cloud I suspect the same thing might happen with large language models um so I don't know if the data privacy concern is going to you know endure but certainly the other ones might um and so yeah we'll see mhm so you figured out your kind of

[25:33] business case for doing fine-tuning and now Hamill's here to tell you it's not as hard as you were told what does that process look like what is the first thing that you need to do when you are committed to going down that path and you're you know past the doing it via an API level yeah so there's some really good tools out there the for the help you find tune my favorite tool in sort of Open Source land is Axel and I really like yeah it's called Axel um and it's made by this guy wing wing lean pretty pretty cool guy he you know so basically what it is it's

[26:24] like these config files and I hate config I normally really really hate config driven development like I hate anything that like you fill out a config file and then that's how you program like there's nothing I hate more except except for this like I hate for AEL yeah like I hate kubernetes I hate GitHub actions like all that stuff just want to throw it out the window but like um it's a very useful feature for kind of this like fine-tuning large language model is because these config files like they kind of like trade in the community like oh how did you train whatever like mistol 7B on this data set

[27:05] to do get that outcome like oh let me send you my config file that's all you need to know like and what does the what's the scope of the config file yeah so the config file has what the data set is that you're going to be fine-tuning over um a lot of times you might be referencing a data set that's on hugging face but it can be local data set it'll tell you what the prompt template is so that's like a really huge thing like the prompt template is like a it's like the thing you have to like really worry about that in terms of like string

[27:42] formatting how you format it is it is it compatible with the base model if your base model has already been you know fine-tuned with a template you want to kind of preserve that as you keep tuning and just to take a step back well a couple of interjections the first is if you can't find Axel autle we will put a link into the show notes page so that you can just click it and find uh Axel Auto but um to take a step back like in terms of the prompt template uh and and fine-tuning you're fine-tuning you know on your data so presumably you've got some you know data

[28:25] set of examples that you want the that are representative of where you want to improve the model and you know are those typically in and this is kind of getting to the prompt template are they typically in like a question and answer format or are you know is it better to keep them more neutral so that you can then impose a prompt template like what should the yeah the the data set look like so the data set needs to look exactly like what is going to happen in production so if your users asking questions then it needs to look like questions and answers if it's going to

[29:01] be multi- turn conversations then it needs to be multi- turn conversations if it's going to be text a SQL it needs to be just text a SQL um and it needs to match exactly what's going to happen at inference time so whatever the prompt template is if it's like you know if you have some like bespoke prompt templating syntax like angle bracket Sam forward slash at the beginning of every message then you have to make sure that that is the exact same kind of templating use at inference time um and that I see is like a really big foot gun a lot of PE that is the

[29:42] thing that is the nightmare of of everyone doing this meaning they think they've collected you know this very valuable is X but it's really Y and like oh why doesn't it work oh I left out of space here I left out this colon here in this like place you know cuz like the model was trained with one I like oh like I I'm pretty sure the model was trained with like prompt template X but then when I go to inference it I'm like using a different one and it's like well why isn't it working and so there's many such cases of of like yeah I'm missing a

[30:19] space or whatever and sometimes prompt template like is it abstracted from the way you've collected the example uh is it like a is the idea that you know it's at a higher level than the examples themselves that's a good question so like the prom template is just like you know so it's all it's all a string at the end of the day that you feed into it's all a string the string gets tokenized and so the question is like how is the string constructed before it gets tokenized and so when the user asks a question you know they're not saying like they don't have like you know they

[31:02] don't type type in this is the question section and then put their question they're just typ in the question that should be the right user interface but then when you send it to the large language model you need to put some like you need to give it some idea of what the hell is going on kind of like a human I don't want to enterprize things too much but you want to say hey like you want to give the language model some instructions like in the QA in the QA example you want to say hey there's going to be a question and the question is going to have is going to be inside

[31:35] question tags and then I want you to produce an answer inside answer TXS and so you put some guidelines that help the language model understand what the hell it is in this blob of all these strings like what is going on like what is the question what is the answer what is the function call what is the function call result and all that format then is your prompt template which is different from your examples or kept separate for you know the convenience of reformatting typically your from the examples themselves a lot of times it's kept separate yeah in ax LEL it's kept separate and you can choose which one to

[32:14] apply we like kind of you say Okay I want like the chat ml format or I want the alpaca format or whatever and you know those are different whatever templates um which one to use is kind of up to you like different templates have better support or they have like some templates are like too very simple like they don't even have like sections for certain things they might not have you know yeah they might not do a good job of like multi-turn conversations they don't have an affordance in the templating specification so you kind of like have to make one up and then then you might decide you know what I'm just

[32:59] going to use like a different template that actually like has thought about multi- turn conversation stuff like that um you know when I'm fine-tuning so like just to give you some insight because when I F tune I'm doing it for these very narrow use cases usually I don't like to use a model that has any templating at all or I just do my own templating like little bit of templating or like zero temp complaining cuz I'm usually going from like text directly to something um so yeah I like to start with base models and not instruction tune models instruction tune models are already there like you know instruction tune

[33:43] just means you're trained a model to you know have question and answer which implies that there's a chat template or there's a prompt template that you have to worry about so I just don't want to deal with that mess I just try to use the base model and just come up with my own like simplified template that's specific to The Domain and is the is that true even if the domain is chat or conversation like is yeah the domain is chat or conversation is helpful to have a template I mean it's helpful to like think about it and like yeah sometimes you want to just keep training on the instruction tune

[34:23] model but if you I try not to so okay for so chatbots um in a lot of situations it's you know it's usually like not narrow it's like a COR like there's this you know when you put a chat bot in front of a user you have to work really hard to scope it um there's this like an inherent user expectation like oh this is like magic AI Chad GPT and so usually what ends up happening is like at least when I'm involved is I say okay like let's not do that like let's kind of you know pick a more narrow use case to do fine-tuning um I will say though like I

[35:10] have fine tuned models like that before um and it's totally fine um but I would just say for the most part yeah like I don't usually not fine tuning for chat usually fine tuning for like a very specific purpose of like doing some kind of data transformation or extraction or something like that okay uh I think we were talking about like what is in the Exel Auto config oh yeah so it's just like it's your data it's um sort of like the the prom template and it's all these like hyper parameters like bad size learning rate learning rate schedule all that stuff and what's useful about it is there's

[35:50] like a whole that sounds like all the hard stuff from Deep learning that you just said we didn't to worry about worry you don't have to worry about because you can just say okay if you want to train whatever some model you can just like look at the config that everyone because you're not building the config from scratch you're no you're definitely not getting the config okay got it yeah that's the magic part you don't have to worry about all this like fiddly crazy stuff you're just like okay look I'm going to copy and paste this config may not even understand the presumption then the presumption then is that um those

[36:24] types of hyperparameters are not you know deeply sensitive to you know the distribution of my particular data uh it's more maybe correlated to the model that you're uh fine-tuning that's a good question I think because otherwise I you know I've got the hyper parameter problem so so okay like it can be sensitive but it's not as it's not especially when you're fine-tuning like a Laura which is in a lot of case people are these like uh adapters it's not incred sensitive it's kind of like if you remember random Forest how it was so it was like kind of hard to screw it up you just like

[37:04] pointed at your data and it kind of felt like it was working sure there was some hyperparameters but generally people are like oh yeah this is like very easy like just called do fit and it's like working and like you know it was less fiddly than deep learning yeah like for sure like 100% is the same feeling now with training large language models is very like it just works like you get something that at least you get something that seems to work um and then you can always make it better of course by like you know kind of varying these hyper parameters but you don't you're not like oh it doesn't

[37:45] work at all like I'm just lost like what the hell you know kind of it you know it's not like that and like starting with a known config is a great starting point you know get where on this journey you've decided that you're going to find tune you found Axel Auto via this podcast or wherever you found Axel Auto you've got your config file are those config files just in the Axel AAL you have to go find yeah they're in the repo they're in like an examples folder yeah so you've got the config files from the repo like do you need to like deeply understand Laura have done the math have

[38:20] read the paper or is it just a kind of infrastructure that you don't have to think about no you don't have to do you don't have to think about it what you do have to worry about is like installing Axel dealing with Cuda errors and all that stuff so um you know because you said you wanted to run it yourself so yeah you said you want to run so like yeah so you're going to have that that problem is not gone anywhere um you know there's a Docker container so you can use that and that seems to be fair you can get it to your gpus you know how it is like I

[38:55] can I can say yeah it's going to be okay but like I mean that's not I would be lying if I told you that so yeah you can just uh you can use it to you know fine-tune a model on your data so I mean like where's the the thing that we're waving our hands over in this whole conversation is like oh yeah like you have a you have your data and like we shouldn't wave our hands over that like we've people have been waving their hands over that for ever ever since data scientists were coined um as a profession and the thing that's really interesting is if you already if

[39:34] you start with not fine tuning like you're using open AI or whatever and you're you instrumented you're already in yeah you you instrument it a bit hopefully yeah then that can become your source of data which is really fun so like you and you have I guess the question is like how many examples do you need to effectively find yeah how dependent is that on your use case yeah I mean so honestly I don't have the answer to this uh of like how what's the minimum number it doesn't feel like it's that much um I just kind of get as many as I can within like the

[40:16] time budget I have in the compute budget I have so like you know you have to make sure okay like one you so you have to make sure you're instrumenting your system so you're capturing ing all of this large language model invocations you and then you have to have a way of like curating and filtering that data you don't want to have you're going to waste a lot of compute if you have like bad garbage data uh that you know Ed to train your model um but once you curate that um yeah I can get really good example like really good results in like a thousand

[40:52] examples and a lot of times what I you can do is you can even synthetically generate data based on examples So based you know so I've done that before I've like expanded data uh significantly by using a very powerful model like gp4 to like expand my data set um so yeah there's a lot of tricks that you can do to generate that data and kind of bootstrap yourself into getting data which is the key difference between classical machine learning is like oh I don't have data I need to go label everything so I guess we're stuck there's like lots of ways to get unstuck you've mentioned Laura a couple of times

[41:35] and positioned it as an option like if you're going to fine tune and use Laura then you know it becomes easier like well first of all I'd love for you to like spend a few minutes talking about you know Laura and what it is but uh also what is it an alternative to or what are your other options uh when you're kind of filling out your config file yeah yeah so it's basically um I think it stands for low rank adapters low rank adaptation or something like that yeah something like that it's basically like so you're not doing a full fine-tune of all the weights in

[42:09] your model you're just fine-tuning a very small like this adapter um and it just makes it a lot easier in many ways um it reduces the amount of GPU vram that you need which makes it a lot more tractable on like consumer gpus or other kinds of gpus um and it's also yeah it tends to I guess like some much like smoother lost surface is like a lot it's like you know it's not that as many parameters um and it's it's a nice way to get started at least in terms of like training these adapters and actually can be fun to there's a lot of tricks you

[43:01] can use with these adapters so like these inference Frameworks allow you to like load the base model and then hot swap adapters okay so which is really cool cuz you can you don't have to incur the cost of loading an entire model just load the adapter for that you know forward pass and then you know go on your day um and that uh incidentally if I'm not mistaken is a framework that folks like you know Google and Amazon use to allow customers to fine-tune their own models they you know are using Laura and they have you know the customers basically building their own adapter and then they're like hot

[43:45] swapping the adapters out at inference time I have no idea it's it's my understanding yeah I saw some advertisement of like Apple the what they're doing on device um it seemed like from their explanation that they're doing something like that um but you know I don't know exactly what they're doing but it seems like very similar mhm um yeah so that's what it is um it's just a way to it's like a parameter efficient kind of way to fine-tune you mentioned inference Frameworks what are the inference Frameworks that you are thinking of and what's their role yeah VM and Nvidia uh there's like the Nvidia Triton there Nvidia Triton which is like

[44:36] the front end the back end they have like tensor RT llm um I like VM because it's a it's like really easy to use and straightforward um it's like kind of a good tradeoff if you're not wanting to squeeze tremendous amount of performance out of your models but um have you know it's like a pretty easy to use framework for inference so I would say those are the two that I kind of use it's either the Nvidia stack the trt and you know the trlm with uh some kind of inference server um or VM and what is uh like what specifically is it doing is it

[45:24] like um I I don't you remember Bento ml like is it analogous to this where you've got like an inference server and dist distribution load balancing and all that kind of stuff or is it just I glossed over it too much so okay sorry so inference servers a lot of time there's like a front end and a back end and what does that mean so like the back end will do things like model compilation um you know so like quantization model compilation different kinds of optimization to to the model to make it run faster and then the so meaning you your model may be distributed as some weights some set of

[46:08] Weights in like a safe tensor format but the inference process is probably going to be using some other form or artifact in memory and the back end that you're referring to the inference Ser is going to get as close to that as can be cached and like start there at for an inference just for performance reasons that's kind of what I'm hearing I have no idea exactly what is going on concretely like I don't I don't deeply know what compilation does I have a some general idea but I don't want to I just like actually treat it just like magic I'm like okay this is going to make my this

[46:50] is going to make my uh model run faster it's going to so I'm just going to apply the steps um that they're telling me to apply like you know yeah like there's a lot that you can you know yeah you know do this for people like and not have to know those details that it speaks to I think a level of maturity in the tool chain and like abstraction that um you know wasn't here you know a year or two ago and probably was the reason why everyone said don't even bother trying to find tun unless you're Facebook or Google because you had to invent all this stuff

[47:32] yourself from the ground up sure I mean it helps to know a little bit like okay it's helpful to know what quantization is so that when you do and you want to measure everything so it's okay to like generally speaking you want to try stuff there's a whole host of different options and a lot of these inference servers and backends have lots of knobs lots of high parameters that you can tune and honestly like anyone that I talk to um in Industry like they don't understand all of it and they're they just try stuff like they they try a lot of stuff they fiddle with it um it

[48:11] really depends on like you know your like throughput versus latency requirements like how to actually tune these things to do what you want them to do and so the reason why like it's okay you know why I'm okay with sort of treating some of it like magic is because I end up measuring it all you know and then kind of reasoning about what's happening it's helpful to be somewhat familiar with the terminology so you can kind of know like why certain things are happening MH but I would say like you don't need to be an expert in all of it to start using it um you can you can definitely measure

[48:54] things as well and kind of like you know you don't need to like know all the ins and outs of like model compilation and exactly how it works you can you can do it and see like okay what's happening to my performance latency so on so forth so yeah so and you know that part is not non-trivial like hosting your own model is actually non-trivial there are bunch of startups out there that will host your model like fireworks or uh replicate um modal things like that but yeah you have to think about like those costs and you know how that makes sense given your scale and so it kind of that's why it

[49:37] goes back to this whole like in the beginning just use off the-shelf model um because those off the shelf the models like actually they're they're very like attractive in their costs like they don't cost that much especially at a small scale and they have right somewhat low low latency and they're aail able most of the time so it's a great place to begin it's very actually very difficult to compete with that M so yeah I think I sted far from what you're asking but yeah like inference and I'm not an inference expert there's people and there's people that came into the course that were like inference experts

[50:14] actually this like whole debate about fine-tuning uh I had two guest speakers one was uh Emanuel Amon from anthropic his the title of his talk was why findun is dead and then there's another another talk from uh Kyle Corbett who is CEO of open pipe and he was talking about he was also saying like you should usually not F tune but he had some other perspective of like okay in these situations um you know you it is very extremely helpful helpful to find tune and open pipe is a uh is a tool that helps you find tune based on your logs So based on you know how we're talking

[50:58] about collecting those logs for fine-tuning um so I would highly recommend watching both of those um those are like posted now or do you need to register for the course no no you don't have to register all the so by the way the whole course is going to be open is most of it is open they're just going through we processing the videos you know how it is it takes time to whatever uh so um yeah we I can give you the links for those and you can put those in the show notes okay awesome awesome so the question one level up on the stack was what were your Alternatives in

[51:39] terms of Laura versus was that was the answer to that VM and other inference servers it like is or is that I feel like that's mixing up different levels of abstraction so like you can use um you can do inference with Laura you can like the adapters you can also merge the adap back into the model so that you no longer have an adapter or you can choose to keep it separate and the inference Frameworks are all good about okay like if you have an adapter let us know but you can yeah you can just um it used to be the case that like when it was

[52:11] earlier on like not everyone had support in it support for adapters and the inference side of things so you kind of had to merge it uh but now you don't so are you typically choosing between Laura and some other alternative when you're filling out your config file and like what are you thinking about as you're making that choice or yeah so okay I don't go I don't try to optimize it too much like um I try to start with the simplest thing first so I usually try like Laura and try to fine-tune it on the like a like a consumer grade GPU or workstation grade GPU not a server grade

[52:51] um just to see what happens and if it's good enough if it's pretty good I'll just you know call it a day and say like okay this is like really good not try to like you know hyper optimize things you can sure you can do a full fine tune um but you know like yeah full fine tunes can give you better results um but I've had really good luck with using the those adapters to be honest this like worked really well so I haven't really had to go I haven't really had to do a full fine tune and can you do this reasonably on Apple silicon like just on your laptop

[53:34] oh interesting I haven't tried that so I actually try to keep surprised by what you can do now uh on Apple silicon after you know years of just not having any GPU options yeah uh on a Mac I know that it's like people are inferencing models a lot on their apple silicon I haven't tried to fine-tune I know there's some Frameworks out there I think it's like mlx or something that may metal or something metal extensions I think yeah something like that mhm again like there's too many things going on a lot of moving parts that I just tried to simplify a workflow I don't want to deal

[54:11] with that yeah so I just try to remove like these variables that can complicate things for me so I just stay on Nvidia uh cuz like yeah just like cleaning the data and like designing the application as hard hard as it is mhm and it sounds like another uh kind of Lynch pin in all this is uh evaluation and measurement and so that you can in in the case of this conversation like feel good about all the things that you don't necessarily know like the inner workings of because you know the impact that they're having on your data and your models performance so like maybe let's segue into the way you think

[54:54] about uh evaluation instument for llm apps so evals I think okay so evals is the most important thing so like I think like when it comes to large language models that's that's all I talk about nowadays because like um in the Consulting work that I do so people call me when their AI doesn't work it's like hey like we kind of prototyped something it um you know we're excited about it but it's like not really working kind it demos well but it doesn't work like in the real world or it's not work you know there a lot of failure modes and we don't know what to do it seems like a

[55:31] lot of the game right now is like I can get a really nice demo but how do I get it to production and actually I mean we've been saying this for a really long time in ml and AI yeah yeah but it's especially the case in large language models because you can get to demo insanely fast like with within a day for sure and so um but even like the very polished products like okay people don't know how to get unstuck they're like okay I use like open AI I've built like some interesting product around it it solves a problem but like the AI just like fails too much

[56:08] to do something or makes too many mistakes and so the key like you know and then like what people do to start off with is they do prompt engineering is you know they see a failure mode of the something going wrong and they update the prompt but then like really fast it becomes very unwieldy and you end up making a change to a prompt and actually causes some other error um and you kind of get stuck and so that's where this notion of evals comes in is you want a systematic way to test the efficacy of your system so that you can make a change and you can see

[56:51] immediately like okay what you know did did you fix fix your errors did you create new ones things like that and it's like very analogous to software engineering in a way like you would never write software without unit test it would be very difficult Beyond a certain point it would be unsustainable same thing with large language models like you need some kind of tests and it's really the only way you can build AI so it's it's like integral to the building process of AI product it's not it's not even an optional step the interesting thing is there's not too much information out there around how to do it there's a lot of

[57:39] tools um so you know there's a lot of people uh you know there's a lot of different startups and things like that and the thing that trips people up is they focus a lot on the tools but not the process of of doing the thing um so the tool can help you a little bit in the sense like make the process easier but if you don't have a good process on like how to think about evaluations and do the evaluations then it's not going to help you and so with evaluations really it's you know a process of looking at lots of data first of all removing all the friction you can in

[58:20] looking at data so making sure that you have a way of looking at all these logs these llm logs in a way that makes sense for you like in a domain specific way um then that allows you to look through the data really fast and navigate the data and and also navigate it in a domain specific way that makes sense for your business but then also um you know writing tests the different kinds of tests so not going straight to LM as a judge so a lot of a lot of people when they think about evals they're like okay one failure mode is they go to

[58:59] tools and the tools have kind of this box of evals off the shelf like here's a toxicity eval and here's a conciseness eval and here's a helpfulness Evo the problem is is that all that stuff usually is not helpful for you in your use case what what you yes totally generic and um and so what you what you need to do is like write lots of different assertions lots of other kinds of tests so like you want to make sure that you write um as many assertions you as you can before getting to LM as a judge and then when you do go to LM as a

[59:43] judge you need to make sure that that judge that you agree with it so the question the problem with LM as a judge is the question quickly becomes like how do I trust it how do I trust this AI to score my data or to like do I know and so you can imply you can apply like a systematic way to reason about that which is like have a human also score or you know make a judgment about that data and then measure uh kind of like how much an agreement those two things are and then try to align the judge over time you know with the the human so like that's

[1:00:29] that's like in a broad sense kind of a process of like looking at data writing tests um you know and doing that over and over again and then making changes to your system and that that also ties ties into fine fine tuning because if you that is the work also if you do that properly um if you have like a good if you have good instrumentation good way of looking at data good way of like labeling curating data um and you know you have all these tests that can fil that you know that can also filter out bad data you can curate data extremely fast and you

[1:01:12] basically accomplish all the work of fine-tuning um so it does tie into fine-tuning but it's more General than that it's like even if you're not fine tuning at all it's really the only way that you can systematic ically improve your AI um and you can almost like if you're not doing any evals at all you can almost guarantee or you can you know pretty much guarantee that you can improve your AI by doing evals because you have a way to test things and improve things and there's like different techniques on how to improve things but even things like you know even things like um uh these automated prompt

[1:01:57] optimization Frameworks like DP to yeah like dpy or you know dpy is probably the most kind of well-known uh kind of tool in that genre you know if you have really good tests in metrics then that is going to then you can actually do something with those prompt op optimization things if you don't really have a good handle on that and not really dialed in then yeah it's it's going to not be as great so once you do evals you unlock a whole host of things that are available to you um in addition to just being just helping you make progress you lock all these other

[1:02:42] tools um whether it's fine tuning whether it's prompt opposition um or really anything else um and yeah so that's that's the thing that people struggle with I would say a lot of people that pick up the phone and call me and they're struggling with their AI you know one of the first questions is like hey tell me a little bit about how you're testing your system and tell me about what we're measuring a lot of times there is nothing like know a lot of times just hey you're just looking at it like as user so yeah I think it's the most important thing um at least at moment in terms of

[1:03:28] what is the bottleneck in terms of you know moving the community forward in AI yeah yeah um do you distinguish between kind of Frameworks and tools when you think about testing meaning you said you know a lot of people kind of jump at the tools first and um you know get stuck behind what the tools offer you know is the alternative to that like building everything from the ground up from scratch custom to your use case or there some medium level Frameworks that you can turn to to help organize things so when I recommend people do is actually like use try to use what you have first so like use your existing

[1:04:19] unit testing framework use you know whatever CI that you have um if you're going to like log the results of your test log the results in your test into whatever analytic system you have it's not going to be great but start there and do the process like two or three times and then you have a good foundation in which to think about tools the tools can certainly help the process that you're doing two or three times like yeah like is to like go through to like write the test observe like the results of the test try to make your system better try to you know so bang your head against it

[1:05:01] a little bit until you feel like you're try try to you know make some progress on your system like he I have this error let me try to make it better and then like fix that error observe the error rate going down validate that like okay you kind of have tests that make sense and then it like will click in your head like okay this is what this this is a good way to think about my system this is what I'm trying to measure this is what's important to me um you know I need to look at this kind of data this is the kind of failures

[1:05:29] that I even have um you know this is a process like who is going to look at the data who is going to write the test um you know all of these different moving Parts like you kind of have an idea and then when you approach a tool you can actually use the tool you can like appreciate the first yeah you can appreciate the tool and you can evaluate these different tools and like you know for your domain um and then you're like way likely more likely to be successful rather than yeah if you're just going straight to a tool it's really hard for people to you

[1:06:09] know they just do whatever the tool says on the front page of the documentation they don't know so um yeah that that's kind of what I would recommend but there's different kinds of there's there's there's a open source tool so there's um there's something that JJ has been working on but JJ is the he created R Studio fusion and all that stuff back kind of all kinds of crazy stuff so he's recently created something called inspect which is a open- source evaluation framework so yeah like he has one that's like open source there's not too many open source things there's another one called open elemetry okay

[1:06:51] um but I think even open elemetry is vendor Affiliated um so I think JJs is the only one that I know of that's that's completely un non-vendor Affiliated um but then everything else is vendor vendor stuff so as I said like first yeah let's like do it yourself based on the tools you already have maybe if you don't want to do vendor stuff or what some reason maybe you're just by yourself and you're not like wanting to do yeah you try out this open source thing and then from there maybe try one of the vendors or just skip the open source thing and go straight to a vendor if you want so the

[1:07:32] key Insight is uh bend your head against a problem in your own words and in your own terms before you jump to some tool to define the problem for you because you won't know what you're kind of missing in that Loop until you do yeah absolutely um it's not that bad it's not as it's not hard as it seems It's actually it's a good exercise to go through yourself it's not that you're going to create all this technical debt or something by doing it once with the tools that you already have there's a lot of people that just stick with that by the way they just

[1:08:08] like use the tools they already have and they like kind of like it um but you know the yeah so is good to stay like to start there and do you find that it's obvious to folks what to measure or um to what degree is it use case specific versus kind of standard um you know just by virtue of the fact that it's an llm system yeah it's usually very all domain specific okay um like when we look at the data I think with enough coaching you know 100% of people that I've come across have been able to learn how to do it very effectively there's some skills involved

[1:08:59] in learning how to look at data and sift through it um basically teaching people to be very skeptical and saying like hey something's broken like let's let's be skeptical of the whole system you know let's look at your data very carefully um a lot of times it takes PA programming and Pa debugging but I find that when people people usually get it really fast they're like okay you know what this is it's kind of like it's kind of similar to the scientific method you know it's like a way of thinking like hey I need to prove to myself like why there this's like black box in a sense this large language

[1:09:44] model don't neily understand like exactly how it's working but you can run some experiments you can measure some you can take you can take some measurements and you can change some things you can you can observe the changes to those measurements and you can think about that and I think people kind like a lot of people have that inherent skill of thinking about that or like working that way and it's like once it clicks in their mind where people are able to do it yeah maybe as a followup to the question about kind of the domain specific nature of these measurements like what are some concrete examples of like you know a

[1:10:21] project and the the important things to measure you know that became apparent to measure in that project relative to you know other types of projects yeah so okay like the real estate uh the real estate CRM example the one I was talking about earlier the company reat so like their errors are quite boring in the sense like okay um you have a u ID that's showing up in the output that's that's contained in the system prompt and it's contained in the system prompt because it's trying you need it to use uh you need it for a function call later on you might need it for function call later on for whatever

[1:11:03] reason you know that user uu ID has been reproduced in the final output and that's not good MH or um you know certain sentences are being repeated twice or um in reat case because they're rendering these like uis these widgets not just text they have these special delimiters they're using to indicate uh the like what the Met insert the widget yeah yeah where to insert the widget and things like okay is the delimiter you know is there something wrong with the way the delimiter is placed you know there's all kinds of assertions like that um and so so at one level like there's you know there's query of response and

[1:12:00] you can like you you know probably want to be tracking everything at this query and response level but it sounds like also um you're almost like

[1:12:16] featurization is that the idea yeah definitely yeah the the idea is like and the reason harping on assertion so much is a lot of times people think they can't assertion they're like hey it's fine I just it's very subjective and that that is the case sometimes but often times I will look at data very closely and I'm like I find kind of these patterns of failure modes that can be expressed as assertions um it only comes to like looking at the data very carefully and that's kind of the first line of defense it's like hey you should try to solve all these dumb failure modes that you

[1:12:54] can tell with assertions before moving on to things that are more subjective or require like higher level of reasoning then that then you have to go on to use like an LM as a judge or something like that interesting which kind of goes back to the like bang your head against it like well collect the data and bang that your head against it because I'm imagining like you collect the data you you know bang your head against it and you'll start to identify these common patterns of things that can go wrong and then you can both longitudinally track them but if you know that anytime like a

[1:13:32] bad delimiter you know is you know appears in your response is bad you can just like put up a guard rail against it in the you know in the application so that you're not showing this bad thing to your users or you're taking some action regenerating it or if you can fix it fix it manually or whatever like there's you once you start to once you get the process down of like building a mental model around the things that go wrong or expressing that in terms of assertions there are lots of places that those assertions can manifest or be used that's a great point and so this is like

[1:14:10] why evals are not really a separate thing it's very much just part of building and you're not you can't build without it and kind of as you evolve the testing you know you don't want to let's say uh Silo the assertions in your unit testing framework for example you want to keep it in some other location where you can both use it in your unit testing framework but also you can use it in your LM invocation pipeline so you can do things like test for that and then do some self-healing depending on how expensive that specific test is the assertion is cheap so you can do it um and so like all that stuff

[1:14:49] is you know it's it kind of very much Blends into the process of building the AI mhm very cool you know another example like the honey honeycomb example the text of sequel It's like not really text a SQL text of that domain specific language but the assertion there is like is the query syntactically correct yeah and not only syntactically correct like in the abstract sense like is it syntactically correct given the user schema and things like that so you push the assertion as f as far as you can go in a way that makes sense like in a naive you could say is it a valid

[1:15:29] honeycom query okay that's one but like if you think about it further you can you got say okay like let's think about the schema as well and so um it's always the first line of defense yeah interesting I I came into the conversation thinking that like the conversation around if evals would be like very I don't know like maybe dialogue Centric or rag Centric or um you know Precision recall that kind of stuff uh which comes up in a lot of conversations around rag but this is much more of like a software engineering yeah I mean um you all you do want to measure the various components of your

[1:16:12] system like Rag and you definitely want to measure retrieval like actually rag is a fantastic place to apply uh evaluations because it's a very mature like retrieval is a very in mature field especially information retrieval so you can apply a lot of the tools that we've learned over the last few decades to rag in terms of um the ability your ability to like you know retrieve the right information there's a lot of metrics there's a lot of science out there around like in tools um but even before you go there it's good to have these like more simple kind of holistic evaluations of your system and then you

[1:16:55] start to like measure these different components and stuff like that awesome awesome um I think that's probably the most important thing is to like look at your data and it's really to look at your data it's easier said than done we've been saying that you've probably been hearing that on your podcast for like you know as long as the podcast has been alive um it's very easy to say no one does it um is why I have a Consulting business is because no one's looking at their data if people start looking at their data then I you know I could do something else but they're not and um it's I think

[1:17:31] it's a little bit cheeky because there is some skill involved actually that we can take for granted as data scientists like um there's a skill in like how to sift through your data really fast having a nose for what's wrong um how to like kind of analyze your data like look through it and with the lens of the machine Learning System and kind of intuiting like why something might be happening so it's like it's funny for me to say like hey look at your data but there's a lot there um but you know starting to practice it is actually is very critical to this whole building AI

[1:18:13] stuff you know it makes me relate back to a comment that you made earlier in the context of fine-tuning like we're far from like the you know the hard days of uh training deep learning models and you know even before that was like old school data science but like you're kind of pointing to this idea that it's you know either it's a cycle or it's all connected or it's all the same thing like it's all about looking at your data and understanding how to intu it your way around uh the relationship between data and some objective that you have yeah definitely yeah very cool well haml

[1:18:52] thanks so much for jumping on and sharing a bit about you know what you've been learning and seeing uh with regard to eval and uh fine-tuning and and Building Systems around llms it's great to to speak and and catch up yeah thank you [Music] [Applause] [Music] [Applause] [Music] [Applause] [Music]

---

About this transcript
Read from YouTube's own caption track and laid out by ViewRank AI (https://viewrankai.com).
ViewRank AI finds the videos already beating a creator's own average on Instagram, TikTok and YouTube Shorts, transcribes them from the audio itself in more than 60 languages, and turns what worked into new ideas and scripts.
Free transcript tools, no account needed: https://viewrankai.com/tools
How to read any video this way: https://viewrankai.com/llms.txt