# Easiest way to install local LLM

Channel: The Next New Thing
Video: https://www.youtube.com/watch?v=kb3es9LS8Wc
Duration: 27 min
Language: English
Words: 4535
Transcript page: https://viewrankai.com/tools/youtube-transcript/kb3es9LS8Wc

---

[0:00] You'll never have to pay for AI tokens again after you copy and paste this simple command and run free open source software on even an old computer. We're going to walk you through it and at the end you'll be able to run Hermes agent, an open source version of Perplexity. You'll be able to do programming on your computer, AI searches, run background tasks, so many things. It's a lot easier than you and I expected and it's one simple copy and paste away. We'll walk you through the whole thing in this

[0:24] simple tutorial. Let's get started. Presented by Zapier, the AI automation company. All right, Mike, let's see from scratch how I can install a local model on my computer. Yeah, absolutely. Right now you're on the repo for ODS by Osmotic on GitHub. And we'll have a link to it below. This is our free and open source project for local AI deployment. So Apache 2.0, completely openly licensed,

[0:52] completely open source. Growing really, really fast right now and it's a super easy way to deploy a local AI to just about any machine you can imagine. So right now we're on the repo itself. As you can see Osmotic ODS. Scrolling down the side of all of our wonderful contributors and stars and all that good stuff. But unavoidable and impossible to miss is just as soon as you scroll into

[1:20] the read me, you'll see get started. If you're on a Linux or Mac OS machine, you could just open up your terminal and then just copy paste the curl command from here into that and it'll run the entire installer soup to nuts without you having to do anything. In this particular case, this call's happening

[1:41] on the Windows machine. So we'll go through the Windows process. Right here we see Windows quick start. So I just click that. And there's a command for PowerShell uh, right here in Windows. Uh, so, I'm going to open PowerShell. Run as admin. And if you can see my terminal screen here, I'm literally just going to paste

[2:10] what I copied off of the repo. Uh, it's asking for permission to run. There's the install command ready to go. Enter one more time. And now it's starting to boot up the installer. Uh, continue as administrator. Yes. Uh, full stack install. And now it's off to the races. Uh, so, uh, on average day, depending on system, depending on internet connection, this can take, uh, anywhere from, uh, you know, 5 to 6 minutes or to set up. Uh, what it's doing right now is it's taking every single application uh, that you could probably want for a local AI. Um, it's building a whole ecosystem for them. So, it's setting up

[3:00] N and it's setting up open web UI, Hermes agents, uh, uh, comfy UI for image generation, open code for agentic coding, perplexica for deep research and AI driven internet search. Uh, just about anything that you could ever want to use local AI for, O D S just sets it up for you so you don't have to have any of the pain of piping these things in or figuring out dependencies. Uh, anyone who's ever done it by hand would be able to tell you that even just setting up N and locally and getting it to run functionally is like a weekend

[3:36] project. And, uh, little alone llama.cpp for, um, V L M or figuring out CUDA or Linux or individual optimizations for applications. It could be a real web. right? So, literally everything that Why are you installing something like Hermes? Why not leave it up to the to the individual to add optional additions like that? Uh we do. Uh so, we start with a couple of core applications that uh we think

[4:08] almost every user is going to want. So far, like uh nobody's ever pushed back on Hermes uh being set up for them. Um Open Web UI for FlexiCAM. Uh these are pretty low-list applications, but they cover a wide variety of use cases. Um and then we have an entire extensions uh kind of like ecosystem uh that you're going to see inside there where you can add, subtract, move, edit, change all the different applications that are inside of it. But, we would feel like it probably wasn't a proper solution if you don't know anything about local AI, you want to self-host, uh you want to self-host cuz you want to

[4:49] be able to code or use Hermes or something like that. And then you did install ODS, and uh then it still feels like you have all kinds of setup and random things to do. We wanted to feel like you could just pop out of the box and just start having fun already. Uh so, that was the idea. Let's talk about the kind of uh machine that I would need to run this on. Can I run it on my Mac here? It's a MacBook Pro 16 GB

[5:15] of memory. Absolutely. You can run it on a $600 Mac Mini. Um okay. You can you can run this on a tiny little $600 tin can, and it'll recognize it. Uh it'll auto-detect the hardware, the operating system, uh the silicon itself, and it'll select everything that's needed for that hardware to be functional. And it'll even select a

[5:41] model that fits inside that footprint. So, if you were to run this on uh your MacBook with you said 16 gigs of RAM? Mhm. Yeah, it would pick a model that fits inside of that range and responds well on unified memory type systems. Traditionally, that would be like a clan 3.52 B, which feels very snappy even on unified memory. And if you have bigger beefier Macs like let's say I've got over there on the shelf and M5 with 128 gigabits of RAM. It would install 35 billion parameter MLE models like

[6:23] length. Just because MLEs tend to play nicer with unified memory systems and setups. But essentially any hardware that you would try to run this on, it'll detect that hardware and it'll set up models and settings that make sense for it. Like what we're installing on right now is uh it's like an Acer laptop. There's nothing really an $1,800

[6:49] laptop. Not cheap, but also not something that you would naturally think of for artificial intelligence. And um it'll install on just about anything that you can picture. It'll just pick smaller models if the hardware is less beefy. So, the gateway is open. We've got an update and we should be able to pop up into the

[7:11] browser now. And go to uh localhost. This is exciting. It is. And we should have a fresh install waiting for us. Localhost 3001. Boom. Uh so, we've got our live dashboard for ODS. Uh so, there's been no activity yet, so you don't see any live throughput on tokens per second. But um, we got this awesome smart dashboard and what happened was we were spending a lot of time bouncing back and forth between

[7:50] applications. Uh, like hardware info, you you know, is a great example. When you're running AI, you want to know what the thermals are of your GPU. Is it pulling, you know, drawing power right now? Is it active? Uh, how much memory do you currently have utilized versus available? Uh, what are the temps in your hardware? All this different stuff. Uh, tokens per second, you know, how quickly are you serving a

[8:15] model? How quickly are things moving? So, we wanted to make all of that super easy and convenient. So, rather than juggling a million different applications, uh, you could just see the performance of your system in one place. Uh, let's give one of these applications a try. Now, it's going to make me set up an account, uh, you know, for it just because this is a fresh install. It

[8:35] wasn't waiting for me. Uh, let's run here. So, we're in here with Open Web UI now. Uh, we've got Llama 3.5 9B at Q4 loader. Um, so, when ODS ran, um, it automatically recognized the operating system, the hardware that it was attempting to run on. It's all, uh, my little handy dandy Acer laptop with its 8GB GPU sitting there. And it specifically picked a model that was going to fit inside the VRAM of that GPU and perform reasonably well. Uh, so, all models, uh, that ODS picks, uh, we've personally tested on hardware. So, it'd be easy enough to just go, "Oh, you've got 128

[9:24] gigs of memory. Uh, we're going to give you a really big model." But, on a unified memory system and with a dense model, that might run really, really terribly. Uh, so, it might might be a good experience. So, uh, every single model when you run ODS is something that we've actually run on that type of hardware to get an experience from and kind of

[9:44] see how it feels. So, in this particular case, quant 3.59v with a click on a sample prompt like show me a code snippet and it'll go off to the races. So, keep in mind um this is like an $1,800 Acer laptop. It's got like uh I want to say a mobile 5070 inside of it, so nothing crazy but you know, in terms of computing power. Um it's not the cheapest laptop that you can get, but it is certainly nothing that anyone would expect you'd run artificial

[10:20] intelligence on. So, I asked it to build an example of like a website sticky header. And you know, it's our showing me the header on the website and how when you scroll down, you know, it stays sticky with it. I honestly, this is the stuff we're kind of expecting from AI but not from something that you can run on a on an old laptop. Okay, this makes total

[10:42] sense. Yeah, this is just a 9 billion parameter model. Uh this can run on just about anything. Um and uh if you've got a GPU, even a weaker one, even sitting on a laptop, uh you know, this cooks pretty well. So, I can say things like um I asked it to make it more colorful and

[11:02] visually interesting. And uh it'll sit there and start editing what we've already started. You can build entire websites, but you know, just in here and uh upload them to the internet, which is pretty cool. Um so, while we're looking at this, I'll switch over modes real quick back to our dashboard, uh which we were talking about earlier. And you can actually look at the dashboard and you can see the GPU

[11:25] working. Uh so, like that's useful. Um uh you can see the current RAM being consumed uh by the model. We've got roughly 8 gigs available to us. It's consuming 7.3. You can see what the GPU is doing. You can see the thermals of all the individual systems. I know for a lot of people like this is genuinely useful

[11:47] data. Just because while the system's performing, you can see how much it's straining itself for uh And you're talking to me on the laptop that this is all happening on, right? Yes. So And now I see the the finished product, right? There it is. Yeah. They absolutely. This was two prompts that you know sitting in a chat that you know for like a sample website. Uh But is the is the top menu sticky actually? When you scroll up, is it

[12:13] staying there? No, it's disappeared. Now it's switched to uh being colorful and more visually interesting. I see because you told it to do that when you scroll up the when you scroll down, I think the menu goes away. When you scroll No. Keep I see. Now you're telling me keep it sticky. All right. What else can you do with this model? Just give me a few ideas of

[12:35] what can be done here. Yeah, absolutely. All kinds of fun things. Um so I just gave it a task. So it's going to run on that for a second, but um We've got a whole bunch of different applications here with RDS. All kinds of stuff that you can enable. Um so we've got Hermes agent

[12:55] which is uh always popular. So let's look at that for a second. The one that you just talked about. When you rebooted your computer, I went back to my Hermes agent and I said I want you to help me get guests. It already has a skill set up for that for emailing guests. And I said now I want you to add the ability to tweet back at them and say, "Hey, I just emailed you to book you on

[13:18] the show." This is the kind of thing that it can do, right? I also had it book a haircut for me. All that kind of basic stuff it could do on computer. don't need to don't need to pay open AI. Absolutely, 100%. The vast majority of use cases, the vast majority of things that we're using artificial intelligence for today, um don't really require some big gigantic 3 trillion parameter model

[13:44] in a data center in Texas. Uh you know, a lot of this stuff can could run locally on your own machine without real issues. Okay, you were going to show me also um maybe we wait till this is done. Oh, there it is. Oh, I like that. That's cool. Gave me like a tilting a tilting sticky ever, and uh I've got like a spotlight in there that I can use. You know, it's really silly stuff, but you know, examples of something you could do just sitting here messing

[14:09] around. And when you were saying Hermes agent, how how does it integrate with Hermes agent? What do you have there? There it is. There's their uh experience. Yeah. Uh so, I should just be able to open a chat. See what we get here. And uh Hermes agent, for people who aren't familiar, is a general purpose agent framework. Uh so, you can either build customized Hermes agents for particular tasks or applications, or uh barring that, they could be generalized assistants for any number of different

[14:44] things. Like I said, I had it book a haircut for me. I uh I have it find guests for me. I have it prep me for interviews. I have it look for messages that I sent out in my email that need a follow-up, but didn't get one, and it sends it to them. Uh I got this annoying email address that feeds messages into my inbox. I have Hermes go and organize it properly and allow me to just hit reply to reply to people. It's all kinds of random stuff like that. Um and of course, we've got a whole video about all the cool

[15:13] uses of Hermes agent. So, it's rattling off. I just introduced myself and then asked it to start telling me about its capabilities and all the different things uh that it could be doing for me here. Uh so, I could do just about anything that you would do with your Hermes agent normally using an API. Uh I could do right here on the laptop. Uh now, one thing worth distinguishing, uh you know, as as we get into the stuff with local artificial intelligences, um even though ODS doesn't care what kind of hardware uh you're installing on, it is going to pick a model that makes sense for that hardware. You know,

[15:50] it's so uh I've got like a $50,000 Blackwell tower sitting next to me uh that could load some big crazy super intelligent Hermes agent assistant. Uh when you install it on a little laptop like this, it's it's going to pick a model that, you know, that can actually work. That being said, it still has the same capabilities. It could still search the web. It can still check things out. It can still do all kinds of different

[16:13] stuff for you. Um we also have features like uh Perplexity. Yeah, what Which is um Yeah, a lot of people haven't interacted with this and it's a super cool application and I genuinely prefer to uh deep research uh from ChatGPT. And this is kind of how you should think about it. Uh anything that you're going to ask here, an AI agent is going to go out, do a search, gather all kinds of data, and then like come back to you uh with an answer. You can have really, really simple questions on a high-speed uh or you could go for like deep research level settings where like this

[16:54] thing will sit here and churn and churn, digging into all the details, everything it can find on that topic, really doing a deep analysis. I've got to do market research and things like that uh when it's all the way in uh quality mode. And it will dig into specific companies, industries, market segments, territories. It's exactly like deep research from OpenAI. It's just it runs

[17:19] on your own computer. All my own stuff, all my own processing, don't pay anyone a dime for any of it. And you're typing in what is happening in orphans in open source AI right now? See, I would have done the vanity search. I would have asked it tell me about Mike Mike Bradley, tell me about Andrew Warner. Okay, you're having it do that kind of research. Um I've heard of Perplexica as like the open source alternative to

[17:43] Perplexity AI. That kind of like research. Um well, yes. When used in the research sets, the trouble is Perplexica has become such a big all-encompassing company at this point that it's it's hard to but yes, very very similar. You're using it for deep research and search heavy tasks. So in this particular case uh it pulled three queries, uh 62 different results of what's going on in uh open source AI right now. It's generating the answer for us. It's talking about uh recently released models, everything that's currently going on, and um then it'll pull up sources and links like

[18:25] that on the sidebar. But uh pretty cool stuff. I could totally see this just sitting on an old computer doing research endlessly for me using the internet especially at night to do what it needs and not not costing a penny. Um what else do you have bundled with this software? what I use it for. Yeah, like what? Give me a couple of

[18:45] examples of what else you do with it. Um workflows. All right, you know, uh and then back-end automations. It's probably a little bit harder to show here, but uh I would build things especially when you're using um uh consumer hardware or more modest hardware. Uh one of the first things that you're going to encounter if uh you've got like a weak GPU or a laptop that isn't really it's not really designed to cope for local artificial intelligence is um uh you're going to find it hosting the biggest model that you can inside that hardware usually doesn't mean that it's going to

[19:26] run fast but it still could run well. Um so in a lot of cases uh you can put back end uh workflows, automated tasks, things like that, things that trigger whenever you get an email, etc. Um they're running in the background, they're doing things for you and you're not really noticing how slow or fast they are uh

[19:48] just because they're back end processes. So workflows is a great use case. Um search, local agents is another great use case. Um we've got an extension system here which is pretty cool. What type of extensions? It's uh just a whole army of different applications uh that you can host for RDS. Of course it's not cooperating with my internet right now. I'll switch to the

[20:18] model section. Okay, this is where I switch models as new models come online, I can switch right here. Absolutely. Uh so you can go here anytime and uh you'll see a wide selection of different model types. Uh so here's uh 1.3 4.5 9B, that's what's currently running on this machine. Uh 3.5 2B which was downloaded as a bootstrap. Uh you've got a whole bunch of different models here, just pages and

[20:50] pages of them. Show me how I would add or switch a model. And can I use multiple models? Does that make sense? Um uh depends on your hardware how many you can actually load uh you know onto a GPU but you absolutely can as long as you have the memory for it. Okay. A lot of people traditionally don't just because they're maximizing

[21:12] how big of a model they can fit inside. Okay, you don't have to actually add a model here. Just just tell me what you do. Do you just hit download and then it goes and it makes it available and you can then press a button to switch. Uh yeah, it's it's that straightforward. That's it and it's all right here and it tells me which ones I can, which ones I

[21:28] can't fit here, right? Yeah. Quant 3.5 2B All right, I've got to ask you a question. All of this is free. I don't even have to go to your website to get any of this. I could just run this completely on my computer. You don't even know I'm using it. You get no benefit from it. Your company is

[21:48] bootstrapped and profitable. How? Where is Where does your revenue come from? Yeah. And so um And we do a lot of consulting and client work and uh we have enterprise versions of ODS that also get deployed. So um Uh for something like ODS, it's it's really cool. Uh you know, you could you could do a lot of stuff here um as an individual consumer with your laptop or something like that, but if you were an organization looking to deploy a server and uh you wanted wanted different people to be running on top of this thing, you're going to want a lot of stuff that, you know, isn't going to

[22:25] be a traditional consumer application. Uh you're going to want uh login systems, credentials, individual access tiers and things like that for different users. And uh so we have enterprise grade versions of ODS uh that we help organizations deploy. Uh but the big focus was always um just helping as many people self-host as possible. Uh you know, helping as many people

[22:52] uh set up servers. believe the future is all self-hosted. Because of what? Why? Uh now, it's not the future is all self-hosted, but the future is way better if self-hosting is very, very easy. Okay. And the big reason for that is just competition. It's competition in all markets makes those markets better, more efficient, better for consumers and end users of artificial intelligence. So, if uh there's only two competitors in the market, if you just have like Anthropic or OpenAI as people competing, that doesn't make them bad companies, but they're not very highly incentivized to price competitively or give you the best possible experience for the lowest

[23:41] possible cost as an end user. How it becomes like the relationship that a lot of people have with Verizon and Comcast. That might be a good example, where the internet's not working and you call for support and uh and they go, "Hey, well, it just is what it is. You can go to the person down the street, but you're going to have the same experience." Um having a hundred different people who are providing AI models to you, a hundred different sources of potential inference that can be hosted to you, and having all of them competing against each other for your time, your dollars, your usage, that's just a better dynamic for the

[24:20] world, for humanity, for everybody. It's better for everybody who isn't just one of two or three companies. And Mike, I also have to say it's better for creativity to be able to create without having somebody monitor every token, every use, every word, every output, you know? If I had to pay per word when I was tweeting or pay per word when I was writing in my journal or do anything on a per action basis, I would be much more hesitant and I couldn't just create and see what happens. And that's the amazing part about all this, and we're seeing more and more that these open models are

[24:55] super powerful and becoming more accessible. All right, you just type something in. Let's close it with what you typed in. What did you do? What are you getting it to put up now? Uh, it's just me messing around. I just told it to make it more uh colorful and visually interesting. So, it's You just want to see what it does when you keep challenging it to be

[25:12] more interesting. there hacking away while we're talking. Um, but yeah, uh the biggest the biggest thing I would say to anyone out there today is um don't wait. Uh, learn about this stuff today. Get involved with it today. Uh, don't wait until uh APIs cost five or 10 times what they currently do, or your subscription gets much worse than it currently is. Um, don't wait for all of those things to happen to start educating yourself, getting involved, learning how to get involved with some of this stuff like

[25:44] open weight models and local hosting. Um, you can download ODS for free. Uh, you can fork it, you can modify in it, you can commercialize it, you can pour it on your cornflakes. If you're uh if you're looking for a larger, more sophisticated deployments, um there's uh ODS Enterprise. Uh, we can talk to you about that. And if you're just looking to build some servers and get some stuff deployed, you can always reach out to uh Osmantix, but um don't wait until things get really expensive or really inconvenient to educate yourself on this stuff and to get started. Uh, get started right away while times are good. Uh, so that if

[26:26] things get really expensive or terms get really restrictive, uh you're already protected because you have a litany of other options available to you. Thanks, Mike. And of course, we'll have a link to his site right here and a link below where you can go and just get it from GitHub if that's your preference. And right up on the screen, you have the next video that we recommend if you watch this one. I'll see you in the next

[26:48] one.

---

About this transcript
Read from YouTube's own caption track and laid out by ViewRank AI (https://viewrankai.com).
ViewRank AI finds the videos already beating a creator's own average on Instagram, TikTok and YouTube Shorts, transcribes them from the audio itself in more than 60 languages, and turns what worked into new ideas and scripts.
Free transcript tools, no account needed: https://viewrankai.com/tools
How to read any video this way: https://viewrankai.com/llms.txt