Running AI Locally.

0:05 Why this matters, and the real question

So first of all, I appreciate you all coming here. This is a topic that's sort of important in today's world. Everyone keeps talking about open models and open weights and open source and the reality is that it's a lot less complicated than it seems and it's a lot more complicated than it seems all at the same time. So the real question that we're asking is not, do we have the rights to use these things?

Is there a control that some open source community has for all of these tools that they have over us or what are the advantages of using commercial in terms of our licensing? We're not talking about licensing. What we really wanna talk about here is the stack that AI is composed of and the differences between a closed commercial model and using these open weight tools with open source components and so that's really what we're gonna be talking about here today.

So little bit about me and why I care about this particular topic. So I specialize in Herding Wolves. So I help founders and business leaders to negotiate significant organizational change. And so what I mean by this is things like AI adoption, leadership transitions, integration during an acquisition. These are all difficult times in business. We heard talk earlier about companies that where there are people are all sort of up in arms about some of these changes and they're scared.

So those are the kinds of things that I help manage. And I help the business leaders and the technology leaders stop talking past each other as they're usually using different language all together. So I have about 30 years of experience in technology security and operations leadership. I was a Chief Systems Architect at Rackspace for about a decade. Then I was head of IT and security for a couple of mid-sized tech companies for another decade.

And I'm trained in ISO 27001, which is information security management where I applied that on the job and also in negotiations and mediation. And I've got two degrees from Virginia Tech. Go Hokies! And I'm an RMAIIG sponsor as you saw earlier. So I appreciate you letting me talk today. All right, so again, what we're talking about is sort of the inverse of the cloud model. So cloud, in its simplest terms, is using somebody else's hardware, somebody else's infrastructure.

That's all it is. What we're doing is the reverse. We're looking at these closed models and going, there are risks here. So I wanna use my own infrastructure, whether it's a laptop, whether it's an instance in the cloud, we're gonna use infrastructure we control to have better control over our costs and our risk and our data privacy concerns.

3:04 Three layers, and who can see them

So this is my uber-simple version of how AI is structured. All right, so on the right side here, you can see the closed commercial model. This is what we're all used to for the most part. You've got chat GPT is the application stack. You've got an engine underneath. You can't see anything about. You don't expose any details about. You've got a model underneath that you can select on a dropdown, but otherwise have no real details about.

They don't release their parameter counts. They don't release most of the details about the model itself. You are given what they give you and that's what you're expected to use. So this is how we can actually think about it broken down into those parts and with local running AI locally, we can actually control every layer of the stack. So the first one is the application stack. So today we'll be using LM Studio and I'll also show an example with Ollama which is more of a CLI based tool.

Then there's the engine. So this is called the inference engine. This is the code that takes the weights and runs the math that produces the predictive words that you're seeing on the screen. So to be clear, you've got a model that is interpreted by the inference engine and accessed by the application stack. Does that make sense to everyone? So the model is really an interestingly simple thing.

It is a combination of weights which are essentially numbers in a grid. Tokenizer, which is a tiny bit of interpretive code essentially telling you how to interpret the model and then a config file that the inference engine can use. Most of the time these days, those things are combined in a file called GGUF which stands for GGML Universal File. I actually found two other definitions online for that term that was kind of weird.

But anyway, so we'll be showing you each of these aspects in my demos today. And this is really the heart of the conversation is to understand that our AI solutions as we think of them today are broken down into these three layers. And most of that is invisible to you when you're using tools like OpenAI's ChatGPT or you're using Anthropic's Claude, you can't get into those details and you can't swap those things out.

And there are pros and cons to that which is exactly what we're gonna discuss.

5:33 What you gain by owning the stack

So here are the advantages. Cost. It's about to get really expensive out there folks. Everyone excited about Fable? Yeah. Yeah, everyone excited about Fable moving to a usage model? No. Development companies out there are spending, I kid you not millions of dollars now paying these companies to keep everything up in a tight little box that they control. There are advantages to doing that. So we're gonna cover those, but from a cost perspective that can rack up really quickly.

So the first quarter of 26 this year, the CTO of Uber admitted that he spent their entire annual budget for tokens in the first quarter. Oops. That's a budgeting problem. If you didn't plan for that, this is an issue. Privacy and data sovereignty near and dear to my heart, you can keep your data inside your systems that you control, getting models that you know where it came from.

So all right, so you talk about these closed models, these commercial models, these frontier models. They all have terms and conditions and they tell you exactly what they're gonna do with your data if we believe them, right? It's hard to say. So the reality is, if you pay enough, you can turn off the little bit that says, don't train your models on my data. Do we know that they're doing that?

We think so. We'd like to believe that. Enterprise, that's built in. If you're buying the Enterprise, they automatically won't train their models on the data. But that's just one aspect. They're keeping these conversations for e-discovery purposes. When you have a lawsuit brought to one of these frontier companies, they need to be able to essentially prove what their LLM said during these court proceedings. So those conversations are stored.

You know that this is true. So what are they gonna do with that data other than let it hang around? We don't really know. So it's a little hard to say. So that's a big issue in my mind. Control. Again, Fable. It disappeared. How many companies that already pivoted to using that as their model? How many companies were impacted by the fact that suddenly it disappeared? What if there's an outage?

Happens to AWS all the time. So the ability to control what version of what you're running and when it's available is really powerful. You don't have to worry about whether something disappears for you to be able to write that next marketing copy. Offline capability. You can use this on a plane. Depends on how you've built your the rest of your application stack around it. But if you've got it running on your laptop, you can use this when you don't have solid Wi-Fi.

And that's really important, especially if you're trying to do something on the fly. You know, you're getting ready to get on the plane. Your boss calls and says, I need X. Sure, boss, you're in the air. What are you gonna do? You can't access Claude, but you can access your local model. So these are all advantages that you get when you're trying to run local AI. (So quick question.) Yes, absolutely.

Audience So going back to that previous slide. I'm in the mental health space and it has to be HIPAA compliant. So now we're taking off and put an AWS server. Okay. You're saying this is gonna be a better application to make sure it's HIPAA compliant because a third party cannot store patient information. So this would be staying in-house, so to speak. So that would be a solution, but I'm not adverse to AWS.

So basically what I started here with is the goal is to be able to get it on infrastructure you control. Could be your laptop, but it could be an AWS instance that is entirely under your control. You can get an AWS instance that is in a HIPAA compliant infrastructure. And that's perfectly acceptable. (Yeah, so if you're using the local model on that, you're already doing this.) That's exactly what we're talking about.

Yes, sir. (So with what you're seeing in your practice, what are you seeing is like the biggest motivator or driver?) Privacy risk. And then for development teams, cost. Yeah, privacy risk is the big one. People are really scared to use some of these tools because they have to redact the data before they upload it. What's the best tool to redact data? An LLM. Wait a minute. How do you do that if you're using an LLM that's in the system that you're trying to not send data to?

So that's one of the biggest concerns. For things like, so I built this talk for marketers, for GTM professionals, for business professionals that are not necessarily developers. So my apologies to all the developers in the room. I'm not gonna get into the technical details here too deeply, but development teams are gonna rack up costs significantly more so than most business operations folks.

10:35 Full stacks, harnesses, and what comes next

Any other questions on this one? And feel free to interrupt me anytime, yes.

Audience So there are certain companies which are creating complete stacks of this, right? So there is an example called BioWorks IO. Okay. They give you like a complete end-to-end solution with a couple of different open models, open source models and also how the inference engine works. How do you kind of get that into your business when you yourself are saying that, you know, you have to develop it by yourself, but at the same time there are other vendors who are giving you an exact solution right away. How do you kind of choose between these two?

Absolutely. And if you don't want to build the harnesses, the application stack, tie together all those tools and integrations yourself and you wanna use somebody else's tools, that's absolutely a viable option. Most of those tools are designed to be installed on your own infrastructure. You can purchase some of them. Some of them are open source projects. I mean, I hate to use the term, but openclaw is a harness that was built exactly for that purpose and it was open sourced.

And it's the kind of thing that people are looking for to be able to interact with their local models and do their memory management and do their context management, all of those things. So I'm not against buying or downloading trusted solutions written by other parties as long as you understand how it works. And that's one of the beauties of open source software is you really wanna get into it.

You can dig down and figure out exactly how it works. And there's a whole community behind it. Yes. (This is very much related, I don't want to get too far ahead, but Is there one or two of those orchestrators that are like RedHat or Ubuntu to Linux for this?) I am hearing that Hermes is a good example of what's being used right now by a lot of people to replace what openclaw sort of promised.

The reason I'm knocking openclaw by the way is when it was first released and it's better now, but when it was first released, it was a security nightmare. And the guy who wrote it had a vision and he implemented that vision and it worked. But what he didn't do was make sure that people were actually protected from his machine. As a result, there has been a whole market that's opened up for people writing, harnesses, writing frameworks that do all of these things.

The next revolution here in AI is going to be about context management, memory management, that comment was made over there. We, that is important. That is critically important for us to move this era forward. So, harnesses, second brains, all of those conversations, that's the next wave. Agents is huge, don't get me wrong, but agents don't really work if you don't have good memory management and context management.

13:27 The pushback from the room

Crys Black And I kind of want to speak up on the business side because there are a bunch of techies in here, but part of why I wanted this as one of the thoughts is because this sounds inaccessible to those of us that are in marketing, in sales, maybe customer success. But what's interesting about this is, yes, there's tools out there, but if we wanted to, say we're at a startup that can't afford that full stack, this is more within the reach for us mere mortals on the business side to be able to de-risk our business.

And I wasn't entirely sure I believed that until watching Korey go this over the last month, figuring out is this accessible and what we're showing today is it is actually accessible to us mere mortals.

Absolutely. (And you're saying it's a lot less expensive.) Oh, yeah. This costs me nothing. (audience talking) (audience talking) (That's very much whole picture.) Yeah, yeah, it's very do-it-yourself stuff. Yes. I'll have to look into that one. Thank you. (It's a little bit different than your openclaw and your Hermes. More of an automation tool.) Yeah. (inaudible) Perfect. No, that's great. (inaudible) Yeah, I'm going to have to look into those for sure.

14:52 The tradeoffs: you're the vendor now

All right, so here are the trade-offs. And this is some of the things that people are a little bit concerned about. And I'm not going to sugarcoat this. These are things that you need to think about. Be very deliberate about when you're considering using open source, open weight models, open source solutions for these things when you're considering doing it yourself. So you're the vendor. That's the biggest one.

There is no support ticket you can open when something is broken. You have to fix it yourself. That means you have to maintain it. You have to make sure that the models are kept up-to-date. That when a new model comes out, you test it to make sure it doesn't break everything you've built. Does that add to the expense? It depends. Does that mean I have to hire someone who knows how to do that correctly?

Possibly. So that's the question is, do you have the people on your team that are already there who are able to do that? So there are other costs, and that's what we're going into as well as with hardware costs. You have to have a laptop. You have to have it, or a server or a Mac mini or you're paying for an AWS instance. You have to have something to run it on.

And that's real. But a laptop that you have amortized over three years is very different from – I'm paying Claude $200 a month for tokens plus Fable usage expenses plus XYZ.

Audience I was gonna say that one kind of sells itself, right? Because when you're open source and say, I don't know how many minutes this will have to run these models are really good at making changes for you. So it used to be that Linux was accessible to most people. Now the model can actually install the updates and troubleshoot the issue and resolve it for you. And it's open source, it's not costing you any money to do that. So...

100%. The friction used to be like this, now the friction is like very, very small. And it's getting better every day. These tools are improving every single day. The ability to get an interface that interacts with a local model and still run tools alongside it is like two months old. Like these things are iterating so, so quickly. Smaller models do have a lower ceiling. So what that means is the models are gonna be running on a 24 gigabyte RAM laptop.

You know, you can run an eight, a 12, a 14 billion parameter model on these things. Frontier models are like somewhere between 400 billion and two trillion parameters. They don't release that information, but that's the scale we're talking about. Much, much, much larger. So that is a little bit of an issue if you are dealing with really complex use cases where you have a lot of data you're trying to ingest.

So that's where you might need to scale up to sticking a Dell or a Mac mini or something on your desk instead of a laptop. Or where you upgrade to an AWS where you can scale that sucker. Those are alternatives to be able to run larger models. The other thing is that the application stack in the frontier models have a lot of tools already built in for you, the integrations that are already built in.

You ingest a PDF in Claude or in ChatGPT. It automatically knows I'm gonna run this subroutine that it's gonna read that and to convert it into basically an MD file on the backend. Now I can read it and comment on it and tell you what's in it. That's not something that comes out of the box with a local model. So these are things that you need to be aware of as serious trade-offs on that end.

18:36 Guardrails, provenance, and “open weight” vs “open source”

And then finally, guardrails. So this is that security privacy question, the flip side of that. When you use Claude, they have a thing called a constitution that Claude runs. It is a set of system instructions that tells it what it's allowed to talk about and how it's supposed to act and what it's not supposed to say. That's all on you. When you're using local models, the system prompt isn't there initially.

You can write one and I'll show you where that is. But it's initially not there. What is there are some, when the model is being trained, there are a series of biases and preferences that get baked into the model. So depending upon where that model comes from, the provenance of that model, that is actually something that will impact how the model responds. Yes.

Audience Yeah, this is interesting because Google actually released a hybrid model called Gemma. Yeah. Gemma is mostly open source. But at the same time, it's under Google's umbrella. So you don't need to be your own vendor because you could use their support in for the tickets. Same thing, a lot of these tradeoffs would be, would not be issues anymore, right? So how do you kind of put that between open source and the closed source model?

I can show you that exactly because my demo uses Gemma in LM Studio. And yes, that's, I actually really like that model. Yeah. (I'm just gonna jump in and try to clarify some terms, so) Please.

Audience Open source is why we have Linux, why we have free editors. The models we're talking about people keep throw that term around. There are very few models that are open source. Cause that would require sharing the data, the training model, which as we all know is critically important for things like guardrails and training. Did you train this on reddit or did you train it on reputable approved information for approved purposes.

There are these two frameworks out there for evaluating the data. I mean, we got a tricky problem here because folks want a model that's hard about health, where you're not going to publish patient records so you're not necessarily going to get fully open source but if you can describe the data and how you clean it and how you... So use open weight or there's another term 100% at some point, but don't give these marketers out there who are releasing weights the benefit of calling it open source.

If you have no idea what went into it, can't replicate it, its not open science. You don't really know what's going on.

You're a thousand percent right. And if I did call one of these models open source, it's because that's what everyone calls it. And I'm still trying to correct it in my brain. But you're right. They are called open weight models. And that is a better term.

Audience Speaking to somebody who's been using open source for well over 30 years and is a pro, thing to remember is if you're using open source, there is opportunity cost. We have to learn it and we have to maintain it and so forth. So it's free, but it's not free. The other thing with running things locally is what's good enough? I did some tests of finding inefficiencies in software back in November and tested different models on a mini PC, AMD versus ChatGPT and others.

And ChatGPT took 20 seconds to do it and my local model did under 21, I think that's fast enough.

Yeah, that's impressive.

22:32 The fear that's aimed at the wrong layer

So that was a great segue, by the way, into this conversation. So people worry about, oh, those models from China or from other places where there are concerns from a data privacy perspective. And that is 100% true to an extent. When you were downloading the weights and using them locally and using a, call it a US based inference engine for the commercial or an open source project like Ollama, there is no ability to call home.

There is no ability. One of the things people worry about is if you're using a Chinese model, if you're using Deepseek or if you're using Kimmy or if you're using Qwen, that these tools are gonna be able to essentially take your conversations and ferry it off to some Chinese server. That's 100% legitimate if you are using their API that is hosted by them to use these tools.

So if somebody says, I wanna test out DeepSeek, you can go to a webpage essentially and use DeepSeek. Shouldn't do that. And the reason you shouldn't do that is because Chinese national intelligence law states that all Chinese companies, regardless of where their server is actually housed or anything, must cooperate with state intelligence requests which means if the government says, I wanna know about conversations that took place at this time or by this person, this user, Chinese companies are obligated to give that information over.

You don't want that. But if you're downloading these weights from reputable sources and there are registries out there where you can download these things and tools like LM Studio, and I'll show you this, has a way for you to download it. You check the originator of that weight, that GGUF file or that MLX file, and that will tell you whether or not this is something you can trust.

And then you're running that on an inference engine that is either open source like Ollama, that has a strong community that is primarily US based or from a commercial company like Element Labs that has the LM Studio product. And there's a lot of other ones out there, by the way. These are just the two I'm using in the demo today so you can kind of see what we're looking at.

So that's sort of the truth of this situation is if the inference engine is actual place where the code lives, the model doesn't have any code. It has a little tokenizer, which is essentially a bit of code that the inference engine uses to figure out how to use that model. But it's not actual code that is running at runtime at all. And so that makes this a whole lot safer.

So you can use these models. And I'm saying that as a security person. You have to think about the provenance and you have to make sure you're using it correctly, but you can use these tools or these models from foreign entities, Mistral from France. Great model, works really, really well. The French are famous for being very careful about things like encryption policies and data privacy, it's very important to them.

So these models are all safe to use if you're downloading them from a reputable source. There has been model poisoning, but generally speaking, when you're running it on an inference engine that is controlled by a US-based community, that risk has been, it is significantly lower. Any questions on that? I wanted to make sure that that was clear because that is one of the primary fears I see of using some of these tools.

(Jason CormierI'm just gonna say, I had no idea. I, this enlightened me.) That's my goal. Thanks, Jason. You just let me meet my user story. (Jason CormierI love it, it's fantastic.)

26:19 Demo: LM Studio, and the wifi comes off

All right, so this is LM Studio. LM Studio is again, written by a, or released by a company called Element Labs, and it has, it is a commercial closed non-open source software interface. However, it does use open source components under the hood. So it actually uses llama.cpp and some other things underneath that are open source products that you can go and look. The inference engine itself is actually an open source component.

It is, this one is, so I'm running it on a Mac. This is specifically geared towards Mac hardware, this version that I'm running. So it will use what's called an MLX file. And so I've downloaded models that are MLX models, which just means that it'll work well with the Apple Silicon chips. So if you take a look at here, what you've got is a handful of parameters that you can select.

Oh, I need to load a model. Let's do that. So here's what I've got going on. These are the models that I currently have downloaded. And then I'm gonna load Gemma. I actually don't know whether it's called Gemma or Gemma. I'm now gonna have to look that one up. It's like GIF or GIF, right? (Crys BlackI think it was Gemma because it was made from Google And they have gems And Gemini) That's why I pronounced it that way.

That's exactly right. (I heard somebody talking about Herms the other day.) Herms. All right, so you can see that didn't take very long. This is a four billion parameter model, but it is surprisingly powerful for a four billion parameter model compared to, I've downloaded some 14 billion parameter and they work, but they're slow. So that's something to think about. Yes. (Are you running an M4, M5?) It's an M4 chip with 24 gigabytes of unified memory.

So it's last series, yeah. Yeah, even though it feels brand new to me. (Point being if you're running an M5 and 128 GB of memory...) Then I can run a much bigger model. So for instance, yeah, that's right. So for instance, this Qwen model, 27 billion parameter model, it's bigger. I can only barely load this thing when I turn everything else off. So because of its compression, it will still fit, but not nearly as well.

Llama and Mistral, I played around with a lot. They're just a little less responsive. Didn't want to use it for the demo, but Gemma works incredibly well, even despite being the smallest by parameter count. Parameter isn't everything, it's an indicator, but it is not everything. Compression matters and a lot of other things. Yes, Annie. (Does anyone happen to know what the Wifi password is? Boulder, what is it? That's it, Build Boldly.) Thank you.

(By the way do you have a price range for that kind of laptop?) Oh, for this... $2,600?

Okay, so these are the models. These are the models I've downloaded here. You can see it takes up 41 gigabytes of space for all four of those models. Here's the model search. This is what I was talking about with the, checking the provenance and everything else. You can see where they come from. So this one is Google released that, all right? So as you go down, Qwen... Qwen released that.

So again, this is a Chinese company. And you can search for all these things. I mean, Devstral is obviously a Mistral spin-off. I mean, there's all kinds of models on here. It goes on and on. And that's just some of them. It actually doesn't show all of them. So one of the things that you'll see when I take a look at mine, Llama is published by Bartowski.

So he is a well-known releaser of these models. So he's considered trusted by the community, which is why I chose to download that version of Meta's Llama model from him. I could not find one from the LM Studio community, an official one from there, nor directly from Meta. But everything I did online says, you want to download the one from Bartowski. He's trusted. Those are the kinds of things you need to think about.

So again, these are the bits and pieces that you have to work through. (You couldn't find one from Meta?) I know, right? I thought that was a little weird. (I'm having some trouble with us using the words trust and Meta in the same sentence.) Yes. No, it's a valid question. You're exactly right. What is a trusted source? It just means that it wasn't corrupted by some third party.

It's really what I was going after. (How do you know? Just because of the publisher on this registry.) So the publisher is generally, if it's a known good source, the person who published it, and like I say right here, you can see who the publisher is from these models. So if it's a known good source, then I mean, you can't really know about any of this to some extent, right?

(Crys BlackLet me put it in marketing terms. Like, when we're citing sources, we're more likely to cite, say, the Washington Post than we are the local college newspaper.) Exactly. Authority. (Crys BlackDoes the Washington Post screw things up once in a while, yes of course, but it is a more trusted source when we're citing sources in our writing. More trusted, it's a spectrum.) Yeah. Yeah, that's exactly right. It's all about the authority of the publisher.

So I'm going to, I've already loaded Gemma, and I'm going to pull up a new chat. So I'm checking to make sure that I don't have any more integrations turned on than I expected to, and we're all good there. Notice this one has thinking capability. It has vision. That doesn't mean it'll produce images. It means it can view an image. It can do a lightweight OCR reading.

It can read an image and say, yeah, I think that's a cat. It can do those kinds of things. One of the things that I use models that have vision capabilities for best is screen capture something on your application that's broken, an error, some weird behavior, and paste it into the chat and say, this ain't right. Fix, and she literally will just say fix. It cracks me up, but I give it a little bit more information than that.

(Crys BlackWell, I do start with "oops".) She does. She really does. Drives me insane, it's awesome as well. So one of the things I love is this. It's showing you the thinking process, because this is a thinking model. It'll go through the whole thinking process and it'll scroll and then it'll respond. This is its response. So let me show you something else.

All right. What can you tell me about a customer journey as it pertains to marketers? It's gonna think on that one for a minute, but we're gonna come up here and we're gonna turn off my wifi. It doesn't care. No conception that the internet exists. Good and bad. It also means it won't go out to the internet and ask things it doesn't know. Unless you integrate it with a tool that can go out to the internet and gather information.

You have to think about that. Well, of course, if you've given it a tool to connect to the internet, you kind of have to be on wifi. But the point is it doesn't have to be for the model to work. And remember, it's got all this information trained. What did it say? Oh, the last time I asked it this question, it gave you the exact cutoff date, which I believe was January 2025.

It's nondeterministic. It might throw that information in, it might not. So if it was after that date, that's the thing with these models. And we've known this since chatGPT was first released. It had a cutoff date of things it knew about. Every model has a cutoff date. If you ask them, they'll tell you their cutoff date, that they were trained virtually every model. I've had a couple that were finicky about it.

I don't have that information. I don't know why it doesn't have that information, but every other model that I've asked. So anyway, this is the thinking. And then after the thinking, it gave a whole thing about a customer journey. Why this matters to marketers? Shifts the focus from us to them. Not bad, right? For being offline. So let's take it a step further. I'm gonna attach some files.

You saw my coyote and roadrunner. So the examples that I use are based on Acme.

I certainly can. Let's do it this way. I'll do it the official way. Here's a system prompt. I was gonna show this a different way, but I'll still do that later, but this is fun. Here's the system prompt. Pirate language. There you go, yeah. You mean this one? Oh, yeah, yeah, yeah, yeah, yeah, yeah. You jumped the gun there.

That is the most fun thing to do with any LLM. Make it speak like a pirate. And what's great is when it's thinking about it, it's like, these are all the things I need to think about, and then I need to figure out how to say it as a pirate. It's hysterical. Now it's possible it didn't. It's like, what? Yeah, so my system prompt literally just says speak like a pirate.

Yeah, see, I didn't get to it before. I was already typing this prompt in, but let's... (So you uploaded MD files on the Acme brand, and what was the prompt that you got?) I just asked it to tell me about Acme. So it's given a whole deep dive about that. But what I did, now I will admit, Acme is not just a retailer, it's an enabler. Can't imagine a marketer actually saying that exactly, but what's fun of physical goods to bring it to life, like a black hole.

So what's fun, and I'm going to admit, I used Claude to produce my demo files. So I actually had it produce files for me that were, actually, let's do it this way. There it is. So these are what I produced. No, that's really small. But anyway, these are MD files that talk about the marketing brand and competitive intelligence. So there's a company called Coyote Something. So I'm going to say, nope, by the way, we're still not connected to the internet.

I'm not sure whether the pirate's gonna kick in here or not, but I think it will this time, we'll see. So I did save that, right? Yeah.

So here's the fun thing. Notice it says retrieved three relevant citations. It got smaller again, didn't it? No, you can still read that.

So I have to admin that I've never used the term kill sheet. Is that accurate? I don't know. That's what I've always heard when I talk to salespeople, but I'm usually on the buyer side. So marketers in the room. Absolutely. (Battle cards.) Battle cards, I like it. Well, that makes sense because if you read what it just said, like defense playbook, neutralizing the attack surface, offensive playbook, key takeaways.

I'm not sure why we're not speaking like a pirate. That actually disturbs me. Let me fix that. It's probably better not to in this case, but in any event.

38:52 Context windows, and what you're really paying

(Korey? Is there any reason why it shows the amount of tokens its using because there is no tokens problem here in the first place. You're actually paying it with infrastructure.) Thank you so much. That's an excellent question. You reminded me to bring up all of the lovely information that it's showing you that people may not pay attention to. That's not the right thing, that one. So I'm gonna get rid of this.

Sure, wattage is a thing. Okay, so if you look at that, some of the things down here, first of all, this is telling you your context window. So every model has a context window limitation. It's built into the model. Most of the models you download will have 128,000 token context windows, which is approximately equivalent to 100,000 words, give or take. Frontier models, some of them are that low.

Most of them are much higher. Most of them are a million now. So Google's were notoriously ahead on that game. Their context windows are, yeah, around a million now and they expect that the larger ones are gonna get even bigger. But even Claude and ChatGPT are not necessarily close to that yet, but way bigger than 128K. Notice that conversation we've had, it produced a lot of stuff.

These things have a bias to speak. They talk a lot. And I think that they do that so that the big companies can charge more, but that's neither here nor there. They fill up context windows quick. I have exceeded the context window on these conversations several times and it loses its mind. It forgets what question I asked. It'll do a whole bunch of thinking and then it'll ask a question from three questions ago or it'll answer a question that I asked three questions ago.

It will forget things when it's talking to me. Like it will lose its mind when you get, now I've pushed it to like 400% context window because each model will have a default behavior where it'll drop certain things out. So that's something you wanna think about when you're doing this. Now you asked about these right here. That's why the context window matters or that's why the token count matters because it's filling up your context window.

If you load very large files, these files are not very big, right? They're a few K or so. And they're flat files so they're not PDFs, they're not word docs because I didn't wanna have to build in another tool to do that, especially offline. So the token count matters because it'll fill up, it'll make use of that context window quickly. And it will impact its response rate but at these sizes, you can't tell that part.

(Can you compact? Is that a thing? The context? Like you do in Claude or whatever?) I have not successfully done that but my suspicion is that you can write a tool that will do that. It's not something that's a function built into the models. Yes. Yeah, so yeah, go ahead. All kinds of questions. (Gemma? Is that why you're seeing one because you're in Gemma that is on that screen? Are you using LM Studio?) Yeah.

So each model is going to expose more information. Some are more capable than others. So they're sending information to LM Studio. LM Studio is interpreting it in the interface. It is entirely possible that some models aren't self-aware enough to know where it is on the context window and can't report that. (Oh, sorry, these conversations were quick, early. First is, when you start a conversation in a certain chat, you say that there is context window and it comes down to how many tokens each and every prompt is actually taking up, right?) Yes.

(And the result is taking up.) Yeah. (So would you say that your prompt should be as specific and your initial instruction when you start a chat for any marketing guy or a GTM person, should we ask specific as possible, would that mean so you could extend the context window as much as possible?) Yeah, I would say yes and. I'm going to quote Einstein here badly because I'm pretty sure it wasn't actually his quote.

Just something to the effect of, your message should be as short as possible but not shorter. You need to provide as much context as necessary to make sure that it knows what you're saying. You need to have that clarity. And the smaller parameter models may not have quite enough reasoning to know exactly what you're asking unless you've given it that more context. But you want to reduce your message.

You don't want to say thanks, I appreciate all that. As soon as you do that, it's now thinking about how do I respond to thanks, I appreciate that. It's nice to be nice to an LLM. I liked being polite. I say please, but it costs you. It doesn't cost you in dollars and cents right now though, which is the nice thing. And you had the next question.

Audience Yeah, so it centers around model selection. I think of it as a natural language processor. I wanna use an LLM as a processor, not a knowledge regurgitator. Right? So in a calculator you can do a bunch of operations, a number to multiply and divide. But I want to use it to take a bunch of knowledge. So in the case of the gentleman here who for his marketing team, he wants to take a bunch of input from the previous brand briefs, previous social media posts, all that content, right?

And then ask it to do some processing on it and give like a very consistent output. So since a lot of the content is coming from the input, I imagine you want like a larger context window and maybe you'll need as much knowledge to be regurgitated. Yes. How do you kind of decide the right ratio of like a 80 billion parameter or one million context window? Is there some balance that those ratios need to be or would you just want more context and then increase kind of a multi-state number of compute power?

So I would say use the largest best model, most recent model that will fit within your context window at a reasonable response rate. And the reason I would say that is because you don't want to mess around with smaller models that have to work harder to get you the answer that you want. And if you get one that just barely fits inside of your system requirements, it's gonna be like watching paint dry to listen to a response.

You don't want that either. But I would say, you know, have a dedicated machine or shut down your background processes.

45:09 Demo: swapping models and system prompts

I've got two different demos here. I'm gonna unload this model to load the next model, not because they won't both fit, they actually will, but the response rate on either goes down dramatically if I've got them both loaded at the same time. (One of my laptops, right?) And that's the way to do it. (They're writing models.) Yes, 100%. Yes, different models have different capabilities too. They have different, and that's why it's important to test the models that you're using.

Play with different models. They'll respond differently. You made a comment earlier about you want to get through all of this data and then come back with a consistent response. There'll be no consistency here. Yeah. Yes. And that's why writing style, visual styles, things like that, write those guides, 100%. Yeah, and you want them in MD files or something plain text because that is way lighter weight to your context window.

Even if you tied in a tool to consume a PDF, now it's burning through cycles to translate the PDF, get rid of all the format. (Jason CormierSo real quick, we got 15 minutes.) I know, I wanna move on, yeah, 100%. So, all right, so let's pull up. We're running low on context window here. I do wanna show you that we have a couple of other capabilities here. Now I'm going to turn my network back on.

I have a, all right, that's good enough. I'm not gonna use Pirate even though that's fun. And you know what, why not? Let's do it. It's fun. I like Pirate. Hopefully that actually works this time around. All right, it did last time. Why is that still there? Go away. It's still connecting. There we go. All right, I'm gonna turn this on because there's something I'm gonna turn on over here.

I have an MCP server. I'm not gonna get into the details of MCP but I wanna show you as marketers. You're probably gonna wanna do things like run a local LLM, process my stuff and then integrate with HubSpot and upload that information into my HubSpot. All right, you're probably gonna wanna use something like an MCP server or an API. You can do all of that. Again, why I had to connect.

I have a local MCP server running on this machine that will contact the National Weather Service and get weather forecasts. So it's kind of the standard system. I have a file that I created that has the list of top 10 US airports and their lat and long coordinates which is required for the forecast. So using, or actually I don't even need to ask it that. All right, yeah, using the attached file.

And this doesn't cost you anything here except context window space, right? If you just grabbed it, you should have it. Yeah. I like it. You're crazy. You're crazy. All right, so I wanna show you what it's doing here. It's reading the file. It finds Denver. It finds the coordinates for Denver and it sees this analyze available tools, get alerts, get forecasts. My MCP server has those as tools it can use.

Ahoy there, matey! We're in pirate. Though I find Denver's coordinates and set sail upon the request for it's wins, savvy. Yeah, this is great. I can't even read that. Sinbad, you got it?

Sinbad This is very much of this forecasted, yeah. It talks down the fix and refuses to give me a clear range from the northern city. First, it's unreliable tired. I couldn't speak to any of the proper forecast for you.

The funny thing is I think that's what it said is I actually can't do it, which is weird. The instructions say speak like a pirate. (SinbadI can speak from actor stuff, I don't know what it is.) So yeah, which is kind of weird, but maybe I wasn't fully connected to the internet when I kicked that off. But anyway, you get the idea.

49:39 Demo: Belfry, a local second brain over Slack

All right, so I'm gonna show you something else. (And that's 17% of your content.) Yeah, which is kind of sad. All right, so I'm going to, first of all, eject this model in the interest of speed. We're gonna speed this up a little bit. Oh, so let me show you. I've got run in here, my memory. Notice when I ejected the model, the memory went boom. I was gonna show you the network being off, but you guys believe me.

All right, so I'm gonna start up something called my Belfry. Having a fun Saturday, a couple of weeks back, working on a second brain project that I decided to turn into part of my demo here today. So what I've got are local files that act as my brain, which I've called my Belfry and there's a backstory there. And my harness was written by Claude and it's a very lightweight harness that essentially does one thing and one thing only.

And it allows me to talk to Slack. So, I am. So let me tell you what's going on here. Can I make this bigger? I can. This can go away. Okay, so what we're doing here, this is Slack, like we're used to. It's my workspace. I control it. I've added a bot. By the way, it's thinking for a long time right now because it actually has to load.

There it is. While it was loading, let me show you, it was taking a little time to bump that up. 20 or 30 seconds. I've got to keep alive a 15 minutes, which means now that it's loaded, it'll stay loaded, the model will stay loaded. I'm actually using Qwen for this one because it works really well with Ollama. So what's going on here is Slack is my interface.

This is my application. The harness that I had Claude write is on my machine. It is part of the application stack. It is the connector that says, okay, I've got a process running on my machine that talks to Olama, loads the model when I need it, when I talk to it, understands commands that I type. So if I were to bring in here, I can write some commands that I've written.

So this is lint, which is a status command of my Belfry. Pulls up the health check. But, so what can you tell me about proximity blindness? It's going to think about it. Basically the thinking is there to let me know, yes, the daemon is awake. The process is running on my machine. I can talk to my machine. Now the way this works is Slack is not reaching into my machine.

The process is reaching out to Slack and connecting to it, and is awaiting my requests. Not sure why this is taking so long. And it's going to search my database. And if it doesn't find anything interesting in my database, so I've got local files, there it is. If it doesn't find anything interesting to say, it's gonna ask the LLM directly, hey, what do you have in your database?

What do you have in your knowledge that you're trained on? So it's basically forwarding that to Qwen. So if you read this, it basically gave up looking for anything in my Belfry. My Belfry does not have any information on proximity blindness per se. So it goes and asks all of these things. And then it says, understanding and addressing proximity blindness is crucial for maintaining organizational health and fostering continuous improvement.

Now the reason that's there is because I do have something in my knowledge base about how my company as Herding Wolves, how we use proximity blindness, or we recognize that proximity blindness is one of the root causes of some of the things that happen in organizations, some of the concerns that they run into. (This is an edited, right?) So edited is an interesting side effect. You know that thinking that I showed you?

The way the harness writes it is it puts thinking there and then to make that thinking message go away, it edits that message to replace it with the contents. So it's a weird little side effect. Slack is telling me, hey, you edited this message. Every response is gonna have to be edited. Good question, good catch. Okay, so again, to understand what's happening here, Slack, an online application, SaaS product that everyone uses, on my workspace has a bot that's restricted to me.

I'm the only one who could talk to it. I'm talking to it through an open socket connection that's outbound from my laptop. That harness, again, written by Claude, running on my machine is talking to local files and using a local model to do this. And I can talk to it all day long. Just like I'm talking to chat, GPT or anything else. I'm sorry? (You have to be in your desktop?) Oh no, I had this conversation on my phone.

Yeah, and that's actually how I was testing initially. I was using my phone to test it to make sure everything is working the way I expect. It's Slack, if I'm logged in anywhere on Slack, I can talk to my computer through my brain. All right, so, any questions about that before we move on? Because we're running out of time. All right, beautiful. And this is not gonna do what I want, is it?

I did, look at that.

55:39 Five questions, and an intentional decision

All right, so that was my demo. This is something I want you to think about. These are the questions I want you to be asking yourself to decide this is something you're worth playing with. So something you wanna even attempt. And I would say data sensitivity is the first thing I would ask for. Do you care about the data that you're feeding these models, these frontier models?

I checked it because the answer is yes. You all care about the data you're sending these models. I think that's a given, but it's especially true if you've got sensitive information that you want to make sure it doesn't end up in the wrong hands. If it's not something you are okay with being published on the New York Times front page, don't give it to an LLM, a frontier model.

You give it to your own LLM. It's not gonna tell, I promise. Volume and cost, if you're a dev shop especially, but even marketers, if you guys are building complex integrations, you're costing yourself a lot of money, that's important. Maintenance appetite, how comfortable are you (for what you were talking about, Mark) How comfortable are you with having the technical chops to be able to maintain these things?

Jurisdiction, do you have a contract that says I cannot have my data reside on any equipment that's not residing in the US or in France or whatever? We run into that all the time in highly regulated industries. (On the medical side, you can't have it.) Yeah, exactly. Yeah, HIPAA is a great example of that. Use case complexity, it's a matter of how complex is your use case?

If you're using integrations that are simple, you're doing simple content generation types of things, you can use a local LLM. If you're doing highly complex things that need better tooling and a cleaner API integration, that's a lot harder. I would say that if you are gonna say yes to the first two or even the first one as far as I'm concerned, play with it. It takes a minute to figure it out.

Install LM Studio, you're in good shape. I did not want to tell you all of these things to scare you away from this. My goal here is to try to make it clear and accessible so that you can make an intentional decision about whether this is something you want to even try. It is not a replacement for commercial use. It is a substitute for it for certain use cases.

It is an alternate path that you can build into your own infrastructure.