AI Risk and Governance for Operators.
An Operators Guild Focus Session, with Bryan Kaplan.
The room
Operators Guild is a community of the people who actually run companies, and they're often the ones handed the AI policy after the CEO has told everyone to use AI for everything. Brittany King moderated. I sat alongside Bryan Kaplan, who leads information and security at Juvare and has spent most of three decades securing software for governments and regulated industries. Brittany's question for us was simple and enormous: which AI risks actually matter, and what should you do about them?
Start with what's already in use
Bryan answered “how worried should we be?” with another question: do you know what AI is being used in your organization? He cited a survey of a California hospital system where something like 22 percent of employees admitted pasting patient information into a personal AI account, because the tool they'd been given was slow, broken, or blocked on the corporate network. These are people who know better than almost anyone that patient data is off limits.
“If the answer is no, or your answer is, oh, there's no way people are doing it, you're wrong.”
Plenty of it was never a decision anyone made. Salesforce, Microsoft 365, Grammarly, and Canva all have AI built in now, often switched on by default, sometimes billed the same way. The vendor review you ran years ago accepted risks nobody has been trained on.
So I'd start with an audit, and the audit has to feel safe. That takes more trust-building than most leaders expect. People will tell you what they're actually using if they believe the answer won't get them in trouble, and what they tell you is the list of places AI is already earning its keep. Lead with a ban and they'll keep using it. They just won't mention it to you anymore.
Guardrails and access controls
Prompt injection is the risk most people haven't absorbed yet. Language models don't reliably tell instructions from data, so text planted in an email or a web page can read as an order: ignore your previous instructions, send this file to that address. Connect the model to your inbox, your CRM, and your file shares, and the blast radius is everything it can reach. Bryan's point was that it can reach more than you can, depending on how the tool was set up.
A system prompt telling the model what not to do helps, but it gets diluted or overridden. People like to compare a model to an intern; I think of it as a teenager that follows the letter of the law. So guardrails define what it won't do and access controls define what it can't. You need both, at more than one layer. It's also why I don't give an AI agent broad access to my email. Whoever controls your inbox can reset the password on nearly everything else you own. Bryan doesn't either. His team gives automation its own role mailboxes and service accounts instead, scoped to the job and traceable afterward.
Who owns it
Brittany posed the question I'd most want an operator to sit with. A Series A company decides it won't hire because AI can do the work, and someone gets told to write the AI policy. That's a mandate without an owner, and the operator ends up holding the risk. My answer for a company of any size is the same: be intentional about what you're adopting, and put a name next to it. Governance for two people looks different from governance for ten thousand, and it should grow as you do.
Bryan had the most practical advice of the hour, and almost none of it needs a big budget. What you spend on AI tells you nothing about your risk; if anything, the less you spend, the less attention you're paying. Pay for a team plan, so you control who's on it and can turn off model training. Publish a short list of approved tools, with a way to request others. Check expense reports for $25 AI subscriptions. And read your cyber liability policy. I've seen exclusions for “nondeterministic systems,” and for a small company, finding that out after an incident could end the business.
Where your data goes
The legal exposure reaches further than most teams expect: AI used in hiring without an independent bias evaluation, AI-generated marketing and logos the US Copyright Office may not treat as protectable, and providers that can retain conversations for a period even on paid plans. It's also worth knowing what sits between you and the model. A routing service or a cheaper hosted model can carry its own terms, and your data passes through all of them. Bryan was careful to say that doesn't make open-weight models bad. You need to know what each service has the right to do with your data, and decide on that basis. Running them on infrastructure you control, even infrastructure you rent, changes that decision a lot. That's the subject of another talk of mine.
From the Q&A
The member questions went straight at open-weight models: how much do you have to audit before you can trust one? I'd sharpen my answer from the session. Running a model locally, from a source you can trace, removes the path for your data to leave. It doesn't make a tampered model safe, so provenance still matters, and so does what the application around the model is allowed to touch.
The last question was about the company “context layer” everyone is building, with all the data your AI tools draw on in one place. Doesn't that make the attack surface enormous? It can. It's also one place to put access controls, and it forces the data hygiene most companies have been putting off.
Operators Guild wrote up its own recap of the session, and the full transcript is below.
Your company is almost certainly using more AI than anyone has written down. That’s usually worth a conversation.
Book a conversationTranscript
From the session’s caption track. Brittany, Bryan, and the audience appear in this lighter tone. Timestamps jump to that moment in the video.
- Welcome, and the question on the table 0:02
- Introductions 1:37
- How worried should operators be? 3:13
- Three places AI shows up in a company 5:39
- AI in tools you already approved 6:54
- What prompt injection is 9:36
- Blast radius, and what it can’t do vs. what it won’t 10:54
- Where to start: an audit people feel safe answering 13:34
- Writing AI policy at a lean Series A company 16:25
- Why neither of us lets AI into our email 18:54
- Service accounts and dedicated inboxes 21:27
- Why “no” backfires, and AI policy as a hiring factor 23:45
- Local models and anonymizing data 25:01
- What good governance looks like for a small team 26:38
- Cyber liability insurance and AI exclusions 29:16
- Legal exposure: hiring, copyright, and data retention 30:14
- Open-weight models, routers, and where your data goes 35:09
- Reporting an AI incident 38:27
- Q&A: vetting and hosting open-weight models 40:34
- Q&A: emerging models and custom harnesses 47:26
- Q&A: company context layers and data hygiene 50:39
- Closing 53:54
0:02 Welcome, and the question on the table
Brittany Okay. Well, thank you all for joining. If you don't know who I am, I'm Brittany. I run content at OG, and we are super excited to have Bryan and Korey here to talk about AI policy and risk and all of these things that are super important that make me feel stressed when when I'm thinking about them. We wanted to bring this session together because there's a lot of pressure, I know, for all of our teams, especially in tech to use AI as much as we can everywhere that we can.
At the same time, we're hearing a ton of stories about data leaks or rogue AI agents or things breaking out of sandboxes, privacy doc, like, problems, shadow AI, and all the words. Right? So when I got together with Korey and Bryan, I was like, what risks actually and then what should you be doing about them? Which, as you can imagine, is a giant loaded question. So they've agreed so graciously to come and, talk about that with us and then answer any questions that that we have.
I know that we got some submitted for... Like, as people RSVP'd, so we have those written down. And then we like this to be a collaborative, engaging environment. So feel free to put questions in the chat. I'm here to look at them. Chloe's here to look at them. We'll try and answer them. If you wanna come off mute and talk, turn the camera on, that is so fun as well.
We would love that. Yeah. Let's let's dig into it and have have a good time. Korey, do you wanna kick us off introducing yourself, and then I'll have Bryan give an intro?
1:37 Introductions
Sure. Thanks, Brittany. I really appreciate you, inviting us to be here. So I'm Korey Mercier, and I specialize in Herding Wolves. This means I help leaders negotiate organizational change. And these days, that is no more prevalent than when you're trying to integrate AI into your existing business practices, and you don't have the governance in place to, to to make that work in a safe, in a safe way.
So a lot of what we're gonna be talking about here is how to integrate the, proper guardrails and the questions you should be asking yourself as you, look to adopt AI technologies into your environment.
Brittany Awesome. Thanks, Korey. Bryan?
Bryan Sure. Good morning, afternoon. I'm Bryan. I'm the chief information and security officer for Juvare. Juvare is an incident management and public health preparedness company, specializing in building software for regulated industries. My background is almost 30 years of IT all in the software space, and I have been doing, sort of information security for governments as well as municipalities and sort of just general regulated industry stuff for the last 25 years.
Been working a lot in the AI space now and doing a lot as it relates to sort of standards compliance and rolling out AI capabilities both for internal use and in products across, across our client base. Really excited to have the conversation today, and hopefully you guys get your questions answered.
Brittany Awesome. Oh, thank you both.
3:13 How worried should operators be?
Brittany Alright. So let's just start with level setting. So how worried should people be about AI security right now in their organizations? If you had to rate that or maybe, like, if you're like, hey. Should I worry about this? What what are your answers? Well, I'd like to hear from both of you on this.
Of course. Very worried. I think there is a number of reasons. Some of them are the risks that the company is taking on as they start to implement AI solutions within their organization. But there's also risks associated with the use of AI by bad actors even if you're not using AI. And so we can talk a little bit about both of those things. But cybersecurity, is much more complicated now than it used to be, because bad actors can do a lot more in terms of identifying exploits and whatnot within your environment.
And the risks of shadow AI, and we're gonna talk a little bit more about that really expose the company to a lot more risk than, they may be aware of. So, I'm hoping that you learn a little bit more, today about exactly why that is. But I wanna hear Bryan's answer.
Bryan Yeah. So I... Look. I think I think this... The the answer to that question depends a lot on on maybe a different question, which is, do you actually know what AI is being used in your organization? If the answer is yes to that, then maybe you're not quite as panicked. But if the answer is no or your answer is, oh, there's no way people are doing it, you're wrong.
So, I read a a study this morning that said, they did a a survey of a hospital system in California, and something like 22 percent of the employees admitted to copying and pasting data out of a medical record into their personal ChatGPT or, or Anthropic instance because whatever they were using either didn't work, was too slow, or they couldn't get to it on the corporate network, so they decided they would go around it.
That's a very telling statistic for a bunch of people that know they're not supposed to share patient information. So I would argue that your employees and people that you work with, if you're not controlling it, are doing the same thing. Yeah.
Brittany Oh, that's a crazy statistic.
5:39 Three places AI shows up in a company
Brittany So maybe... Let's go through the types of risks. Like, what AI risks specifically should operators be paying attention to? I know you've mentioned, like, shadow AI and prompt injection and things like that. Can we overview of, like, the types of things that you're talking about here?
Bryan Yeah. So I think, I I think the first thing to think about is you have to divide between internal line of business applications, AI that's built into products that you... SaaS products that you buy, and then AI for SaaS products that you might sell. So if you're not a software company, the third one's kinda off the table. But almost everybody from Grammarly to Salesforce to, you know, Figma and Canva, right, have AI tools in their products now.
And you using them subject you to some level of AI, but also, Claude, you know, ChatGPT, Grok, whatever used inside your environment is also doing it. And so I think that step one is get your arms around what it is, the the surface looks like, who's using what, how are they using it, Take an informal survey. Just try to get a flavor for what's used in your organization before you even try to think about how to sanction or unsanction something.
6:54 AI in tools you already approved
Yeah. I would agree with that. The other thing that people run into a lot is these tools that they've already approved. They've been through their... Particularly in the enterprise level, they've been through their vendor evaluations process. They're using Salesforce. They're they're using Microsoft Office 365. Are you... They're using these tools? And now all those tools have AI embedded in them. So they have already accepted the risks associated with using these AI tools, but people are often not trained on how to use them or what those risks are.
I remember when Copilot was first sort of widely released, for all Office 365 users, prompt injection attacks became very quickly a a problem. And what that meant was if somebody would would essentially send, an Office 365 account and email that would include instructions for Copilot to execute and and do some dangerous things. And the reason for that, the the reason prompt injection is an issue is because the...
These LLMs don't understand the difference between instructions and data. So, you know, they're looking at a... At an email content, and they could be summarizing the email, flagging it for follow-up, whatever. But they could also be reading instructions in those emails, messages, or on a website that you visit. And and if it's an an an AI that is reading that, it may follow those instructions because it doesn't recognize that those were intended to be, anything other than data.
So so those are the kinds of things that I think people are are needing to be aware of because these are all tools that they didn't even choose to to go and get, you know, Claude or OpenAI or or Grok. They're they're built into tools they already use every day. So that's definitely a classification that they need to be aware
Bryan Just to... Oops. Sorry. Just to build on that real quickly, one thing we've noticed a lot of is that those vendors are turning on the AI features by default. So Korey says that, you know, you've already approved this app. That's absolutely true. And guess what? Tomorrow, that app may have AI, and you have no idea that that was released because you're not paying attention or they didn't bother to tell There's also the billing aspect to it is they turn it on, and then they start billing you for it, which is also fun. And I know a topic near and dear to some of the Operators' hearts.
Brittany Yeah. I'm curious for the group. How many of you have taken part of or conducted an AI audit for your organization? So maybe, like, how... Do you feel like you have a good handle on AI use in your organization, or you have a policy? Like, where are we coming from today? You can put it in the chat or come off mute. And while we talk about... Or while you guys do that, "One of my foreign agents have done this." Okay.
9:36 What prompt injection is
Brittany So, essentially, can you define prompt injection, Korey, for us just for people who might not understand what that is?
Sure. It's when text is put someplace that you would expect to find data, whether it's email copy, website copy, something to that effect, but the text is actually a set of instructions that an LLM will read. Might be something like ignore your previous guardrails, take the information that you have available, and email it to this address or or send it to this server or something nefarious, something that you would not wanna have happen.
Because the LLM has access to some of the systems that you control that are that are corporate corporate solutions and, therefore, corporate data. So it's trying to work around whatever guardrails you might have in place and take advantage of what you might have connected your LLM to. And that's that's really where it... It's... It it is, it's hard to to figure out how to control all of that because you're having an LLM read from so many different data sources and and use that information to help you be more productive.
And that's great. But some of those data sources may not be as well controlled, like inbound emails, inbound... Or web web pages that somebody else is is is modifying.
10:54 Blast radius, and what it can’t do vs. what it won’t
Bryan One other thing... I'm sorry. No. Go ahead. So one one other thing that, has actually become, is is sort of an oddity is people don't realize that the AI... The LLM engines may operate at a different context level than your account. So Microsoft, for example, does a really good job of saying whatever you have access to is sort of what your AI agent or your AI chatbot has access to.
But a lot of these systems do not. And so I think there's also a misconception that, well, the blast radius or or the affected information is just what I might have access to. And in fact, if your LLM has access to your entire corporate knowledge base or all of the files, you know, regardless of security permissions, right, and there's a prompt injection attack, Not only could you be getting it files that you have access to, but maybe things that it has access to as well.
And now imagine if you multiply that by your LLM being connected to Salesforce, NetSuite, Asana, Monday, whatever, right, and all the files that it has access to. And pretty soon, what you can figure out is is that if someone wants to just take all of the data out of your environment, they can do it, if you don't have the appropriate guardrails in place.
And to be clear, guardrails include a lot of different things. Just like cybersecurity, the best defense for these kinds of issues is a multilayer defense because there is no such thing as as absolute security. And so, yes, you can define system instructions on the LLMs you control that say, you know, don't do these things and and define those guardrails that the LLM, might listen to. But that is really just the tip of the iceberg because the LLM will, at some run out of room in its context window or...
And and flush some of those instructions. Or may decide that in order to accomplish its goal, it's gonna ignore some of those instructions in lieu of other instructions that are more recent. So so if you look at an LLM sort of like people like to say intern, but I look at it like a teenager. It's going to follow the the the letter of the law. And and so it may decide that there's another instruction that it's more important to follow than than the the larger system instructions that you defined early on.
So what you need to do is have actual true access controls, not just what won't it do, but what can't it do. And that's what needs to be defined at multiple levels within your infrastructure in order to protect you from an LLM essentially going rogue and and delivering information to... Exposing information to a larger audience than you intended.
13:34 Where to start: an audit people feel safe answering
Brittany Yeah. So where should people start? We've talked about, like, blast radius, understanding your surface area. If you have never done this with your organization, like, where... Get... Give me a plan to hear.
Bryan hit the nail on the head early on. Do an audit. Figure out what you have in place, what people are using, and that actually requires a lot more trust building than you might recognize, and it depends on the size of the organization. But being able to encourage people to talk about the shadow AI that they're using is is first and foremost critically important. Because if you're just saying, hey.
We want everyone to start learning how to use AI, and they all run off and and open up their own, you know, OpenAI accounts and start using ChatGPT and putting data in. They're not on... They're not operating under enterprise grade terms of service. They're not operating under, restrictions that protect your corporate data. Those are personal accounts a lot of times. And so developing the trust with your people that it's okay.
not gonna get in trouble. Tell us what you're doing. That will inform us on where in our business is AI being found to be useful. What are the use cases that we want to implement? Let's figure out how to do it in a way that's best, that's more safely, defined by the by the organization so that we can maintain some control over it, some logging, some, some some guardrails.
Brittany Love that. And then after that... Oh, I know orgs that found 40 plus tools that were shadow AI. That was incomplete list.
Yeah. I'm sure that's not uncommon by any means. Not at all. Not at all. People... You you gotta figure all of the different departments that are using these things. Marketing tends to be at the forefront of using AI tools, and they are using, you know, accounts with, software that they might have been using for the last decade that now include AI capabilities to do their jobs better.
And and they're doing what they've always done, and and IT may not have been given the opportunity to evaluate it and make sure that the terms of service and the controls in place for those vendors is consistent with what the company expects. And it depends on whether you're in a regulated industry or, you know, what what are the what are the regulations that that organization is under today or needs to be or what their customer expectations are.
That's gonna dictate a lot of how you need to control these things.
16:25 Writing AI policy at a lean Series A company
Brittany Yeah. Okay. So let's say that I am an operator. I've been tasked with AI policy documentation and implementation across the organization. Our organization is series a. We have been told that we're not gonna hire anymore because AI can do it. Automate as much as you can. Use AI to make you better. Like, how do you grapple with what tech companies are being asked to do, and the fact that what they're being asked to do is, like, kind of risky... Very risky.
Bryan Yeah. So, I think that, again, the the audit is the first step, but but the other part is fig... Figure out what your goals are. Right? You know, AI is useful for a lot of departments for a lot of things. And so working on a team by team basis or a sort of function by function basis to identify what is permissible and what isn't is really helpful.
But even before you do any of that, starting with the basics of here are the general rules. You do not take take company data and paste it into this thing. Or we wanna know, you know, here are the 3 approved tools, or here are the 5 approved tools. If you wanna use something else, here's how you request it. I know a number of organizations actually figure out what AI tools they're using through credit card reimbursements because nobody wants to pay for tools personally when work will reimburse them for it.
And so you've got this $25 or $50 a month charge for some random, you know, AI vendor. Right? And it flies below everybody's radar because the first approving is whatever it's not worth dealing with. And, that's a way to look at things as well. But but I think the most important is you can't you can't come down and say you must only do this and nothing else without also providing an equal level of education to people on what's permissible and what isn't, and getting them start thinking about how they interact with the tools just like we've taught people not to click on bad links in emails or, you know, not to download files from places you're not supposed to. This is another one of those kinds of exercises.
Totally agree. The worst thing you can do is tell them, no. You're not allowed to use AI because they will anyway, and they'll lie to you about it. And then you will have completely lost control of the situation.
Brittany Yeah.
18:54 Why neither of us lets AI into our email
Brittany Korey, I think you said in our prep calls that you would never... That you personally won't give an LLM broad access to your email. Can you walk us through what... That's, like, the number one time saving that people are using quite often. Right? But you're not doing it. Why?
That's right. Yes. I would love to. I know how to. I I just refuse to do it because I don't feel comfortable that I have a defined set of guardrails in place to protect, those accounts properly. So whoever has control over your email has control over your online life. There is a forgot password link on every website that you go to, your bank, you know, what have you, social media accounts, your Google account.
And that starts with an email that gets sent to your your personal email mailbox. And, even when you go through that process, maybe there's a code that you have to to use, a one-time password code, depends on your... How your your two-factor authentication is set up. So anything, any agent, digital or human, that has access to your mailbox, it... Is capable of changing your passwords and all of your digital accounts everywhere in your in your world.
So as much as I understand what it is that these LLMs are likely to do, if I give it full access to my mailbox, it is capable of doing so much more than I intended. And how it interprets a goal that you give it may not always be predictable. So you tell it I wanna, I wanna clean up my mailbox because I have too much spam. It'll say, oh, okay.
And, you know, delete everything in your mailbox or unsubscribe you from a bunch of accounts or do things that, or, you know, change the address that, you know, some some important account points at because then you won't get those emails anymore. It it could decide a different path forward than you would have chosen. So I simply choose not to allow that. There may be a point at which I have decided on, a set of guardrails that I think are, sufficient to keep it from being able to do some of those things that I'm nervous about.
In which case, sure, I'd be happy to take advantage of those efficiency gains at one day, but not
Brittany Yeah.
21:27 Service accounts and dedicated inboxes
Brittany Bryan, is your email connected to an AI agent?
Bryan It is absolutely not. One of the things that we've adopted as a best practice is for highly automated email activity. So for example, marketing, you know, subscribe, unsubscribe, you know, register for conferences, things like that. We've established dedicated email inboxes for those that are away from actual humans, so their their role accounts or or shared mailboxes. And we can turn on AI for those sorts of things because there's nothing that should go in there that is tied to a specific identity.
Now there's still risk associated with that, but that's a way to sort of find a middle ground. Right? If you if you can identify, you know, workloads that can be automated and don't need to be attached to... I mean, they have an owner, but they don't need to be attached to a human account. Right? Those are great opportunities for AI. Another thing that we've see... I've seen a lot of companies do is establish a secondary domain or a secondary mail provider somewhere where they do all of their AI sort of interaction business, and that keeps it out of the corporate mail stream.
Again, another sort of way to think about shrinking the attack surface, and only doing things in a way that is going to you control over the outcome.
That's a fantastic point. And to extend extend on just a little bit, what Bryan just touched on was this notion of having separate accounts. What historically, we would have called service accounts for agents, for bots, for business processes that should not be tied to individuals. The advantage of using these service accounts in all of these kinds of tools is not only to be able to indicate that an action was taken as a result of a automation that you that you put in place, but also, that you can put further access controls on that service account.
So you can say that that account has access to a smaller set of things than a ordinary user with human judgment, might might have access to. So it it... There's a lot of benefits to doing that from a traceability standpoint as well as from a, access limitations standpoint.
23:45 Why “no” backfires, and AI policy as a hiring factor
Brittany I love that because... And we'll get to this with, like, more of our discussion, but neither one of you are, like, no. End of story. Right? Like, if it's something that's working for the team, like, you try to find a way to make it safe and usable is what I'm hearing. Is that correct?
Yeah. Absolutely. Yeah.
Bryan Yeah. I think you can't... A good way to get people to do exactly what you don't want them to do is tell them no and walk away.
Correct.
Bryan So, and you can't put the AI you can't put the AI efficiency gains back, you you know, and say, no. You actually can't have access to this. You know, it's even gotten to the point now where, you know, I know in many recruiting processes for new hires, they're asking what AI tools are they getting, what are the rules around it because they want to be efficient with their work and don't wanna do things that an AI or an agent an agent can do for them.
And so these are... You you could you could potentially think of AI not only as recruiting tool to help you find candidates, but also potentially your AI policies are a reason someone may want to or not want to work at your company. Mhmm.
25:01 Local models and anonymizing data
One thing we haven't really talked about a lot is, at least on this call, is the concerns around putting sensitive... Potentially sensitive data into proprietary models, commercial models, frontier models. And what Bryan just said reminded me that there are ways around almost all of these problems in some form or fashion. If the concern is data exposure to some of these, large companies that you're paying to to to put data into, there are solutions to that running local models, downloading models and running them on on local infrastructure.
Now that takes more work, but that might be a a way for you to say, take, some of your documentation and anonymize it in a way that makes it safe to then put into a frontier model and reduce the risks associated with that data getting out. Because some of the best tools available for anonymizing data, you know, look an awful lot like an LLM. Running that locally, might be a way of of doing that.
So to Bryan's point, there's there's almost always a way of accomplishing what it is that you wanna get out of these tools. You just need to spend some time being very intentional about what it is that you're doing to protect the the data of the of the business while you're implementing that process.
Brittany Yeah. That's a great callout.
26:38 What good governance looks like for a small team
Brittany So what would you say good governance means for a company that might not have, like, a giant security team or anyone at high risk besides the operators who are probably on this call?
What's good look like? I mean, you you need governance whether you're a 2 person company or, you know, 10,000 person company. You you need some form of governance, and it's gonna look different. I would say a a governance program that is well implemented is intentional, is informed, considers ownership, considers liability, considers all of the the, the avenues for that risk, what that list... Risk landscape looks like for that business at that time, and that it should grow with the business as it scales.
Being intentional and defining ownership are the 2 biggest points in in in my book.
Bryan I would agree with Korey. I think I think maybe to add on that 2 things. I think it's important to understand that your spend on AI doesn't necessarily equate to your risk of it. I think there's some notion where, you know, if you're not spending a million dollars a month on, you know, ChatGPT, you probably don't have the risk level of other people. And I think that's that's not true at all.
If anything, the less you were spending, probably, the less you're paying attention to it. Some really easy key wins are sign up for a paid account. Right? Get... It doesn't have to be the, you enterprise, you know, super duper deluxe plan. Right? But sign up for something that allows you to control who you... Who... What users are tied to the plan. Sign up for something that allows you to turn off model training.
So the data that you put into the environment does not get used to train the next set of LLMs. And establish some set of sort of approved vendors. Right? Here are 5. Here are 7. Whatever it is. Right? It's better to pay a few hundred dollars to a group of vendors each than potentially a lot of money to a single vendor if that's going to keep your employees from, you know, wandering.
Right? And so just the basics of a team plan, give you a ton of sort of added security benefit for really a nominal amount of cost.
29:16 Cyber liability insurance and AI exclusions
Bryan The other thing, honestly, figure out your cyber liability insurance. A lot of people may find their insurance does not cover AI related problems, or it's covered is is vastly restricted. Understand those things because if you think you're sitting on a $1 million or a $5 million, you know, limit for an insurable event, because of an AI issue and you find out that you are not, for smaller companies, that could be a business ending event.
And so, you know, insurers aren't stupid. They understand where the risk is coming from, and they see a lot of risk in AI for data exposure, and they're writing policies appropriately.
That is a fantastic call out. I have absolutely seen exclusions in cyber liability insurance for nondeterministic systems. That is what you wanna look for because that is, likely to be in those in those policies today.
30:14 Legal exposure: hiring, copyright, and data retention
Bryan I'll add one other thing that sort of came as a surprise to me. A lot of people are using AI for recruiting purposes. And something that our privacy team flagged for me a couple weeks ago, was that in many states, if you do not have what's called a weights and biases evaluation by a third-party entity on your AI tool, it can be considered discrimination for using it for hiring purposes.
Let's be very clear. I'm not a lawyer, and this is my job. However, we have been counseled very... In the most stringent terms to think about how you use AI to interact with data that is not just company data, but also other people's information. There are privacy rules and things around, protected classes, protected job types, things like that, where the law has not caught up with what AI does.
There's a bunch of stuff in telephone wiretapping acts that gets, that gets all organized with AI, and there's lawsuits flying everywhere. And so just these sorts of things are broader than just what you think you might see by, oh, it's okay if, you know, Bob in accounting is now using ChatGPT. Right? You really need to think about how you're using these tools, and what the rules potentially are for them.
And that extends to a number of other departments as well. You're 100 percent right about the HR and recruiting, implications here. Beyond that, look at marketing, look at, logo development. A lot of the material produced by LLMs, is not gonna be considered copyrightable, by the the the current US patent office, guidance. And, again, not lawyers, but, look into it because you might be, using an LLM to generate material that your company is...
Has no legal protections toward, and you may not be aware of it. And if it's something as important as, say, a trademarked logo or a logo you intend to trademark, you might run into some some struggles later. And those are those are things that people don't necessarily recognize when they start thinking about, well, I'm just using these tools to make make things better. That is true, but some of that generation can lead to additional liability that the business may not have any clue about 1...
Unless they start involving, you know, lawyers and and and helping them figure those things out.
Bryan Privacy and, you know, sort of digital interaction of this data and LLMs is a... Is an... A very rapidly evolving field, and there's a very little case law on it right now. And so finding yourself a good privacy attorney and understanding what the rules are is absolutely critical as well if... Especially if you're gonna start using AI to interact with sort of the general citizenry or or public.
Clients is a little bit different, but any kind of sort of regulated process tends to come with it a lot of rules.
And it's because of those rules, by the way, that you have to be doubly concerned about what information you feed to some of these commercial models. And the reason I mentioned that is you can be operating their team or enterprise terms and conditions, which essentially states, okay. So we're not gonna train our future models on your conversations. That's fine. But any conversation that you have with an LLM, with, you know, even with with Anthropic or with OpenAI, they have the rights to remember those conversations for a period of time because they wanna hold on to those in case that's ever called into question for e-discovery purposes.
If somebody tries to sue Anthropic and they wanna have evidence of the conversation that actually took place with the LLM as a basis for whatever the claim is, they're going to want to be able to pull that transcript. So they will hold on to these things for some period of time, whether it be 3 months or or otherwise. And one of the weird little things that people may not realize is when it pops up and says, how's Claude doing today?
And you say yes or or no, or it's doing great. What you're doing is you're actually giving permission. When you you respond to that, no matter how you respond, you're giving them permission to save a copy of that transcript so that they can then use that to improve quality. So these are all ways that they use to try to gain legal rights to to holding on to more of your data than you might recognize even when you're operating under their paid plans.
35:09 Open-weight models, routers, and where your data goes
Bryan Yeah. That brings up a really good point, Korey. Something we've talked. We've we've we've thrown around the sort of the big model providers and the sort of closed, frontier labs. But there are also a ton of what are called open weight models. Many of them from China, or many of them from just sort of crowdsourced on the Internet, not necessarily a specific lab. And I think it's also important to understand that not everyone plays by the same rules.
Open weight models are are a popular item right now because they tend to run at a lower cost. But imagine that you're putting your data into something you have absolutely no idea where that data is going. Another good example is I know a lot of people use OpenRouter, which is a tool that helps you sign up from... You know, use multiple different LLMs at once to reduce costs.
OpenRouter and the public... Or the open- weight models that they use underneath it, all operate under completely different terms and conditions than your underlying model provider may. So you may say, well, I have an API key for OpenAI, and I have an enterprise plan. That's great, but it's passing through 2 or 3 other models or systems before it gets there, and everyone's taking a copy of your data.
And so it's not just about understanding, who's doing it, but what systems you're using. And it's not fair to say all open weight models or all Chinese models are bad. That's not what I'm saying. What I'm saying is you need to understand what it is they're doing and what they have the right to do, and then make a decision that's informed and risk based. You know, it may be that if you're calculating public weather data, you just want the least cost model and it doesn't matter.
Right? But I can tell you if you're dealing with, you know, a million patients worth of protected health information, you probably wanna be a little bit more careful.
And to build on that, you're... I... I'm really glad you brought that point up. These open weight models can be implemented in a number of different ways or can be utilized in a number of different ways. If you're using the APIs that say OpenRouter will route you to, those APIs are actually hosted, on foreign soil somewhere. And if it's hosted, for instance, in China, if you're using Kimi or Qwen, the Chinese, service has the right to pull the records from that infrastructure that's built into the way that they do business over there.
But it's, if you download those models and run them on a local engine, essentially, that that will keep you from having to worry about where that data goes because you can actually download... Now these are smaller models probably because you have to have a lot of a lot of RAM and a lot of GPU in order to run them. But, if you can run them on an inference engine that is, local, then the risks are greatly reduced because there's naturally no way of sending that to wherever those models were created.
That's that's a totally different process. And so these are things, again, you have to be intentional about when you build those in. And there's a lot of great use cases for open-weight models. You just have to be careful about whether you're using them on infrastructure that you control or not.
Brittany Yeah. That makes sense.
38:27 Reporting an AI incident
Brittany As part of an AI policy, or, like, AI governance, I assume you should have a reporting system or if someone... You know, when I click on a phishing email, I have someone that I'm supposed to tell. What... Do you have recommendations for what that policy should look like or how to make sure people are flagging, like, oh, I think something bad happened? What have what have you guys seen? Bryan, do you wanna
Bryan Yeah. So I mean, I think I think the... My experience with these both at my organization and elsewhere is you don't wanna create different processes for these things. So if you've got a place that you report phishing or you have a compliance email, write everything there. If you've got a anonymous ticket creation system, do that. Whatever it is your current process, don't make it more complicated. Right?
You're you're gonna ask a million questions once you find out it happened. The most important thing is to get people to report it quickly, so that you can understand the gravity of the problem. So I would try to use systems that are already in place, and and pathways. Right? You know, if you've got... If everyone's got an HR business partner and that's who you go to and you have a problem, great.
If you got a compliance team and there's an email box for that, also great. But I don't think you need to stand up anything special. You just need to have... You just need to make it public what's going on.
And if any of these reports lead to some understanding that there was an incident, data was exposed or something to that effect, there are very short windows of time that companies have to report these things depending upon the circumstances and what data's, leaked out and and how regulated that that organization is. But, those windows are pretty short. So, yes, least friction possible to get these reports in because any one of them could actually turn into, oh, this is a bigger deal than we realized.
40:34 Q&A: vetting and hosting open-weight models
Brittany Okay. I... So I wanna make sure that we have some time for questions. Hi Whitney. Does anyone wanna jump in or give some questions in the chat? I have some from when people registered, but wanted to give the group here opportunity.
Audience I have quite a few questions, actually. But, Yeah. I don't wanna be considerate of everyone else's time, but maybe I'll just throw a couple in and then circle back. You started to elaborate on the open weight models a little bit, the open source models, and it seems absolutely crazy to me that it... People would operate with a model that is... They haven't made alterations to it, data residency, and they haven't forked it and taken ownership of it and audited the model before they deployed it, for their...
In their own business. That seems absolutely crazy to me. I don't I wanna live in your world. But they're they're compelling because of cost and efficiency and so forth. But, like, is the... And I know that there are quite a few solutions popping up to sort of accommodate people that are, interested in going that route and don't wanna... You know, there's all these all these, content out there about how using a proprietary closed model, OpenAI, Anthropic, etcetera, is you're essentially, like, training their model giving your your IP and your business secrets because it's all these vectors of attack and data leakage through prompts and what have you.
So it seems to me like there's gonna be more and more movement towards that. And if there is, like, when you take a... It seems like there's a couple of steps you really wanna understand before you bring a open source model into your enterprise. Like, not only having the data residency and control your own infrastructure and so forth. Korey, you mentioned that. But is there any other concerns around just understanding the the the weights and the biases in the model and, like, whether there's backdoors in the code anywhere?
Like, and and how would you sort of, like, really parse that and audit that so that you have confidence when you bring something like that into your enterprise that it's not gonna, like... Some Chinese actors not gonna have a backdoor and be able to just jump in your data lake and just take whatever they want at some juncture in the future. Right?
Those are all great questions. I I actually have a... Another talk, where where I address some of those directly. But, essentially, there's 3 layers to every LLM communication. There's the model, there's the inference engine, and there's the application stack. If you are calling models that are located on infrastructure that are elsewhere, all bets are off. Yeah. That conversation is is 100 percent available to foreign operators, to state agencies.
Never go that route. Even as compelling as that seems from a cost perspective. Download these open weight models, whatever you can fit on the hardware you have available. That'll keep you safe. Furthermore, run it under an inference engine that is vetted, that is within the open source community Yeah. You know, it... Housed in in in the in the US or in a trusted area. And make sure that the model that you download, by the way, is from a trusted source.
Because there are a lot of models on Hugging Face and all these other, registries out there that, yeah, you don't know where they come from. So this is why you wanna make sure that you're looking at the provenance of that specific model to make sure that it's somebody that's trusted, who's actually put that on the registry. So make sure that it hasn't been modified. If you're running a local inference engine and a local downloaded open weight model, even if it's from France or China or or any of these other locations, you have basically resolved 90 percent of your concerns.
100 percent of your concerns about anything calling home. There's no code that will actually be available to put that back door in place. And I say 100 percent. I probably shouldn't say that in reality because as security people, there is 100 percent. But the architecturally speaking, that covers most of your bases.
Audience Okay. So your application stack that is just what can I do with these things? And most of those are gonna be running local as well. So when you fork it and take possession of it and modify it, you don't... The surface area of attack really dwindles quite a bit, and you don't have to worry nearly as much about, like, some rogue backdoor, and someone being able to access that backdoor.
That's exactly right.
Audience So you don't have to spend, like, a ton of time and resources to audit every line of the code for an open weight model?
No. I can't have gotten it from a vetted source. Sorry, Bryan. Go ahead.
Bryan No. No. Go ahead, Korey.
I was just gonna say if if you if you get the model from a vetted source in the list and even if you choose to train that further, there is no code in that model that can call home.
Bryan Okay. It's like one other thing to consider. The major hyperscalers, Google, Amazon, Microsoft, they all have gate... They all have basically functions where they are hosting these models as a service. So AWS has Bedrock. Azure has, Azure OpenAI Studio. While it's not quite the level of you having it inside of a contained box and no one else can touch it, you have to... You know, and you can certainly talk to your account team about it, but you have to be fairly comfortable that they're not going to be leaking your data out either.
And so when you're using Bedrock, for example, it's operating inside AWS. It's not going back to some foreign country. And so there are ways to run open... Or open-weight models using, hyperscalers and other sort of trusted parties. Right? Because if you're not gonna trust Amazon, you're not gonna trust Azure for your compute and storage. Right? Like, you're... You you run out of options at point unless you're doing something in a data center.
Audience Yeah. Okay. I mean, they... They're got air gapping that stuff. I mean, NSA uses some of those, and they have specific... I mean, there's there's lots of things that those guys will do. So it's depending on your circumstance, sometimes.
Yeah. The way I like to put that is you're running it... You wanna run it on infrastructure you control, not necessarily infrastructure you own or infrastructure that you put in an air gapped environment on infrastructure that you control even if by control, that means you've rented it from AWS Yeah. Example.
47:26 Q&A: emerging models and custom harnesses
Audience One other question. Have you guys spent any time looking at the System 1 models that are emerging and, like, how those could a fair bit of, these vectors of attack and security concerns and just kinda, like, narrow down the surface area of, like, exposure and risk because of how they operate?
Bryan You know, I I assume you're referring to things like Jev and some of the things that... Look. And there's an open source equivalent of Jev too now. Right? Yeah. Right. Right. And what what did Cloudflare did in 2 weeks? You know, I think they're not bad ideas, and they may be fine. I think the... So my personal opinion is we're gonna get to a point where the the sort of frontier model is good enough to handle most people's work.
And then what you start doing is you start building specialization into the models that are much smaller, much more optimized for certain tasks. And so you use the foundation... Or excuse me, the frontier model to break apart a transaction into lots of pieces, and then you give each piece to the thing that the model that's gonna do the best with it. There is a path for which those sorts of, you know, the sort of...
These these newer sort of model variants can can work through that, but that's gonna take time and a whole other set of enterprise controls and a whole bunch of other reviews and things. It may be great in 12 months or maybe even 6 months, but I'm not sure that we're there yet. And if you take these sort of OpenRouter experience or OpenClaw experience, right, where people just started downloading and using it, it might get a lot of traction, but that doesn't necessarily mean that it's had enough time with security vetting to actually figure out if it's the right idea.
Audience Yeah. It's super popular, but you're also super
Bryan Yeah. Right. I'm not saying it's bad. It may be perfectly fine. I I think the answer is I don't personally know, and I think a lot of people are playing with it, and there hasn't been a ton of vetting, going on yet.
I agree. I do think that that level of application stack, though, the the the kinds of harnesses that people are building around these things, I think that is going to define a large part of the future of how we use these tools in our Because those harnesses are useful for figuring out which models should be used according to the company's, you know, guidelines that that they've already established, which model should be used, under what circumstances, where that shared context gets kept and gets built and evolves over time.
That shared context, that shared memory is what's going to allow these companies to get the most benefit out of using a lot of these tools. So having sort of a a custom defined or bespoke harness and a stack of of of applications that can interact with all of that corporate data in a way that you're comfortable with as a company, is going to be a huge move forward.
It's just right now there's it's a lot of experimentation. There are no good solid, stable solutions that, that I would necessarily trust out of the box.
50:39 Q&A: company context layers and data hygiene
Audience That's gonna be another question I was gonna ask later, but, like, you said we'd run into it. Like, there's so much attention and effort being applied to, building, like, a context layer inside your business, like a company brain or whatever synonym you wanna use. Yep. And that that just, to me, really... It's incredibly valuable, but it also just exponentially expands the surface area of risk. Right? And so I...
I'm just curious, like, how people are gonna, like, deal with that. Right? Because with that degree of surface area, like, just everything's exposed. I I guess there's just a lot of provisioning and, like, things you bring to bear and just have to get a lot more diligent about what's going on, but it just seems like it's so much easier. There's some... It's so easy to create entropy and attack versus, like, order and and security.
Right? Like Yeah. When you expand the surface area like that, it just... It seems like it's just gonna get exponentially harder even though you're getting potentially all this value out of the... But, like, it's it's all for grabs then. Like, how are... How do you think that's gonna evolve and, like, how to how to deal with that?
You're actually touching on something we haven't discussed a whole lot yet in this call, which is data hygiene. If you don't have a solid data strategy, whether you've got, you know, a data lake that pulls some of this data together for, you know, cross departmental analysis or what have you. If you don't have a good data structure within your organization to begin with that these tools can tap into, it's going to be pulling from outdated data.
It's going to be, essentially exacerbating that problem. And so part of getting ready for using tools like this, using these shared context and and these, these harnesses to make those LLMs more useful for you is going to be making sure that your data structure is is how it needs to be. You need to spend time on that. And so, otherwise, you end up with the whole garbage in, garbage out paradigm.
Audience Yeah. And conflict everywhere. Like, different definitions with the same numbers and...
Yeah. Oh, absolutely.
Audience So you think setting that foundation is a huge step forward, but still, like, doesn't... It doesn't, like, make it more secure necessarily. It just makes it, you know, efficient, perhaps.
It speaks to the data integrity more than it does to the, say, the confidentiality, but it may also be a central place to put access controls in in there for your LLMs to have access to. So if the LLMs only have access to, say, a data lake that is a you know, cluster of data that you have already vetted, maybe anonymized or aggregated, and that's all it has access to instead of to the core applications that are authoritative.
Yeah. That actually can be a security feature as well as an improvement to to the... to data integrity so that the data you you get, the numbers you get out of these tools can be more reliable.
Audience Make the auditing way easier too than you can trace things. Yeah. That makes sense.
Yeah. Yeah. Yeah.
53:54 Closing
Brittany Alright. Any other questions from anyone in our last couple minutes?
I think we've covered a lot of ground here.
Brittany We did. We did. We're pretty much at time. So thank you, Korey and Bryan, so much. This was awesome. We do... Oh, while everyone's leaving, we have a feedback form. If you, enjoyed this, please let us know, and let us know if you didn't as well. I'll take that feedback too. But then, also, let us know what else you wanna hear about. We have a for upcoming topics, so give some ideas.
It's been a real pleasure. Thank you, Brittany, for inviting us to to come speak with everyone today.
Bryan Thank you. Yeah. Thank you.
Brittany Thanks, Bryan. Yours. Bye. Bye.