Name: When Prevention Failed: Real Recovery Stories from the Trenches
Uploaded: 2026-03-25T17:52:06.907Z
Duration: 36 min 12 s
Description: When Prevention Failed: Real Recovery Stories from the Trenches

Transcript for "When Prevention Failed: Real Recovery Stories from the Trenches": Welcome to When Prevention Failed, Real Recovery Stories from the Trenches. My name is John Murphy, and I have the great pleasure to be joined today by Kyle Feeler of Rubrik Zero Labs and Steven Foskett, subject matter expert on all things Tech Field Day. Guys, I'll let you, introduce yourself. Steven, let's start with you. Thanks, John. It's great to be here. Again, my name is Steven Foskett. I'm the president of the Tech Field Day business unit at the Futurum Group, and I have focused on storage and data protection for basically my entire career. Written books, done a lot of seminars and webinars, but more importantly, I've spent a lot of time in those data centers in the trenches, trying to help companies, including my own employers and others, to do a better job of data protection. Excellent, yeah, and great background. Certainly, I think those of us who have been in these areas for a really long time have seen just tremendous change, but I think one of the things that sticks out to me is that we see patterns over time, and Stephen, I'm hoping you're going bring a lot of those patterns out as we talk about this today. Our objective really is to make this as valuable for you attendees as possible, so feel free to post questions, do follow-up work as you will. And Kyle, tell us a little about yourself. You are an integral part of the Rubrik Intelligence team here. Yeah. Thanks, John. Happy to be here. Yeah. My name is Kyle Feeler. I have about a decade of experience in the cybersecurity industry working for mostly vendors across the endpoint network and now with Rubrik in the resilience space. So I am an analyst researcher with the Rubrik Zero Labs team, and we are, you know, sort of studying how threat actors operate and how how resilience and backup data could be an essential source of threat intelligence and how it may inform operators on some of the threats that they're facing. Excellent, excellent. This is going to be a great session, guys. Just to level set for everyone in terms of what we're hoping to get out of this, as I mentioned, it's really meant to be for practitioners to give you practical guidance on things that work, things that don't work, And kind of to the point, what happens when you get that 2AM phone call? And what are the first things you should be thinking about during that phone call? You're going to hear a lot about the fact that we think a lot of this can be done beforehand so that that phone call ends up being more of a rote memory kind of exercise than it is a quick respond and figure things out on the fly piece. But I kind of want to start with just, you know, just kind of setting the stage for what we think of in terms of availability and why that is important for this. So if you think about whose job recovery really is, it's easy to get caught in the, well, it's security's job, or maybe it's IT infrastructure's job, or maybe it's the business resilience team's job. I think at the end of the day, we all recognize it's everybody's job. We're all going to play parts in that recovery. And that recovery really has huge business impact for us. Whether you mention, whether you measure your success in terms of financial or mission critical services that you provide to your customers, the bottom line is that our uptime really makes a difference in people's lives. And if you think about one of the the wanted goals for uptime is 99.999 or five nines. Well, as we talk to business folks about that, they probably don't necessarily understand what that means, but they do understand the impact, which is there's a measurable impact to individuals there, whether that's financial. A five nine uptime for a year still leaves you with almost six minutes of downtime. And most of our SaaS providers, which we're becoming immensely more dependent upon, only guarantee about 99% uptime, which means that there's a whole lot more downtime that you might have to be prepared for. So one of the things we want to draw out here is what's happening in the environment. And we're going to start a little bit with talking to you, Kyle, about what we see from a threat actor perspective and what threat actors are really doing to monetize our success or our lack of preparedness, depending upon what exactly is happening in the environment. I tend to think about the bad guys in a slightly different way. Think about them as competing businesses, right? They are looking at how are we doing things today and how can we force our competitors, which is all of us in the private industry, to make as many mistakes as possible in the shorter term as possible. So Kyle, with that backdrop, if you wouldn't mind, just take us through what you're seeing from the Rubrik Threat Intelligence perspective, Rubrik Zero Labs perspective. Yeah. I think threat actors, as threat actors, your motivation is gonna determine your TTT, your TTPs, excuse me, your tactics, techniques, and procedures. So, you may have a financially motivated threat group who it you know as soon as the compromise has occurred and as soon as they have access to your environment they are going to want to let you know that they are there to maximize their leverage. So, you know, they can do this in a couple of ways. We see a lot of threat groups today specifically targeting identity infrastructure because they know that once an organization is unable to authorize and authenticate its users, granting access to any business critical apps is is extremely risky. And so we see a lot of financially motivated threat groups that excel in compromising identity infrastructure. And so an organization's ability to reset that infrastructure, to roll back any changes that may be made in terms of privilege escalation, you know, role based access manipulation. You know, that's that's a critical capability for organizations. Another thing we see quite frequently is the deliberate targeting of cloud native backups. So Google's most recent Cloud Threat Horizons report, you know, confirmed that a lot of groups like twenty one sixty five, so evil corp is is another name that it goes by. These these groups are deliberately targeting cloud native backups because they know that if an organization's plan b has been erased, that max leverage has been maximized over those groups. And, you know, it's the same with the compromising of identities. If if you cannot grant access to mission critical applications, operations are ground to a halt. What this does is maximizes leverage over the target organization so that ransomware demands will be complied with in a speedy manner. Basically, extortion is conducted more quickly. Anything that furthers their goals happens more quickly in that scenario. And so there's another category of threat actor, the state sponsored threat actor that may be more interested in conducting corporate espionage, eavesdropping, or even implanting a backdoor that they know that they can come back to at a later time and access to compromise the organization. And so those threat actors are going to be more interested in being stealthy, evading detection, avoiding, you know, the the indicators of compromise that would force a response on behalf of the target organization. And so they're going to be more interested in implanting malware in areas that maybe EDR solutions have less visibility over, such as a hypervisor. We saw this with Brickstorm. Rubrik Zero Labs specifically saw this with Brickstorm in in some customer environments, and that's one of the reasons why, as I'll say again and again, backup data is a very critical source of telemetry for threat hunting. Excellent, excellent points. And obviously, for those who are watching this, Rubik's Area Labs has done some extraordinary work recently and gotten some industry accolades for the way that we've used that backup data to actually understand what threat actors are doing long before they launch an attack. I'm going to switch for just a moment and talk a little bit about what happens when the social side of things is engaged. So, couple high profile attacks recently that involved more social engineering than they did technical attacks. And would wonder if you share anything from those attacks that's a key takeaway for our attendees today. Yeah. It's becoming another tired saying in InfoSec that, know, threat actors no longer hack in, they log in. You know, you hear it time and again, but, know, like a lot of cliches, it's it's true. And you see groups like Scattered Spider is one that's had quite a bit of success recently, partially because of, you know, facility with the English language, knowing, you know, cultural savvy, and the ability to, you know, or the chutzpah, I I would say, to to just call up a help desk and say, you know, I need you to reset my two factor authentication. And, you know, an unfortunate number of times, that actually happens. And so that's a group that's seen a lot of success with sophisticated social engineering attacks. And again, it goes back to the, you know, the identity piece. You have to you have to trust in your ability to authenticate and authorize users if you want to be granting access to business critical apps, and so what they're doing is they're undermining that trust by Yeah. You know, by hijacking legitimate credentials. Yeah, yeah, no, excellent points. I wanna pivot for a second to you. In those moments, so we talk about when failures happen, we talk about social engineering, but getting past all that, in those first moments when something happens, what are the things that the first, I'd say, couple of dependencies in your mind that really constrain the ability to recover? Mhmm. Well, I think, first off, Kyle did a great job of sort of laying out where we're at right now with cyber threats. And it's interesting too how much these things have escalated in terms of everything from the profile, the attack methods, the targets, etcetera, and yet how much has stayed the same. Because ultimately, when an attack happened in 1990 or when an attack happened in 2025, you're kind of in the same situation in terms of how are we going to recover from this. And in most cases, the answer is, uh-oh. Am I ready? And ultimately, I think that the thing that can make people feel more confident and poised when that moment happens is practice. It's just like anything else. I mean, there's a reason that police and military and all sorts of other first responders are constantly practicing. And that's because when moment happens, that's not the time to think about, am I covered? Do I have my backups? Does it cover everything that I need? Do I know how to operate it? Do I know what the procedure is? That all can be handled previously preparation for the moment when it happens. Because the last thing you want to do is panic. Now, that all is easier said than done, and has been as long as people have been trying to protect data, it's a challenge for people who are in the data protection environment to understand what is the data, where is the data, what are the applications, What data is needed to What data needs to be recovered and at what process and what pace? What's more important? What's less important? All of these things are questions that can be answered, but questions that have been very difficult to answer for a very long time, which is why it's nice to see when products in the market are now starting to try to help people with that, try to give them heads up if things aren't being protected, suggestions on other areas that could be, and also trying to be more all encompassing. As Kyle mentioned, identity threats are really rising in prominence. And yet, until very recently, data protection products didn't even cover identity management identity threats at all. They didn't offer any kind of ability to check that or to protect that or recover from that. So it's very exciting to see products responding to those kinds of emerging threats in a productive way. The same is true, as mentioned, with cloud backups. That was originally a tool that protected you, and now it's an opportunity for another attack surface. And so there again, we have to make sure that those are protected. But ultimately, I think the thing that people need to think about is how can I be ready when this happens? And the answer to that question is always gonna be the same. It is to understand your data, to understand your applications, to understand your business, and to spend a lot of time and energy preparing for that moment instead of hoping when that moment happens that you're ready. Yeah, yes. Great set of points. And I can tell you as a practitioner for a very long period of time, that idea of just testing until it's muscle memory, test until your eyes bleed, is something that I I can't say enough of. Because I think to your point, it just gets all the questions out. Because no plan is gonna survive first contact with reality intact. But the more that you test, the more things that you see are wrong with your plan. And there's no pressure at that point. Right? What you're learning is free as opposed to when it's a downtime impact. Related question for you, just to help level set, and so you don't have to go into any, any specific, you know, product specifics, but help the attendees understand what we mean by things like Secure Vault or Clean Room, if you would. Yeah. So Clean Room is another rising potential here. I love the idea of it. Essentially, if you have been attacked, if there's a bad actor in your environment, you can recover data to that environment. But how do you know that the bad actor is out of there? How do you know that there's not something else compromised? How do you know that they're not corrupting it or infecting it or attacking it, even while you're bringing it back? The answer is to bring it back in a clean environment, a clean room. And this is something that has always been a goal, but it's always been very hard to achieve because, of course, the last thing you wanna do I mean, can you imagine if if protecting your house from a fire meant having another house exactly like yours fitted out with all your stuff, like right next door, sure, that would be great if there was a fire because you could just move right in. But we can't really do that in IT because essentially, we would have to build everything twice, and that would make everything cost more. Happily, today's DevOps processes, cloud computing has all made it actually a lot easier and a lot more practical to literally automatically spin up a clean environment on demand to recover to. And that is a really powerful feature because it means that you can be sure that if I'm rolling back to before the attack, I'm rolling back in a in an environment that is, as as we say, clean, rather than risking immediate reinfection. Kyle, what do you think? Yeah, absolutely. And I think that preventing reinfection is one of the most critical points in the recovery, and it's also one of the most difficult. So, yeah, having a clean environment like that that you can restore from is absolutely critical. Yeah. Kyle, just to pick up on that, what are the things you look for? What are the signals in a recovery environment that you're looking for, and how do you find them long before we ever get into the conversation of what to do in an emergency? Yeah, and I think it comes down to essentially nailing a couple of the steps in the incident response. Determining the scope of the recovery is the first step, and often one of the lengthier steps in a recovery operation because you have to accurately determine what assets have been impacted to what extent if you are going to successfully roll them back to a prior state, prior to the infection that is. So, you know, that's absolutely one of the key steps to a clean recovery. You have to then be able to quarantine those assets so that you can stop things like lateral movement from occurring, and threat becoming, you know, out of control in your environment, which expands the scope drastically and lengthens the time of recovery. And then finally, think you have to be able to successfully validate that recovery and that's where things like the clean room come in. But, you know, in the validation process, you're scanning for things like known indicators of compromise, those may be hashes, YARA rules, but those are some of the things that you have to look for, and make sure absent before confirming a clean recovery. Excellent points. Yeah, and I think certainly you tie the idea of a vault into that, and the vault being a copy, a separate copy, whether it's physical or logical, from your production environment, and that gives you the ability to conduct those searches without impacting production. You know, where, if you're a bank or if you're a financial organization, you can't afford a lot of extra traffic on your network, being able to do that actually makes a huge difference, and incorporating exactly what you just said, which is preemptively scanning for those things It makes a huge difference in recovery time. And Stephen, I'm sure you've got some firsthand experience with the ability to do that and the impact it potentially has. Oh, absolutely. And it is absolutely amazing to see the amount of information that can be gleaned from data protection. You know, it's one of those things, for a long time, we were not really paying attention to. You know, we were we were protecting data. We were making copies of data. And then suddenly, somebody realized, wait a second. This is just a treasure trove of information for the business to figure out what data is, what the patterns are, what's happening. That was then taken to the next level by these vendors who are able to do anonymized scanning and assessment in real time of the data environment. And basically, like Kyle was saying, finding these threats and exposing these threats, it is really incredible what you can learn from the data. But of course, that's also a source of information for the attackers. And so we have to make sure that they're not learning the same things by scanning and examining our data. Yeah, that's a great point. And probably one of the most frustrating things, I think, for practitioners, regardless of where you've hit in the organization, is just recognizing that they could be in your environment at any point in time and learning the same lessons or more important lessons than you were at that point. Well, especially like Kyle mentioned, some of these threat actors are their goal is not to do ransom or anything. Their goal is just to be there, and to be ready, and to attack at some point in the future. And that's pretty scary, isn't it? It is. Yeah, it certainly is. We talk about complexity. We talk about response time and how quickly we need to be able to respond. From your perspective, Steven, how do you handle the problem of tools, of having so many different built so many different dashboards, so many different points of reference that it's difficult to tell what's noise and and what's actually a signal that we should be paying attention to? Well, you know, there's there's a couple of factors there. One of them, of course, is the question of just the sprawl of modern IT systems. Gone are the days that there was a server that ran an application. Today, even the most simple applications span multiple servers and multiple networked devices. Because remember that attackers are attacking storage devices as well. They're attacking databases. They're attacking all sorts of elements of the application stack. But even beyond that now, most organizations also have cloud computing environments. They've got software as a service. Increasingly, they've got AI applications, AI agents. All of these things are critical business resources as well. And all of these things need to be monitored and protected. So there's a couple of different aspects there. I mean, number one, if you are a data protection pro and you're tasked with cyber readiness, you have to understand where everything is. Not just where the core servers are or where the, you know, the core applications run, but you really have to understand where every business critical application runs. And then you have to figure out how can you monitor that, How can you manage that? How can you protect that data? How can you recover that? Because that's another challenge as well. If something is off-site or with a service provider or something, how can you make sure that that's all recovered? The good news is that products are increasingly covering more and more and trying I don't know that we'll ever get to the goal of a truly unified data protection platform, but that is absolutely the the goal of companies in the data protection space. They're trying to make sure that they cover as much as possible, that they cover pretty much everything. And I really look forward to that. I get excited when I see companies covering new platforms. Even if it's not a platform that I use, having it cover, you know, all of the major SaaS or all the major cloud platforms means that it's more likely that companies are gonna be cyber ready. Yeah, yeah, excellent point. I think the, sort of the takeaway there is, in that time, that preparation time, when it's not a production down environment, is the time to figure out which boards, which screens are you going to pay attention to most. And so just knowing that is going to save you time in response. And that's an area where vendors can, of course, help as well by figuring out ways of monitoring and exposing those metrics. Yes. Yeah, absolutely. It's funny. When I started in cyber, there were not that many tools. We were writing a lot of our own tools. And within the space of ten years, I think we've gotten to the point where actually, fifteen years at this point, we've gotten to the point where the tools sprawl, so many great products. They all solve very specific parts of bigger problems. And so trying to figure out how to bring them all together is something that we definitely need the vendor community to help with. Kyle, I wanna talk to you about recent investigations and what you've learned. How do attackers exploit the fact that sometimes we don't do a great job of making sure that we have a consistent set of backups across certain data sets that have to be joined up, let's say? Yeah. I think, you know, it goes back to what Steven was was just covering. I mean, in the age of hybrid multi cloud, I mean, if you're relying on these cloud native backup solutions, each one may have a different set of procedures for recovery. And I think, you know, part of targeting those backups is the knowledge that, you know, if you're coordinating all that, and it's more than likely a very small team, you know, who's forced to coordinate the recovery across multiple cloud instances. I I mean, there's there's tremendous opportunity for human error or for, you know, for something along the the chain of the multiple steps you need to go through to recover your data to go wrong. I think threat actors understand that and know that it, you know, it tends to lengthen recovery times. So I think that's definitely one thing. And I hear people that I talk to mention it all the time about, you know, how complicated the process of say recovering, you know, Microsoft AD can be, you know, multi page, huge number of steps chained together. If any of them go wrong, you know, often the process needs to be reset. So I think that's something that threat actors understand, and if these processes aren't automated to the greatest extent possible, then, you know, it's gonna lengthen RTL. Yeah, absolutely, absolutely. So I wanna pivot, we talked about technology and best practices from an execution perspective. We talked a little bit about what the most important things are that we could have from a capability perspective like vaults. Next, I want to talk about how do we evidence this, right? So you could write a book on what the board may or may not understand about what we do or may brief them about technology, but I think the one thing that's really clear is that both boards and regulators have a different perspective than technologists do typically when we come to So I'd like to ask the both of you, and I'll start with you, Kyle, if you wouldn't mind. How does SEC, NIS two, DORA, how are they impacting the expectations we have around recovery assurance? Yeah. So I think we're confident in our ability to recover is no longer sufficient evidence that you are a resilient organization. Boards want to know, boards and regulators want to know how quickly can you recover, and when did you last prove it? So, you know, we talked a lot about drilling, testing, testing till your eyes bleed, John. That's one that I learned from you. I think that's critical, those tests to the greatest extent possible. Tabletops are great for understanding what sort of scenarios you may face, but to really test your ability to recover to the extent that it's gonna satisfy boards and regulators, you know, at Rubber Zero Labs, we we definitely are proponents of scenario driven, software driven crisis simulations so that you know not only what an RTO is and whether or not you've made the cut, but what processes, what phases in the recovery are taking the most time? So is scoping the issue? Is determining a clean snapshot the issue? The more granular you can get with those recovery metrics, the more you're going to be able to satisfy, you know, regulators and and boards that we do have mature recovery operations, and we're working to bring that time down. Excellent, excellent points across the board there. Stephen, over to you just for a second. In terms of, you've got vast experience in this world, would love to hear from your perspective, what are the artifacts that you think are best evidence that organizations should look to produce in terms of their ability to recover their ability to evidence for coverability? Well, I think that certainly practice makes perfect, like we've talked about, but I do feel like there needs to be a focus internally on proving the outcome of these tests. Not simply on saying, yes, we have good protection, like you mentioned, but on, We did a test of this, we did a test of that, we did a test of the other. And interestingly, I think that a lot of people are running scared because of GDPR and even though technically GDPR really only affects personal data and DORA really only impacts the European financial industry, those regulations tend to spill over into the rest of the world in a way that actually can help us. And I think that it seems strange to say that because for the longest time, I mean, you know, those those sorts of things are are pretty scary, you know, to have regulators coming down on you and financial penalties and things like that. But, you know, for for data protection and and security and storage pros, one of the challenges has always been getting the attention of of the business and getting access to the resources needed to protect data. I think one of the benefits of some of these regulations, and the impact that regulators can have if there are outages and security problems, is that you can finally get that attention that you need to solve these problems. So essentially, being able to if you're in a European bank, being able to go to your management team and say, look, we're going to get a fine that's gonna be some massive amount, 234% of our global turnover, if we don't have data protection demonstrably ready to go. Well, there's never been a better impetus to having a solution to this problem. And for the rest of those out there, I think that there's an opportunity for basically the folks who are responsible for these tasks, to go to management and say, Look, this is not just a technical problem. This is not just an IT problem. This is a business problem. This is Can we move forward in the marketplace problem? And whether that's because of regulators or because of regulations and penalties, or simply because of the reputational risk that can happen if you have a major outage or lose client data, All of those escalate the importance of data protection and cyber readiness in a way that lets us really get the resources we need to tackle these problems. Excellent points, and I think we could probably spend a whole lot more time on that. But I think you recapped it very nicely in terms of the things that organizations should be looking at, and the perspective that, you know, maybe regulators and regulatory bodies and standards actually can be our friend here in terms of justifying, you know, where do we want to dig in, how do we want to dig in, and why, more importantly. It's not just compliance, it's the outcome that we're all looking for, which is stability. Great points, guys. This has been a fantastic conversation, Just trying to bring it home. We have a lightning round. Just gonna hit a couple quick questions on the way out. Before I do that, I wanna again thank you all for taking the part in this conversation with us. I'm sure our attendees are gonna find this immensely valuable. First question over to you, Steven. What's one thing to test this month that will materially improve recovery confidence for organizations? Go look at your authentication. Look at your active directory. Look at that in that environment and ask yourself, do I have any kind of protection or ability to monitor this at all? And if the answer is no, start there. That's the first thing that I would look at. Because for the longest time, there was nothing you could do about it. Now there is. Go straight to identity protection. Excellent. Yeah. Excellent point. Couldn't couldn't agree more. And obviously, what we see is the first step you want to take is to secure the network when you think there's a problem. But if your identity source is compromised, there's a fundamental question that it's gonna take a long time to work through. And Kyle, similar phrasing, similar situation, but from a telemetry and a validation, a signal validation perspective, what's your advice? Yeah. I think it would be the attitude change of of recognizing that your backup data is an essential source of threat telemetry. Not only for, you know, searching for searching for threats in areas where endpoint detection may not have visibility, but also, you know, acting as a record of of threats that may have been missed. You know, if you've backed up a piece of malware, chances are you've missed it somewhere along the way. So continuously scanning snapshots and backup data for indicators of compromise is just another source, and I would argue it's an essential one. Excellent, excellent points. All right, last question to both of you and Kyle, stay on, we'll start with you. What's one myth about backup and recovery that you want this audience to retire today? You know, I'll sound like I'm beating a dead horse, but just that it's only valuable as a plan B. I think that's one of the most pervasive ones, and I think it does a disservice to security teams. Excellent point. Steven? Well, I love that one, Kyle. I completely agree. Mine would be that these things are fire and forget. That essentially you buy a backup solution, you implement it, and you're good to go. That couldn't be further from the truth. You have to continually manage, monitor, upgrade, expand, improve, test, validate, verify. This is a career. It is not a solution that you just sort of buy and turn away from. Yeah. It's a marathon, not a sprint. Absolutely. Again, I want to thank you guys for taking part and sharing your vast knowledge with everyone. I want to thank everyone for attending. This was an incredible session, and there's so much more that we can learn. Don't hesitate to take a look at Rubrik's Zero Labs or reach out to your Rubrik representative if you want to learn more. Thanks again for joining.