Name: The Bridge to Nowhere:
Uploaded: 2026-09-16T17:49:06.789Z
Duration: 33 min 8 s
Description: The Bridge to Nowhere:

Transcript for "The Bridge to Nowhere:": Welcome to our webinar. My name is Jeremy Kathy. I've been at Rubik for right around six and a half years now. Previous, career, I did a stint at the NSA for a short stint doing cryptography and red team analysis and then went into the private sector for incident response and secure architecture design. Came to Rubrik, again, about six and a half years ago out of a security led project to actually attack data protection and storage solutions. So really, really love to dive deep into why things are built the way they are, having purpose built architecture to help solve real world problems. And so one of the things that I've been noticing from a trend perspective over the last maybe eighteen to twenty four months is that there has been a significant ask from partners, from individual customers, and not not even from a geographic perspective, holistically across the globe, to talk about isolated recovery environments. And so I think there's a pretty good reason why this is a topic that's coming up. I think it's a natural shift in progression based off of what we've been doing from a cyber resiliency perspective. We've sort of moved the goal or the gate, if you will, that used to cause people to struggle with recovery, and now we sort of redefine that in a manner that we now have to think about this in a new way. And so, hopefully, by the end of the session, it will be clear that an isolated recovery environment or that type of architecture is a fundamental next step to help us prepare for what does that look like to ultimately be able to not only respond, but respond efficiently and effectively to cyber attack. So one of the things just from a level set perspective around modern attacks, what we've seen from a modern day perspective, I don't think any of this is probably news to anybody. I will say there there's a minute change that I would make to this slide. I I left it up here on purpose because I think it's important to know how quickly this is changing. So when you look at this, you see the, you know, twenty four days from a traditional dwell time. That's actually reduced significantly. Like, as in it's reduced tenfold in the last two weeks. So right now, we're seeing attacks that actually take less than forty five seconds to execute upon initial entry. K? That doesn't mean that all attacks are like that. We still see the average dwell time being seven to eight days. But what we're seeing is the prevalence of Agintiq AI allowing people to be able to polymorph their code sets to be able to execute more efficiently and effectively is causing them to be more, more expedient in what they're able to do in a shorter amount of time. But the idea behind this is is to illustrate a core concept in cyber recovery that's different from operational and disaster level recoveries, and that is in the fact that it's not the same. In operational recovery and a Doctor recovery, we know generally what the failure was. It was a power supply or a network outage or a storage array or a data center or a hurricane or something along those lines. We trust all of the data that we're recovering. We just wanna get to the latest point in time from an RPO and an RTO perspective. In a cyber event, that's not the case. If I recover to just before the attack, yes, I might bring my services back up and running. I might even not have some sort of corrupted or encrypted dataset, but I almost assuredly will bring back the very bad actors that did the attack to begin with. So I have to understand the context of what's going on from the attack. So when we look at their footprint, it really hasn't changed, a whole lot. Yes. There are different actions. For example, they might do deletion or wiper level attacks. Living off the land and just simply deleting the data to be annoying actually is a thing. Whether they're doing, corruption, encryption, hypervisor level encryption, whatever the mechanism for attack to the data is really designed to keep you busy in response for business services while they're running away with what they exfiltrated from a double extortion perspective. So data theft is the primary, and data destruction or interruption of services is the secondary goal. And so when we look at this, we sorta have to adapt, and we have to ask ourselves questions. We need to understand these things before we hit and we start doing recovery. So there's this expectation of, oh, no. I'm down. I maybe do some troubleshooting internally. I escalate it to security and saying, I think this is more than a server down. Think we may have a security incident. Security does some initial analysis for identification to detection, and then you're like, okay. Let's start recovering. We know we got hit. Right? It's It's not exactly what happens. We have a significant amount of time in multiple phases of forensics that have to take place. So whether it's, again, the identification, the detection, the quarantine, the eradication, There are multiple steps that have to take place before you get to the ultimate recovery phase. And, traditionally, this has been very serial. And what what I mean by that is is the recovery process is actually something that hasn't been able to happen until a lot of that remediation has finished. I can't start restoring into an environment that I still don't have thing things quarantined or eradicated in as an example. And so we have some assumptions, as I was mentioning, that people make, and probably the biggest assumption that I would call out on this slide is actually the first one. Even from a security perspective, I've spent my entire life doing security. I've talked with multiple teams across the globe, and almost every single one of them, they're gonna rely on some sort of log or SIM data, and they're gonna look at the production assets to see what's going on. What we actually see is in the metrics go up and down depending on which organization you're getting your numbers from, but somewhere around 47% of the time, you actually can't perform the forensics that you want from a security perspective in the production environment where the attack happened. As an example, a tactic where I come in, I load myself on a virtual machine, I actually attack you, but as a part of the attack, I actually encrypt myself. Now security has nothing to look at. They have to pay for the decryption key to decrypt the VM to then start looking at services and things like that. So this is a situation where it's not quite as clean-cut as we would like to think. I'd show of hands, who thinks that board level and C suite is saying we want we want less time from a recovery perspective versus we want you can be down for, I don't know, five days. Are we getting more aggressive with our expectations, or are we getting less aggressive? And I I would say probably everybody in here, maybe some of you even from a position perspective, are held to that standard of aggressiveness. And so we want the environment to be up and running much, much, much faster. And so if we already have a recovery process that we're measuring maybe from a quarterly or a yearly Doctor, Now we're introducing all these unknowns, all these assumptions that cause damage. It's pretty easy to see that the the timeline gets elongated pretty significantly. That's why we get to a number that is really a goal of what I my mission is from a rubric perspective is to reduce this, is the average time for recovery, holistic, not rubric, but holistic across the market is actually twenty one business days. It's not that it takes twenty one business days to recover. It takes, on average, eighteen to nineteen days to fully understand and eradicate, and then it takes between two and three days to actually start rebuilding the services. So this is a metric that we're we're trying to attack. And the reason that it takes that time is we have this iterative process where often security comes in, whether it's internal or external incident response, you know, whether it's Mandiant, whether it's IBM or Microsoft, CrowdStrike, whoever it is. And they're looking at things like memory dumps, PCAPs, network logs. They're trying to identify what is it we're looking for. I can't find something unless I know what I'm looking for. And so that initial process is a part of investigation. I wanna understand what am I looking for. And when that process happens, there's generally a handoff to your operational teams. And this process generally involves restoring assets to a point in time, waiting for a restore, scanning those assets with newly improved or newly patched security software to find the new thing that we didn't know about. We wait for a scan, and if we find it, we go back and we do it again. And it goes through this iterative time consuming process that takes the majority of those twenty one days that I mentioned from a metrics perspective. So if we look at that, if I have to go through that iterative process to understand what needs to be recovered, if I have to wait for security in many cases, upwards of plus 80% of the time, to finish with step four here, eradication before I'm able to do recovery, that is ultimately the the race that we're trying to win. K? So if I look at this, if I wanna try to reduce this, how can I decouple the forensics phases that we don't wanna rush? If we rush forensics, we get mistakes or incident response, we get mistakes. And if we get mistakes, we get reinfection. If we get reinfection, we have even longer down periods. So how do I allow my security teams to do their proper due diligence, but also not make the business wait an extended period of time to begin a recovery process. Couple of things before we dive into that key component. Number one is we need to understand the scope of the attack. And so a lot of customers that that I talk with are like, well, do I need to isolate a recovery environment for everything? I think it's best to have something, and I think it'll make more sense as we go through, to enable you to respond to situations depending on what they are. But in not all cases, are you actually going to recover into an isolated recovery environment as an example? If you have a small, maybe a DMZ server that got attacked or a small subset of your architecture, it may be very reasonable to assume security can analyze, can quarantine, can they can tell you what work what to recover back to, and operations are up and running. When we get into more of a systemic breach perspective, this is when we don't know those answers. I don't know how what identities they've taken over, human, nonhuman, what group policies they've changed. I don't know which systems they actually have their command and control software on. I don't know which systems they've manipulated or extracted data from. So when we identify from a security incident response perspective the type of of breach that we're dealing with. This is where we start that playbook. So I wanna talk a little bit less about what the problem is and talk a little bit more about what we're seeing from a solutions perspective. So isolated recovery environment. So a couple of things that I wanna say to this, and I I think this is super important because right now as an industry, and I'll be a little transparent, I think even from a vendor perspective, there's been a lot of misrepresentation and confusion on what isolated recovery environments and clean rooms and vaults are. There's been a lot of interchangeability of terms and descriptions and purposes for these. And so I think if you're gonna understand and have a, like, a universal industry definition of what isolated recovery environments are, you sorta need to understand what the core level definitions are. Right? So if we look at how technology has progressed, what we generally see is we started with backups, and they were for operational and Doctor reasons. And then security stuff started happening. So then we're like, okay. Well, now we need hardened immutable backups. Right? We need them to be immutable and indelible because the threat actors were actually attacking the data protection solutions. And so you look at around the 2013 to 2017 time frame, it was this race towards immutability and indelibility. K? We then shift from the fact that we haven't been able to achieve immutability and indelibility, then it gets into well, now we need a vault. We need an an extra copy somewhere. In some cases, in some solution provider solutions, there's upwards of three copies of the data needed, and there's opening and closing firewall ports and all kinds of wonderful operating things to get this vaulted copy. But that's really where it stopped. It's, okay. Well, if your backups get wiped out, if they bypass these mechanisms, then we know we have a copy of the data sitting over here in this vaulted environment. The problem with this is what do you do with the vaulted copy of the data? Ultimately, what we generally see is people are left with a copy of the data with nothing to recover to, no process to recover, and really slow speeds going through a firewall that they use to lock it off to try to recover back to an environment that doesn't exist. And so this is where there's been an industry sort of evolution saying that we have to have a little more available to us from a technology and a process and a and an intention perspective. So from an ice from a recovery environment perspective, it is designed not to secure the data. That needs to be done no matter where the data sits. It is designed to protect the architecture and the processes that are leveraged for recovery. It is a mechanism to ensure protection of the actual recovery process itself. Okay? What it is not, like I mentioned, is a vault or a random copy of data sitting somewhere. And so, unfortunately, I see a lot of people when they're asked by regulators or asked by, board level initiatives that are looking at cyber resilience, They're designing their response to an isolated recovery environment being they're trying to design here. And what happens is is they do check the box. They have a, quote, unquote, isolated copy of the data. And at that point, if they don't get attacked, everybody's happy because they have they've achieved the the initiative of checking off the box. When they get attacked, things aren't so happy. It's because then they realize all the pitfalls of not having the ability to leverage or use the data effectively, that sits inside of that. So I wanna talk about what my and I say my, from a from a Jeremy perspective, but I also align this. If you wanna look at Mandy, you wanna look at Microsoft, you wanna look at multiple other security level initiatives, what my definition of an isolated recovery environment is from, one, we'll start with the purpose, and then we'll talk a little bit about architecture. And I'm definitely willing to dive into specific deep dive architectures and examples, as as follow-up action items. But the first thing that we have is this forensics environment. And I I call this a forensics environment purposefully not using the term cleanroom as an example because a lot of times people think cleanroom is the entire thing. K? And, yes, it is that idea of if you think of, like, super of microconductors and superconductors where you have this static clean room that you can do production and one granular of dust will ruin the whole thing. Yes. That concept maps over to an IRE, but the forensics environment here is literally used to understand what you've been attacked with. What happens when I do get in that situation where they attack me, but I don't have the ability to look at production? So this is an area where I can go in and I can actually engage in sandbox controlled, secure detonation of payloads that I know are dirty. And I can watch the processes and the procedure calls and the command and control tools, and I can map out to what are my indicators of compromise, what are the command and control tools, what are the malware packages that are actually causing my particular breach. This is a step that cannot go away. Now certain things might might make that step go quicker because it may be that you're hit with something that's known, that's not super evasive, that isn't super complex, depending on the the attacker or who you are generally based off a nation state perspective. But this is an area where I learn. I'm seeking to understand the answers to those questions that I laid out. And I use this to then feed my recovery process. This is the step that is not ran, or changed except for the fact that an isolated recovery environment gives me a guaranteed place to start this immediately. I don't have to wait for somebody to approve me to do something to production. I don't have to wait for the yellow crime scene tape to be cut. I can immediately start learning. Okay? The second stage of this is called the staging environment. A lot of people will say, alright. Well, we found the clean copy. Let's start restoring and let our users have access. It's really hard to travel on an airplane that hasn't been built and validated yet. So the staging environment is meant that once I understand what I'm looking for, this is where Rubrik kicks in and says, hey. We can help accelerate. You tell us what to find based off of your forensics, and we will go isolate and tell you where that is really, really quickly, exponentially faster than the competition. And so if I'm able to learn what I'm looking for, I'm able to find it really quick. Now while forensics is still engaging in the rest of their process and production from a remediation perspective, I can start restoring my workloads into this staging subsection of an IRE. I can have my application teams validate and make sure that they're functioning from an internal perspective. And then, ultimately, the goal is that I can move or migrate those workloads from a network perspective to be customer internal or external facing, the new secure production environment. And this is the idea of my business is back up and running while forensics is taking their time understanding and learning and remediating from a production asset perspective. K? So if I look at this, you know, I I like to I don't look at slide notes. I don't look at any of this stuff. So the reality of it is is I probably talked about most of this, but you can see here from a high level in a forensics environment is to do the initial detonation, the initial learning. It's to be able to understand before I take action. K? Significant benefits from the fact that I can do this without threat actors seeing me. I can do this completely out of band and invisible, and I can do this on my own timetable without worrying about what I'm stepping on from a chain of custody and a preservation of the attack data perspective. And the staging environment, again, this is where I execute my recovery plan. One of the benefits that isn't often understood here is the fact that the secure production environment allows you to do this in a time of war, but also inside of a time of peace. Operationally testing your disaster recovery plans. How many of you recover your Active Directory forest on a regular basis to test how it's gonna work? That's pretty hard. And especially if you get into Entre ID, hybrid connections, how do what do I do with my Entre ID? So the idea of having cloud tenants, on prem tenants as an isolated recovery environment extension that allow me to execute and get better at my recovery process, this enables me that when I do get hit, I'm not doing something for the first time. Right? And then, ultimately, again, from a secure production perspective, this is the idea of I'm bringing the business back up and running all the while I am not waiting for the remediation process to finish. Right? It's decoupling a serial timeline and making it work in parallel that drastically reduces the overall recovery timeline. Here's a very high level example of a Rubrik architecture. If you wanna get into application centric, integrated Cisco ACI networks and contracts and policies, I can get down to the level that you want from a from an in-depth perspective, but the concept is your production environment completely separated from your regular environment. You are not opening ports and allowing communication in at any point in time. It is only outbound connection enabled. One of the preliminary, I think, faux pas or misbeliefs is that it that you do not have the ability to have Internet access in inside of an IRE. I would actually disagree with that. A lot of tool sets and a lot of management mechanisms require that, but what you don't want to do is allow that to be an interactive in and out session. Right? There are ways to do that from a security perspective. So this is why if you guys I don't know. Some of you may be Rubrik customers. Some of you may not. Some of you may be Rubrik customers that are new or existing, but this is why Rubrik fundamentally changed our architecture of how we do replication, that we do outbound over port seven seven eight five TCP only so we can enable that to be inside of an isolated recovery environment and not have to open and close firewall ports. I'm not gonna go into the specifics, but Rubrik is designed to accelerate the discovery process that feeds out of that initial, what am I dealing with? Right? So, yes, we have to keep the data safe, immutable, indelible, zero trust architecture, not relying on any external components and third party vendors, all the wonderful things that make us work and work us fast. But, ultimately, it's give us what you're looking for and allow us to search through petabytes worth of data and return the results much quicker than security and incident response can go through from a normal process perspective. K? So, again, I I I call this a little bit of a paradigm shift. It used to be that it took so long to recover and scan and recover and scan and recover and scan that, actually, forensics and security were done eradicating before the restoration process began. So then you went into a rebuild and then a restore process. Because of the ability to say, okay. I know what I'm looking for. Now can you find this without going through that process? What we've done is we've sped up the ability to understand what to recover, and now people are left proverbially twiddling their thumbs, waiting for a place to recover to. That's the primary reason why isolated recovery environments are being discussed so much at this point is because I need to decouple, and I need to remove the block or the gate so that I can start my recovery process without waiting for for the, full incident response remediation. So if you look at a timeline compares comparison where you're doing this in a serial perspective and you're waiting for things to get to remediation, that's where you begin the recovery process. So your your downtime is determined significantly by the processes that that that, predicate when you can start that. If I look at this from a different perspective with an isolated recovery environment with Rubik's capability to be able to use our data threat analytics to find clean, now what I get is I get a process that has to happen. But now while this is happening, I begin and I start my recovery process, and it drastically reduces the amount of time from the overall recognition of an attack to the recognition of services back online from a minimum viable perspective. Here's a little bit of a a a just a internal thing to drive some questions. Right? Like, assessing, hey. Where do you think you are? Like right? Because resilience is a journey. It's it's not a it's not a I start and I stopped and now I'm good. It's gonna be a continued evolving thing. So I'm gonna run just a brief level simulation here. This is just Mint. This is nothing super special or amazing. This is a example. Obviously, no information, no who who it was or even who the attackers were. But this is an example. This is a walk through of actually what happened with somebody from an attack perspective. K? So from the initial phish to walking through and actually watching somebody take over an identity environment, creating the counts, changing the group policies, slowly progressing through an environment, laying the IOCs inside of that. Even from a command architecture, it I put a lot of this in small print, and I hopefully you can't see it because those are actual real commands that can be ran. But the idea here is it's very easy for somebody to get in and to do this. Like, I can do it right now in a lot of environments, and it doesn't really matter what your tool sets are because it's a live off the land approach that emulates real world users. All I need is a user to make a mistake, and that's why we see a lot of logins versus break ins. And once we see that log in, then we get ultimately to this type of approach. So from this perspective, this was an actual hypervisor level attack. So not a virtual machine or a workload or a file set. Literally, the entire hypervisor environment in both data centers production and disaster recovery were simultaneously encrypted. Hardware encryption is really fast. Software decryption is really slow. It leaves you in a really, really bad situation. So from that perspective, if you were to look at tool sets, for example, Rubrik anomaly detection, reaching out and saying, hey. I got a problem. Like, not not at a guest level, but at a even a hypervisor level, I have a problem. There is something that needs to happen. Incident response and security getting in, and we this is where we choose from a holistic perspective in this situation to actually engage in the isolated recovery environment process. And so with that, as mentioned, throughout the the presentation, we wanna start the initial forensics inside of that clean room. Right? So I want to take a look, and I wanna search for, the threats. I wanna live mount my workloads or instant recover my workloads into the clean room. I want to watch them. I want to learn what I'm looking for from a GPO, from a, from an identity, from a individual system, from a reg registry and a, command and control tool perspective. And I'm gonna use this to feed into the Rubrik Security Cloud to say, I need you to go find these things at this point in time, and I want you to surface where they are on which recovery points and give me a timeline of how this has progressed across the environment. That is the intelligence that I use while my incident response and security teams are engaging in remediation to launch into ultimately the staging environment. Generally, as an example, just a little further into the question that was asked around identity, generally, identity is where you wanna start. Having an application up and running does no good if I can't log in to it. So this is where we have designed our identity resilience portion. This is the goal behind that is to rapidly allow for the rapidly allow for the orchestration and recovery of the identity services. We wanna identify what has been changed. And it's more important not just to know what the changes are, but to be able to leverage a way to restore before, but roll to after and not bring in the actual malicious changes and edits that were made. I want to roll forward the good, but leave out and sift out the bad. And so that's the the capability of identity resilience roll forward. And I'm I'm sure if any of you are going into the identity classes and breakouts, you'll you'll hear a lot more on that. But if I can take my identity services and I can recover those, then this is where I'm going to get into my staging, where I take and I leverage the intelligence that I've gained, both from the security teams as well as the Rubrik output from the analytical the data thread analytics, and I'm going to start actually restoring and testing those inside applications with just my application teams. I'll then leverage migrating those into the production environment once they're validated, still stringent security oversight, significant logging to make sure that we're not bringing something back into the environment, and we leverage this to ultimately bring up and start to recover our workloads one at a time from a minimum viable perspective. Right? I have to get to these services as quick as possible. Once I've done that, I'm operating in a temporary environment. What I mean by that is is it's not meant to be used to run your production environment all the time. I doubt most people have the budget to have just yet another copy of all their hardware for all their performance and all the services of of all the things that are important to their business. This is designed to give you minimum viable. You may have eight, you know, Oracle nodes for your production database in production. You may be running on two Oracle nodes inside of your IRE as an example. It's to allow users, but not necessarily the full breadth of users from a from a numbers perspective. So that's the end of what I had to present, you know, trying to present high level, but be willing to dive a little into the deep here. So thank you guys for for your time and for coming into the session. Thank you very much.