Video: How to Build Observability Dashboards That Drive Decisions | Duration: 3612s | Summary: How to Build Observability Dashboards That Drive Decisions | Chapters: Welcome and Introduction (5.92s), Speaker Introduction (132.33s), Dashboard Demo (273.48s), PerfStack Dashboard (471.555s), IT and OT Dashboards (612.825s), Dashboard Psychology (817.47s), Dashboard Color Rules (1164.35s), NOC Screen Design (1323.665s), Actionable Alerting Philosophy (1478.55s), Dashboard Context (1596.58s), Dashboard Widgets Integration (1710.11s), Customization Power (1949.31s), Root Cause Analysis (2125.685s), Building PerfStacks (2432.585s), Dashboard Iteration Strategy (2728.99s), SLA Best Practices (3007.86s), Communication and Iteration (3371.195s), Engaging Through Examples (3476.105s), Closing Remarks (3579.2s)
Transcript for "How to Build Observability Dashboards That Drive Decisions": Alright. It says we're live. So welcome, everybody. Welcome in. Let me know in the chat if you can see and hear us okay. I'm gonna give a few minutes here at the top. I don't know what that light flash was on my camera. Good. Okay. I got a thumbs up. Good thumbs up. K. Good. Just checking. because we definitely... I was definitely in a webinar a couple of weeks ago where... What we thought we were live and we weren't live, so I just gotta check. Welcome, everybody. This is the uptime hour. It's our new semi regular, live demo, webinar that we're doing to help, with how tos and answering questions on how use observability self posted in the module specifically. So, if you're interested in what we're doing here, we did one previously. I did custom properties deep dive, last... The last time, which was was in September, is available on demand. I can get you a link here in a little bit and pop it in the chat. Talking about use cases, places where you can use custom properties, including dashboards, which is our conversation topic for today. I'm going to introduce Scott Cross who's here to help me out today. I invited him to come and give a talk he's been giving this year about, dashboards that is really good, and he's gonna show us how he built those dashboards. And we're gonna be able to answer you guys' questions as we go along. Little bit of housekeeping. If you have questions you'd like to ask during the q and I q and a, you wanna make sure that we see them, put them in the q and a tab, and, we'll be sure to see them. I will try to keep up with the chat as well, but it's a good chance that I get missed. Any questions that we don't answer today, I will endeavor to get answers for input on Slack later. So if we don't get to something, I will make sure that the answers are out there somewhere. So keep an eye out for that. And to answer the question at the top, this will be available as well on demand. So if you have to step away because, you know, we work in IT, things happen. If you have to step away, you can always catch up later. So without further ado, stop cross. Good morning, everybody, or afternoon, depending on where you're at. My name is Scott Cross. I am the principal solutions strategist for SolarWinds. So just a little bit of background about myself. I've been doing SolarWinds for about twenty years. I learned it when I had to build a billion dollar knock from the ground up using the product. I had a 99.99% customer facing SLA. So that gave me four minutes and thirty seven seconds of downtime a month for 33,000 devices in 40 different countries and a 180 different data centers. So a lot of experience with the product, a lot of real world experience with the product. I joined SolarWinds about ten years ago. Actually, it's gonna be 11 coming up in February. And I was specializing in the professional services side of things. I'm going in and working with DOD, PubSec, and and our commercial customers and helping them actually set up the product, get it configured and customized. And, you know, when you do that kind of work, you learn some really interesting things. You get to see what a lot of different people are doing and how they're doing it differently than than maybe others are and what works and, more importantly, what doesn't work. And I should also share that I have a little bit of a different background because my background actually isn't in IT, although that's where I came from the last, you know, twenty years. I originally started off as a psychologist. And the reason I share that is because I think of SolarWinds very differently than most people do. I think of the SolarWinds platform as a big behavior modification system. So the goal of SolarWinds is to get the human to take the right action or to get the... Or... And with automation, automate the correct action based on the data that we're receiving. So I don't get caught up in in a lot of other things, and you're gonna see that in my dashboard. You're gonna see how clean these dashboards are and how we're trying to share, data with a certain type of person, whether it's technical or nontechnical. And, yes, I just said nontechnical. We are actually going to start today with creating dashboards for nontechnical users. Now you're probably looking at me going, why are we doing that? I don't want some worker going in that doesn't know anything about IT and looking at what's going on. And what I'll tell you is that if you can get your nontechnical executives going into SolarWinds in a very safe way, and I'm gonna show you how to do that, it's going to make your life better. First of all, it's gonna remind them of how good you are at your job. You know, the dashboard that I'm gonna show you, I've always said, these dashboards are your performance review every day. One thing I always say about IT people, we are great that when we solve a problem, we're awesome at high fiving each other. Hey. I just did this. I just solved this. I just figured this out. We're great at that. We are horrible at telling the rest of the world how good we are at our jobs because they don't get it. They don't understand it. And half the time, it bores them, so we don't do it. So this dashboard is gonna be your ability to tell and remind people how good you are at your job. And so it's a really different way of looking at it. So we're gonna start off with a a demo. I'm gonna go through about ten minutes of demos, and I'm gonna go ahead and share out my screen now and grab this window. Okay. And so you guys should be able to now see my screen. And what we're gonna be doing here is I'm gonna start this off with a demo for a hospital. And the reason I use a hospital with my demos is because everybody's very familiar with the things that go on in a in a hospital, and it's very easy for us to extrapolate what we're about to see and and extrapolate it into our own environment. Okay? And we're gonna start off on the IT side, and then we're gonna look at the OT side in a little bit. But for this demo, I'm going to show you a a dashboard that I created for the chief of staff or the CEO of a hospital. Now these are really smart men and women. Right? They usually have an MD, and they've got an MBA. They know nothing about technology. All they know is that they need the technology available to do their job. So I wanna show you what their dashboard actually looks like here. So I'm gonna take you over to their dashboard. Now they can look at this dashboard and immediately see what is the status of the technology that they rely on every day. Is a patient admission portal working, the lab ordering site, medical billing, medical records, the radiology software, email, all the things that they rely on to do their job every single day. Right? But this dashboard actually does two things that most dashboards can't do. First, this dashboard only changes color if the issue is customer or user impacting. In other words, I do not wanna create fear or confusion among my users or my customers. So if I have a web server farm of 10 web servers and I lose a web server, this isn't gonna turn colors. They don't need to know that. That's an engineering issue. Right? But if I lose eight of my 10 web servers, yes. This is gonna turn colors. It's gonna say, hey. We have here a, you know, a a problem where email is down. Don't email the lab about your lab results. Pick up the phone and call them so they know to take a backup action, but it's very easy for them to understand. Now remember, as they walk by this dashboard every day and it's most of the time green, it's gonna remind them every day, hey. Our IT team is on top of it. Everything is working as expected. The second thing this dashboard's gonna do that's a little bit different is this dashboard is actually used by the engineering team, but you're gonna have access to things that they don't have access to. So for example, we can see here email's having an issue. You're gonna be able to come in here. You're gonna be able to click on this performance analysis. This is gonna take you over to our PerfStack dashboards, and it's going to take all of the components with email and put them all together on a single pane of glass. So you're gonna be able to see your networking information, your server information, your application level information, your virtualization, level information, and you're all gonna see in a single place so you know, you know, where your your peaks are, where your valleys are, and you'll be able to dive in and see what's actually going on. The other thing you're gonna be able to do is you're gonna be able to click on this icon here. This is gonna take you to a live SolarWinds map of your email environment. It's gonna show you who's having problems or, you know, at what layer is that problem actually occurring. You you know, am I having problem on the storage layer, the database layer, the application layer, the network layer, the server layer, the virtualization layer? So I'm having a communication issue, but where does that communication issue lie? Is it on the application level? You know, where is it? Now it tells you which team or teammates you need to get involved. Now I'm gonna be honest with you. Don't underestimate the power of this particular dashboard. And the reason that I say that is one of the things that I learned having four minutes and thirty seven seconds of downtime a month is I had exactly zero seconds to spend arguing over whose problem it was, and that used to happen a lot today. The server team would point to the application team. The application team would point to the network team. You know? This solves that problem really quickly. It says everybody is looking at the same data. They're looking at the same dashboard. I can see I have an application level issue, and now you can come over here and click details and go to another perk stack that's showing you what's going on with the actual application here. Where where does my issue lie? And now you know where to go for that. So when it comes to dashboarding, dash... As you know, dashboarding is extremely important, but I wanna let you know that there's one thing about dashboards and about observability and monitoring that we don't think about a lot, but it's actually one word, and it's actually the most important word when it comes to monitoring and observability. It is also the word that we forget about the most. Okay? And that word is outcomes. Now I'm gonna prove that to you. Okay? I am going to take you over to my Star Wars dashboard. So in the movie Star Wars... Think back to Star Wars a New Hope. Okay? In the movie Star Wars, if the Death Star had had a knock, that knock was running Solar Winds, this would have been Darth Vader's executive dashboard. Now he would have known when the tractor beam went down or the trash compactor stopped running or the detention center cameras went offline, and he would have been able to react to that in an appropriate manner instead of guessing what was going on. I always tell people it's not if something's gonna go down. It's when. And being able to minimize or prevent that disruption is absolutely key to our success. So when I put it into this kind of, frame, it lets you see that, yeah, this could have prevented problems that happened, and they would have been able to help resolve those problems directly. So kind of looking at the IT side of the house now, what about OT? What happens on the OT side? Well, on the OT side, it's very similar, and we're gonna be looking at at at what that looks like. Now in health care, for example, the IT side and the OT side, a lot of times, they they communicate with each other. Right? But in a lot of our other environments, whether it's, you know, gas and oil or it's water, or public sector or DOD, you know, they can't communicate. So you may not have that connection that you're seeing here, but for our health care systems, they actually do, you know, communicate back and forth, and so we can go ahead and see we have an OT level issue. Now, in this case, we're looking at the radiology softwares, the PET scans, the MRI imaging, the ultrasound, and the X rays, all of the different radiology pieces that make us up. And here we can see we have a CT imaging problem. Now we can click into that performance analysis to get a a better idea of all the components that SolarWinds is is viewing as well, but we can also click into our map again. And, again, when we come over to our map, now we're seeing our OT side and our IT side. Because it's important to remember that in any OT environment, we are going to have an IT component to that. Even if your IT environment, your main IT environment, and your OT environment don't communicate, there is still gonna be an IT environment within that OT setting. And so here, you can see the CAT scanners talking to the network switch, which then is is communicating with your EPYC database, your PAC server, and your image storage system. Right? All three of those need to be communicating without a problem. And if you're not familiar with, with health care, Epic is a major software vendor, and they handle and tie all this data together from your pharmacy to your doctor visits to the radiology piece and put all that data into a a big database. So here, you're grabbing the images and putting it into that Epic database, and you can see that your PAC Server has an issue. And, again, you can click on that to get more information. So the whole goal here is to really be able to see what's going on within your individual environment and communicate it in an effective way. So I wanna stop here. Crystal, have we got any questions right now? Because I I do wanna start talking. a little bit more about the psychology of dashboards, but let's get to some questions. Okay. First of all, I love that you are gonna start talking more about psychology of dashboards because I do think this is critically important. I think one of the things that you... So what you've shown so far illustrates is how important it is to think about the role of the person who's viewing the dashboard and how they're gonna be getting perceiving that information and then using that information to make decisions. But someone asked a specific question, which I'm gonna ask you to elaborate on a bit, which is, is the email topology map static? So you've shown several maps as part of this so far. Can you talk a little bit more about them? Like, are they static? How do they change? What are the dynamic updates that we're getting on these that you can include in your dashboards? Sure. So a lot of this is is is static work, but using groups, all all you're looking at right now... So let me go back here to our our our our original, dashboard. So all you're looking at here are pretty groups. All I... You know, I I don't mind the the the SolarWinds icons and stuff, but I wanna make it something that looks familiar to them and that they can relate to. Again, that's what we're gonna talk about a little bit more in a little bit. But all these are our groups. Okay? And then when we got to the other map, over here, all these are our child groups within the radiology... Within that radiology department. So you might have your radiology group for your radiology software, and then you're gonna have network, application, servers, database, virtualization, all of those different pieces again that are going... So all... When we're looking at this, all you're seeing is a bunch of groups. So I have my map here and then the groups. Now the groups, as we know, can be built out dynamically. Right? So if you. have a great naming convention or you have a a great IP scheme or you're using custom properties already, utilize those so when you do add a new device in there, you can build it out. A lot of my maps are originally built, though, using the the intelligence feature in SolarWinds. So I'll use the intelligent maps feature, start with that, and then build it out as major architecture changes. Because one of the things you're gonna... That I've learned is that when it comes to a lot of these big systems like us, we're not making major architecture change... Changes on a regular basis. It's, you know, this is the architecture. We may replace, you know, components of that architecture at times, but to completely revamp doesn't really happen very often. And so when you have these maps, you're gonna have them for two, three years. The other thing you can do also, and I know this trick has been talked about before, is if you're using Visio, use that as your background and just put the little green dots on those maps, you know, for... That indicate each node. So you're using maps that your team is already really familiar with. So there is some auto... Automatic work and some dynamic work that can happen, and then there is some static work that happens. One of the things I will tell you, though, is when it comes to your nontechnical users, you wanna make it pretty. You wanna make it easy. No. Otherwise, it it creates fear, and fear without hope, which is something else we're gonna talk about in a minute, is, is not a good thing. So one of the things that you you you learn as a psychologist, it's kind of an interesting story, is when I was telling people about a problem, you know, what their issue is or their diagnosis was, you always wanna, you know, be very apathetic to them and and and help them learn about it, but you want them to know that it's serious enough to take action. So what I would do is you you create a little bit of fear because it's scary. You don't do it on purpose. But the more important thing is what are the steps to get out of that? Giving them the hope, right, is really important too. So, okay, this is what you've got, and this is how we're going to to solve it and taking them through that process, and it's really no different in in observability. When they see the red, if they can't interpret what that red means, it's going to create fear for them. If they understand what it is and you can then provide them with an ETA of how long it's gonna take you to resolve it, now you've given them the hope and the understanding that, yes, it's going to be okay. Excellent. Yes. They, the... Another question that we had was, is this dashboard out of the box, or you have to build it manually with your devices? I'm sorry to tell you that this is all custom dashboards that Scott has built. If you are interested, if he wants to share it, maybe we can convince him to go share it over on black, so people can look at it further or, or diagnose. But I'm hoping also that Scott will share a little bit more on, the how to of building something like this so that you guys can go build in your own environments. Absolutely. And there's... If this isn't a lot of custom work, it's using the features that SolarWinds already has. So there's not a lot of SQL here. There's not a lot of SQL here. You know? If I... And and I'm gonna show you this. My... The lab's a little bit, crazy because it's our our engineering lab. So there's gonna be a lot of stuff in there. But if I go to my manage groups, and I'm gonna actually come down here and go to my SP, you can see my groups down here. You know? And that's what these are. They are groups with devices in them that meet those particular, areas. So all you're doing here is you're really building groups. So while we have a chance, let's talk about the psychology dashboards because I know this is a little weird to do on a technology thing, but it's really, really interesting when you start to understand dashboards. So I'm gonna share a couple rules that I have, and some of them are gonna make you laugh, and some of them are gonna be like, okay. That's a little weird. But, you know, they're just things that I've learned over the... And they work for me. You might wanna tweak them. You might, feel differently about it, and that's okay too. You may have learned something different, and I love to hear from people. Hey. I've learned this trick when it comes to dashboards, you know, because it makes people more comfortable. It helps them understand the data. So my first rule, the only exception to this rule, and I'll show you the exception, is for dashboards for nontechnical people. If it's a dashboard for nontechnical people, this rule does not exist. Okay? And the rule is I hate green on my dashboards. Okay? And I know that sounds weird, but green makes you feel good, but it doesn't really tell you anything. Okay? It's a big hug. Now that said, I want my executives and my nontechnical users to have that hug. I want them to feel very comfortable. Your IT team has this. You know? We we are on top of it, and so I want them to have green. But I want you to think about the ultimate dashboard in your life. Okay? And the ultimate dashboard in most people's lives, whether they're technical or nontechnical, is their car. Okay? Now think about your car's dashboard. Now you don't have a light on your car that tells you that your brakes are working or that your transmission is working. You have lights on your car that tell you that... On your dashboard that tell you there's a problem. Right? And so it's really, really important to say, okay. What do I wanna share? So when I'm doing my dashboards, I don't wanna play where's Waldo, like, trying to figure out where the the the green is or where the red is. You know? I don't wanna have to search for it. I want it to easily pop up, and so it's easily identifiable to say, oh, okay. Here's my problem. This is where I need to go to solve it. So in order for that rule to exist, one other thing has to be true, and that is you have to trust your monitoring and observability infrastructure. If you don't trust it, that's why we have green because we've gotta remind ourselves, yes. It's good. Yes. It's good. Yes. It's good. If you have something that is showing you just issues, you trust that if it's not showing you a problem, there isn't a problem. So I'm not trying to prove that things are good. I'm trying to prove that things aren't, you know, in a nominal state. So it's a little bit different of a way of looking at things, but, again, every time you drive, every time you push on that brake, you... Your expectation is that the car is gonna slow down. Do you have that same expectation when you do things in your environment that if you do something incorrectly or cause a problem, that your observability platform is going to catch it. And if the answer is no, you really wanna go back and you wanna look at, okay. What do I need to be doing differently? What do I need to be looking at? What do I need to be watching? And then so number two, a lot of the... Like, you have knocks, you're running knocks, and you have, you know, rooms with big screens, and they have a bunch of different, dashboards on those screens. In my knock, and I'm gonna tell you right now, and I always laugh because my marketing department used to yell at me about this when I ran my knock. In my knock, one of my screens was completely blank. It was a a blank solar wind screen. And the reason for that is I only had things pop up on that screen that were a problem and it needed to be dealt with immediately. And the reason I did that is with all the screens and dashboards within a NOC environment, they're flashing, they're all different colors, I needed something that they could easily identify. Wait a minute. There's an issue here, and I need to deal with this immediately. And that was a great way of doing that. Now nobody's gonna run up to that mock screen and click on it to see what's going on, but now they know where to go on their laptop and say, oh, wait a minute. I'm having a problem with, this network device, for example. You wanna make sure those lists don't get too long. If you have a list of... Long list of down devices, if you go and you have a list of, let's say, 15 down devices on the on the down device widget, if you get a number 16, are you really gonna see that? Are you gonna be able to tell that you just went from 15 to 16, especially when all those devices might have similar naming conventions? You know? You might have it named, network switch one through 15. Are you gonna notice that, you know, one, five, seven, eight, and 10 were up there, and all of a sudden, nine's up there too now? So you're not gonna be able to differentiate quickly. So I like to have a very clean screen that's gonna show me those problems. Crystal, is this making sense? You know, it's interesting. Your philosophy about showing basically blank screens or not having all greens is very interesting. I don't feel like, I've seen that before, I guess. I I like the gear... In my mind as you were talking about, sort of a blank screen and only things pop up that need to be addressed right now. I was equating that back to a very common talk that I give around, actionable alerting because that's how I feel about actionable alerting. Like, if you're not... If you're getting an email, you should be able to take action on it right now. And this is kind of just an extent... A visual extension of that in my mind, for the not, like, specifically. That was a very specific scenario. I don't think you would wanna stare at a blank dashboard, like, on your on your old screen, but, like, in a knock screen. That kind of does make a certain amount of sense. Like, if it's on there, you probably need to address it right now. Yep. We did have some more, questions that. popped up if you want to answer anymore. Sure. I do I. do wanna add one thing just before we get to the questions that is, when I when I was running my Mac, I also had, screens for the different products in the different product areas, you know, where my developer sat for one product and where my developer sat for another product. They would have their own screens and their... That that were up there, and those would actually also be blank. I didn't want to bother them with glancing up and having to see. And this way, when something popped up on that screen, it was very visible to them as well. But let's go ahead. Let's jump into some questions. I think, I'm just gonna... I'm gonna add on to what you just said too, which is that, it's it's something that we haven't really talked about, but I think is very important to talk about for dashboards is that it's very important to show people things that they're responsible for. If they're not responsible for that, it's just noise, and then we wind up in a place where they're not taking the data seriously that you need them to take seriously because they're so used to having noise on their dashboard or in their alerts or on their reports or whatever you're doing. But, like, specifically dashboards because we're talking about that today. If they have things in there that are not relevant to them or they can't do anything about... Even if they ask you, like, for just information, I found this to be true quite a lot, which is that... It's... But, like, an app owner might be responsible for the application, but they're not responsible for the database and they want database things on there. They might think that they want all of these database metrics because, you know, it's gonna tell them something. But if they're not database administrators and they don't understand how it relates to the health of the database, or their health of their application. All that does is create confusion and, like, enhances that blame game between teams because now they're gonna be bothering their database administrators constantly like, hey. I saw this metric is red right now. What's going on? So, like, it's really important to have the right context on your dashboards, if the things that people are looking at. If all they're needing to see is things that are important to them, then you need to, as your as your SolarWinds administrator, need to ensure that the things that they're seeing are important to them and not superfluous information or information that they can't do anything with. You know, it's funny. That reminds me of a really funny story about, seven, eight years ago, I was working with a customer. And one of their asks was to make sure that the server team could only see server data and the network team could only see network data. And I said to him, well, we can create dashboards specifically for them. He goes, no. No. No. He goes, I wanna make it so they can't even see the other team's data. And I said to him, like, that's really weird. Why? He goes, because, otherwise, they're gonna spend all day looking at the other team's data, pointing out what's... What the problems are. Yeah. And I started to laugh. I'm like, that sounds like it's a culture issue to me, but the... You know, he really... He was trying to solve a cultural issue with a dashboard. Mhmm. And it's like, if if you can see the data, that's great, but put the data that's really most important to them front and center, and then make them go and look for other things if they're concerned or they... They're interested in that data. Yeah. Someone asked a specific question about when your dashboards, which was, can the dashboard show the cause in the dashboard itself instead of having to go to a perf stack and check manually? Now I know you can put perf stacks on your dashboard, so that's another thing to consider. So you could put it in the same place, but, Skye, I was curious to see how you respond to that one. So I... I've actually had that, before, and and we we... I actually used this before. So on a dashboard that you're seeing right here, we had SLAs. And so each one of these had an SLA, and they were customer facing. So if we missed the SLA, it would end up costing the company actual money. Mhmm. And I had an internal SLA. So my internal SLA was 99.995, and my compensation was tied to that number, a large portion of my compensation. So if I missed my internal SLA, I took a hit on my paycheck too. And so very important. So what I would do is as I got close to my internal SLA, if one of these, business services became a problem, I would actually add the PerfMon widget right over here. So... At least on a Knox screen so they could see it. And so we could actually look at the data immediately and see what's going on. But remember, you can have multiple widgets on a a dashboard. And when a dashboard is this simple, if you make this dashboard much smaller, you're still gonna be able to to make it out for yourself. I actually have, one customer that has, you know, multiple dashboards like this, and they actually have it. They... They've turned them into a widget dashboard where they can see five or six of their, their executive dashboards in one view. You know? Now their executive dashboards are broken up by the executive team, and it's a very large company, so it works for them. But imagine this being one widget among five or six other other dashboard widgets. So you can have the perf stack appears another widget next to, you know, the dashboard. Remember, all you're looking at right here... And, you know, this will help make sense. This is how I configure my dashboard. I came in. I made it into a... It's a network map that I've just moved groups onto. And so you're able to actually see that this is a list of all of our dashboards or all of our network maps that we have in place. And so you can, you know, utilize that to be able to, you know, add what you want, but all this is is an individual widget on a dashboard. Scott, could you real quick? Could you switch to one of the tabs there that has, links showing on it? Someone asked a question about that and, went to... yeah. This one's got leaks right here. No. Not... No. The, sorry. The link status between between nodes or between things. Let's see. Let's do this one here. Yeah. So someone asked in the chat if the links only show up when they're red, and the answer is no. They do show up when they're green. Whether or not links are on your map does depend on how you set your map up. So you can choose to have links, you can choose not to have links, and you can ascribe those things to individual, like, interfaces if you want to get specific interface information on those. So So it's really dependent on how you set your map up. I did also want to address... Someone did say it would be really nice to have an in-depth, topic on topology maps, or on the maps intelligent mapping feature. So stay tuned. Maybe I'll do that in a future one. We don't have... I don't think enough time to really dig into maps, in this one because we're focused on dashboards. But, stay tuned. Stick around. I'm sure I will do one in the future because there's plenty of features that we like and care about that have a lot of applications and use cases, and we'll dig into that in the future. Sure. And, actually, let me let me do something real quickly. I wanna see if I can, grab for you guys another map. It looks identical to this, but it's actually got a little bit different. So let me bring this up real quickly. This is something that I did that's really weird and wild. Okay? And it's kind of in a test feature right now, but being able to put the SLAs over here on the right. Okay? And people have... You know, I always try to, I'm working with the other solutions at Jira's. We try to, like, challenge each other. I'm... I was like, have you... Can you figure out how we did that? You know? And so just to give you an idea of how this is done, I was actually able to create an alert that would look at the group availability. And based on that group availability, it would alert... It would run the alert every five minutes, and then the alert action was to add the group, availability into the custom property. And so when you... The action is actually to fill in a custom property, and then that fills in over here as a custom property. So you're actually able to make a list as well. Custom properties are a great way of being able to add information into, your your your maps if you want to. So, just something else that we're working on, but we're always trying to push those boundaries of what can we do inside of SolarWinds. You know, I always joke around, Crystal. I always say the greatest advantage that SolarWinds has over other, observability offerings is how customizable it is. And I also say the biggest disadvantage that SolarWinds has. Is how customer? customizable it is because. you can do so much with it. Yeah. I agree with you completely. It can be a bit overwhelming, but I think that, the key the key thing to remember as you're building these things... And I spent quite a lot of years building dashboards and all... And reports and alerts for all kinds of different environments. And I think the the important thing for me to always remember is that be... You can probably figure out how to do whatever it is that you're wanting to do if you just take a few minutes to to learn how to do it. And as Scott was pointing out earlier, like, these dashboards are using features that are built in that are not custom code. Like, we're not talking about building these things with custom scripts or or SWQL or SQL or anything like that. We're talking about using inbuilt features that, SolarWinds has provided that can really take your dashboards to the next level without having to go learn how to write SWQL or anything like that. So I think that's really important thing to remember is, like, if there's a question about can I do a thing, usually, you can figure it out at some point using custom SWQL or something like that, but I do think that it's important to note that there's a lot of things? There's very much power behind what you can do with the built in features. We had some questions specifically around doing root cause analysis using dashboards, and I know we've we've mostly been talking about building, dashboards today for sort of the the OT crowd or, you know, the... We're at a higher level where you can just see how things are performing and how that's affecting your SLAs. But do you have any advice on, building dashboards to help get to root cause analysis, like, at a glance? Like, I just feel like you really have to dig in a lot with those things, and I I don't know how you feel about that, but, I thought I'd approach you with that since there's been a couple of different questions asking about how much we can show on root cause analysis or or things contributing to the root cause? Well, well, it's kinda funny. It's an an... Up until a couple of months ago, that was always the white whale. That was always the unicorn. How do I get all this information and correlate all this information together? You know? And I myself have worked at it and struggled with it too. What data do I need to put on a dashboard that will let the engineer look at it and go, okay. I I need this piece of information. I need this piece of information. And one of the things I really liked about, NetPath is that it would do it for you. You know? You have NetPath saying, hey. I've got a problem here, and, oh, by the way, here... A config change happened at this time, and here's the line of code of the of the config that was changed. I I used to love that kind of thing, Yeah. And that was always the white whale. And then solar was... Put out this unbelievable, you know, new product called Avior. And Avior takes that piece and handles it for you. It looks at all of these different pieces for you and says, here is what I am seeing overall. And, you know, I I I will say this. As much as I love my dashboarding, I think the world is about to change, Yeah. because our dashboards are going to become less and less important with products such as Aviar, you know, being able to say, okay. I'm seeing a problem over here. I'm seeing a problem over here, and I'm seeing a problem down here. This is what they have in common. This is what the cause of the problem is. And by the way, I already created the line, the configuration change that you need to do to solve this problem, and all I need you to do. is click that approve button, and I'll take care of it for you. So we're entering a new world. If you haven't seen Aviar, I'm gonna suggest that you do because it does handle that. But what did I do in a... Before Avia? You know, Avia is only a couple months old. So in that case, it's... I I would personally be using a lot of the, of the PerfVax. You know? I would create those PerfVax based on the technologies such as a patient admission portal. Because once you know what business service is broken... Remember, a lot of times, that's... It's gonna be reported to us in such a way of customers can't log in. Users can't do x y z. So if I start from their perspective and dig in and I'm able to see all of the different components and then place those different components on a single screen like we do for for, performance analysis and being able to see my networking information, my application level is... Information here. Now I'm able to see this data in such a way, and this is really gonna help you with your root cause analysis because you're also gonna be able to say, okay. These are all the pieces that make up this particular, business technology. But more importantly, it goes far enough back that you're gonna be able to say, wait a minute. This is behaving differently than it has been the last six, eight hours, you know, last two days, whatever it is, to point it out. So PerfStack has always been my go to, and. I usually have these set up and ready to go per technology, both from an IT perspective but also from a user perspective. That's a really good point. I think, two things I wanna add there. One is which, Avior is going to be the topic for the next deep dive live demo that we're doing. So stay tuned for December because we'll be doing that. But if you missed the announcement and the, information that we gave, that was at the last solar Wednesday, so you can also view that on demand. We're not gonna get into it too much here. But, what I think is really great about PerfStack as well is if you've prebuilt these PerfStack out, you can also attach them to your alert. So if you have an alert set up, like, oh, my my monitoring system is down on this one thing, send them the link to the first stack, and that also removes that barrier of, like, them having to go look up and find the root cause. Now you've given them a dashboard where they can go look at the historical time frame. They can change the time frame on that as well to see, like, oh, this alert's been triggering for twenty minutes. I need to go look at the last twenty, thirty minutes and see what's been going on, in that application stack. And you can have that stuff prebuilt and then right into their hands when something is happening. Absolutely. Yeah. Being able to do to to provide that information is absolutely key because you've gotta be able to share this with your teams, especially when teams are are pointing fingers. It's a great way of... You know, I I always call this screen the not me screen. And what I would do... It's kind of funny. This is a great story. When we had a a problem, I would actually have each one of my teams add their data to the perp stack. So the the the systems team would add their servers, and then the network team would add their network stuff. And then the virtualization team would add their stuff. And all of a sudden, you'd see the spike and you'd be like, oh, wait a minute. You guys are the ones with the problem. Yeah. A 100%. Scott, could you walk through a little bit about how to build a first act since we're here? Sure. So if we go to a a a new a new perf stack, what's great about this is you're gonna go ahead and you're gonna start off by adding your entities. Right? You're gonna pick up what entities you you need. And remember, this is a lab, so it's a little bit dirty right now. It's not the most clean. But I can go come in here, and I can select what servers I wanna add. And I can do it based on up, down. I can do it, you know, based on their instance type, and you have a whole bunch of things that you can sort on, including custom properties and that kind of information as well. And once you select your entities, then it's just a question of coming in and and telling it, okay. I wanna utilize this node. And here I'm looking at the basic information like CPU and memory. I'm looking at response time, any configuration changes, obviously, depending on what modules you you may own or that you may have... Be utilizing. That's the kind of data you're gonna see. So you you may not see config changes if you're not utilizing those modules or you don't own those modules. But I can come up here to CPU and memory, and I can grab the average CPU memory and bring that over here. Now what's really nice is I may have an... Another system that does the same thing. In this case, it's different. But if it was the same thing, I can just grab it and put it on the same chart. So I'm not using a lot of real estate here, but I'm easily able to show these are all the CPUs. If I'm doing, like, a web server form, I can show all eight of my web servers CPU... Average CPU load on one single chart. I can then move on to memory. I can then move on to, number of connections, you know, on each one. And so I'm getting that information here. You can also hit these buttons that are over here and say, show me the hierarchy. Show me anything that's related here. And so it's gonna go out. It's gonna grab your volumes, your physical, volumes that are related to that device. And now when I click on that, I can look at my volume utilization, information, disk size, maximum disk used, statuses and events. And so you can grab the related, entities that are related to that device, whether it is, an application, an interface, and all of those different pieces and be able to tie them all together. So you're also able to go in and actually click on it and bring up the volume details if you wanna look at something a little bit more specific before you add it into that and so... To really give you information on what's going on. And that's how you're gonna build this up. Once you have it built out, you can come over here, and you can save what you have. You can export it. You can share it with other people as well. The other thing that I absolutely love is when you are having a problem, you can click on this start real time polling button. This is actually gonna go out, and it's going to real time pull that device and get information on what is actually happening with your average CPU on those devices. And you're gonna see it moving up and down in real time Because a lot... I use this a lot of times when somebody tells me, hey. I fixed a problem. You know? Oh, oh, I I I killed the service that was causing the CPU spike. I'll come into the perf stack, and I'll hit the real time pulling. I wanna see. Did it go down? I don't have to log on to that individual system or that group of systems. I can see it right here and see exactly what's going on. And so by doing this, it's gonna it's gonna tell me right away. Is this problem solved? Yeah. I I see the the drop. Or if maybe it only temporarily solved the problem. So if you watch it for five, ten minutes, and all of a sudden you're like, wait a minute. It's ramping back up. Remember that out of the box, SolarWinds is only gonna monitor CPU every ten minutes. So you might have to wait ten minutes to see what your next, CPU, looks like. Right? And then you gotta wait another ten minutes to say, did it actually stay that way? This is great because it's gonna show you what's happening in real time. So it's another feature that a lot of people don't know about, but it's super useful. And keep in mind that this is working for everything for CPU and for memory and for even a simple ping, but it's giving you that real time information. Other questions that you may have about PerfStack? Because this is a a phenomenal way of being able to build out dashboards and combine them in such a way that you're gonna be able to see data that, you know, for most people might be unrelated, but for you, is important because you're in charge of this part of the infrastructure or this particular business service. So you don't have to worry. about seeing, like, one node at a time. A specific question that came in about per spec per stack, was, in a VMware environment, what's best? Look at per spec for VMs or for the nodes. And I'm gonna say before Scott answers, I'm gonna say it depends on what you need to know, because we are... You pull different information from the node level. As if you manage a VM as a node, you pull different information than what you're getting out of your VM metrics. So it depends is the is the short answer. But, Scott, feel free to jump in. No. And you're you're a 100% right. You know? And it also depends on who's looking at the data. If you're the VM team, you're gonna wanna concentrate more on the VM, probably. If if you're the application level team, you know, you're concentrating more on the application side. But I... And I actually like putting both on there. Yeah. You know, I wanna see the node level information, the applications, and all that... Those pieces, but I also wanna see what is a VM reporting as well, and is that lining up with what I'm seeing on mine? So you're a 100% right. It does depend on on, you know, who's consuming the information and what is it that they need to see. But you know what? You can always put both pieces on there. And then one of the things that I do a lot with the dashboard is I have version one of my dashboards where I put a bunch of things on there. And then as I start using it, I realize what am I using and what am I not using? What am I ignoring? You know? If... I always tell people, if you're going to a dashboard, your dashboard's a a big, long dashboard and you never scroll, are you never scrolling because you don't need the data, because you forget about the data? Should that data be on another tab perhaps? Or... You know? But, you know, how are you utilizing the data? So if you put both of them on there, you're gonna quickly realize which is more valuable to you Yeah. and, you know, get that feedback and then modify it as needed. Yeah. I think that's an important context there. Like, what... What's in there? I think one of the things that I'd like to talk about with regards to that is that while it might be tempting to think you're gonna build the best dashboard ever and then you're gonna be done and you can wash your hands of it, the likelihood is that if that's not true. I mean, let's let's just think about the nature of IT in general. Right? We're constantly replacing equipment or you're changing applications where the software or firmware needs upgrades. There's, like, all kinds of things that change things. I mean, like, we've seen it even in SNMP over the years. Right? Like, they changed the the OIDs that we need information from just because we moved to a different firmware version. And so, like, there's all kinds of things that are constantly gonna be changing. So, like, thinking that you're gonna build something, and then you're gonna be completely done when you finish building it is, a dream, but I don't think that it's it's really possible. But what we really should try to get to is a state of, can you answer all the questions that you need to answer with that dashboard? And then later, if something changes, can you make those adaptations to that dashboard to ensure that all the right information is in the right hands at the right time? That's what... That's how I like to think about dashboards. I don't like to think about them as being done. And, like, I like how you mentioned that you build a version one, and then you think about, like, what can I do better? What what am I not using? What information is not serving this purpose of of this dashboard? And that... That's something that we should continue to think about as we go along. Like, you can put all the things on there, but, like, what if SolarWinds comes out with a new feature that's even better that shows something great on the dashboard? Like, there's all kinds of things and, Yep. you know, maybe maybe you expand your monitoring suite and you add net flow that you didn't have before, and you need to add that to your dashboard. You just need to continue to take considerations into what's best serving, to answer whatever the question is that that dashboard is trying to answer. Right? Like, how healthy is my application stat? Make sure it's serving that need as best as possible. And speaking of NetFlow, someone asked in the chat if you can show a dashboard with, info from NetFlow. And I don't know if you have any prebuilt, but the answer is yes. You can. I don't I don't have any prebuilt, but, yes, you can. Yes. You can. Anything that is, that you're monitoring in SolarWinds observability self hosted or in your module stat, you can add to a dashboard. So, for sure, anything that you need information on. If you're monitoring it, you can put it on a dashboard. Absolutely. We have some more specific questions. There's one about a specific dashboard and whether or not we can recreate it. We don't have time to really dig into that, so I'm not gonna answer that here. I will go check out your blog post, and we will see about what we can do with that later. There's, oh, there's a question about your SLAs. How are you measuring SLAs? Time, count of x x x for lab ordering. Like, how are you measuring the SLAs that you're putting on your dashboards? So that is actually a very long conversation. You could literally do a whole webinar on SLAs and and how different people handle SLAs, from one company to a... To the... To another is very important. One thing that I will share with you about SLAs is you should have them no matter what. Even if your SLA is 99%, you know, you should have them. You should set expectations for what, you know, your expectations are, and you need to meet those. And the reason that I say that is, a, it's it's useful in your performance review. It's also useful for you to know where you stand with what your bosses or your company's expectations are. Okay? So you... So don't be scared of them. Even if it's an internalized way and it'd be... Nobody else knows about it. Set something up to do that. When I set up my SLA, it is usually based on the number of data points being collected, every single data point. Now that said, you have to be able and willing to do what I call data cuts too. So if I am on the server team, right, and my server goes down, And... But it turns out it wasn't a server issue. It was a virtualization issue from the virtualization team. That should not go against my SLA. That should not count against me. That should count against the virtualization SLA. Now why is that important? Well, I don't want that to count against mine, so I have to go in there and cut that data in a reporting, style that says, hey. Give me my SLA report like I usually do, but remove out this data right here and don't count that in my report. And that can be a little bit difficult to do. Some people have to play with it sometimes in Excel. I don't. I actually do it using SqQL code, and and and doing, using the reporting feature. But you wanna make sure that if you got hit by an outage that was not your fault because it wasn't one of your systems, it doesn't go against your SLA. And so there's a lot of rules about SLAs, how to handle them. As I said, it's a huge topic. But most important, you wanna find out, you know, what are the expectations of your company, and how do they wanna handle it? You know? How does it make it business impactful? And when you start getting into customer SLAs, again, you wanna make sure you have your your customer facing SLA, and you wanna make sure you have your internal SLA. Because when you hit that internal SLA, you know, alarms should be going off saying, wait a minute. We've gotta really watch this system because we have a customer SLA now that is in danger. So, building those out, it's a it's a great thing to start researching, and and learning more about. You're gonna also start learning more about SLOs as well, which are service level objectives, and those kinda tie in tie in with it as well. I think that was really great. Actually, my biggest takeaway from that is that if you don't have SLAs, you should set your own because even if you, if you don't set their expectations, then they'll still have expectations. So if you don't set expectations with someone else, they'll have expectations of you, and you may not be coming up to the mark of whatever their expectations are because those communications are not happening. So, like, my biggest takeaway from that is, like, even if you're in a small shop or you don't answer to anyone in whatever application world you live in or whatever, if you somehow live in some world like that, someone has expectations of you. And if you're not coming up to the mark, if you're not responding in time, if you're not resolving things in time, they may be having conversations about you, about your application, about your team. And when those directives come down to say, oh we're gonna migrate to this other application and you're not talked to about it ahead of time, maybe that's why is you weren't meeting expectations that you didn't know you had, they had for you, they had for your application. So like I do agree with Scott. That's SLAs are important even for yourself, even if you're just measuring. How are you gonna know things are improving or that your... Things are behaving as expected if you have no SLAs? You know, and and something else I wanna point out, especially if you have a nontechnical boss, setting those SLAs is really good because it lets them know what their expectations should be. Right? But more importantly than that, I always put the SLAs in terms of time. So you heard me say that gave me four minutes and thirty seven seconds of downtime a month. That makes a lot more sense than somebody saying out of, you know, through, a 100,000 polls, I can miss x amount of polls. If you put it in terms of time, they go... Because that makes everybody go, wow. Four minutes and thirty seven seconds to diagnose, determine, and resolve a problem. That's not a lot of time. Right? So it really lets them know, and it it tells them how important their IT team is to meeting that SLA. Mhmm. So even if it's at 99%, that SLA is gonna be a lot, in terms of time. It's gonna be a lot less time than they're gonna expect you're gonna need to solve it. So it really... It it adds that level of expectation as far as what they can expect from you. Okay, Scott. We're about... We've got about four minutes left, so I wanna wrap up here. And I'll I'll just ask you to share any additional tips and tricks that you would like to share with people for, building their dashboards out. Yeah. You know, o... Overall, there's... Look, when you build out dashboards, it can... It's an endless... You know, as Crystal mentioned, you know, your infrastructure environment is fluid. That means your monitoring and observability environment is going to have to be fluid as well. And so you're always gonna be be wanting to do this. But the number one rule that I want you to take home with you is this is all about communication, okay, and making sure that whoever is consuming the information that you've spoken to them, what do they wanna see, what don't they wanna see, what information would be helpful. And people will say to me all the time, and this is something that I've experienced so many times, I can't get them to answer me. When I I ask them that all the time, and they won't answer me. And my suggestion for dealing with that is give them what you think, put it in front of them, and it's really cool. They'll be really quick to let you know what they like and what they don't like. People are really good at critiquing other people's stuff. They're not as good at telling you what they want. But, you know, you go buy a car, you couldn't tell everybody what features you want in the car, but you'll sit in the car, and you'll tell them you'll tell them what you like and what you don't like really quickly. So just put something in front of them and let them tell you, hey. I love this, or can I have more of this? And we're... Can I have this as well, or I hate this? You know? Let... Just put it in front of them and let them lead the conversation. Because once you put it in front of them, the ideas are gonna start flowing and they're they're gonna be asking you for things and you'll have a much better idea. But talk to who's ever consuming that information. You know? Who's looking at that dashboard and go from there? And you can build out some really cool stuff. I love that, Scott. It's the exact same conversation I've had with people over the years about, like, how do I add... How do I get other departments to add to my monitoring or, you know, how how do we know what they wanna monitor? They just say they wanted to monitor everything. Well, give them what you... Give them a place to start because people, will be over... It's... You know, it's how I feel about Minecraft. This might sound kind of silly, but how I feel about Minecraft is the kind of the same, like, philosophy I think that other people have of, like, there's just an... Oh, there's no directives. There's no anything to look at. It's just a completely open... No quests, no things to look like you just go do, and, like, that can be completely... For me, that's completely overwhelming. And so, like, I think that, I think that a lot of other people, when they're trying to consume information, if you just ask them what they want to see, unless they're trying to troubleshoot a very specific issue right now, they're not gonna have a good answer for you because what they're they're just like, I want everything. I wanna know all the things about this and... But do they really... Like, are they gonna be able to get anything out of that? Are they gonna be able to take action on the information that you're giving them? Is it gonna be enough? Is it gonna be too much? Is it gonna be noise? Like, they don't really have an answer to any of that stuff until they know what you can do. And, I've had many of those moment. conversations over the years to get people to start monitoring their stack in there. And they're like, I've been trying to convince them and they don't wanna listen. I'm like, well, give them something to look at. They need they need something to look at to give them an idea of what you can do. And sometimes when they see it, they go, oh, I didn't know you could do that. And then that leads into a whole conversation. It triggers their... Your... Their curiosity to think about other things that they they could be doing. It triggers their imagination. Very much so. Yep. So this has been wonderful. We're about to be cut off when the show ends here. So, thank you so much, Scott, for joining and sharing your dashboards and your wisdom with us. Happy to help. This is great. you everyone for joining. If you missed any part of it, it will be available on demand later, and stay tuned for December where we're gonna deep dive into Avior. Take care, everyone. Hi, guys.