Video: Fix Your Data, Empower Your Decisions: Data Quality Series for Product Leaders | Duration: 3548s | Summary: Fix Your Data, Empower Your Decisions: Data Quality Series for Product Leaders

Transcript for "Fix Your Data, Empower Your Decisions: Data Quality Series for Product Leaders": Good morning, everyone. We are waiting for just 1 more person, our lovely Steph, to be joining us on stage. Hi. Perfect. Perfect. Perfect. Well, good morning, everyone. My name is Kalina Bryant. I head, field marketing here at Mixpanel. We are super happy to have you guys join us today. Today, it's gonna be all about how to fix your data, how to empower your decisions, and really focus in on our data quality, for our product leaders. And with that being said, we have Rich from our team who's gonna be going deeper into how we can do that with our customers, some of the success that we have seen. And we also have the CEO from Avvo, Stephanie, who's gonna go into really deep dive on governance and how that impacts your day to day. Now housekeeping rules for us today. As we dive into the webinar, you're gonna get a full deep dive on all of the information today. It's gonna come with lots of questions, lots of, different things to navigate. Feel free to look on the right hand side. You'll see a chat box over there. Feel free to start chatting in there. I will be managing that behind the scenes. But, also, towards the end, we're gonna carve out fifteen minutes for you to do a live q and a where you can go a little bit deeper, you can ask more questions. We have the experts here, so feel free to fully engage. And with that being said, I am going to pass the baton over to Rich and Stephanie and we're gonna dive right in. Nice. Hi, everyone. Good to be here. Thank you, Kalina. This is really exciting. This brings me to, Serene. Sure. What is here? And that's here. And everyone sees my screen. Yeah? Perfect. Really excited to be here with you all today. This is something that we have been quite excited working on, and we went pretty deep on this. We got input from, a lot of, people that have been fixing data quality for a living for over a decade, including Rich. I will never forget the first time I saw Rich present. I had a mixed panel event, his advice for data teams and product teams on how to manage data to actually leverage it to empower better decisions. So I'm really happy to be here with you, Rich. So but with that, I'm gonna quickly cover what we are going to mention today, and then we'll do some intros, as well of ourselves and and sort of our mixed panel before we dive into all of this. So, we are going to, first and foremost, talk about what do data quality issues cost? What is the business impact of those? And the reason why we're going to spend some time on that, not a long time, but we want to give it breathing room, is because we've seen that a lot of people have a hard time to pinpoint exactly what this costs for their business. And we want to empower you to have that conversation internally. Then we wanna talk about real life learnings. So we we wanna share some customer stories, that you can then read up more on, asynchronously. Then we want to take you to your goal state. What does good really look like? What are you striving towards is what we want to sort of highlight. And then finally, we want to give you a plan of action to better data. All of this in the short meager time of, forty minutes or so so that we can also get some questions from you. But I wanna, sort of repeat what Kalina also shared. We want this to be interactive. We want to hear from you, so please post in the chat where you're calling in from, what your role is, in the in the company so that we know who we're speaking to. We really appreciate that. Please ask questions. We will try to cover them in the q and a. Some of them may be asynchronously. And if we aren't aren't able to cover them today, then we will try to follow-up with them, after this session. So please, see this as your time and ask us what you wanna learn more about. With that, I will also hold this here on the screen so that you can go directly to the blog post that, accompanies this webinar that we already released, the business impact of data quality issues, which touches on all of the points, that we're also going to touch on today, at a at a at a little higher level, what they look like, and how to think about the fixes. And so please go ahead and sort of say this in your bookmarks, and, tell us what you think. Alright. Over to you, Rich, for tell telling us, how incredible, both Mixpanel and yourself are. Let's let's say that. Yeah. Hey, all. I'm Rich. I've been in Mixpanel for just over seven years. Started doing implementations, which we'll talk about a little bit later, but now I lead our North America customer success team. If you're unfamiliar with Mixpanel, we're an event analytics platform that, we strive to make sure analytics is for everybody. Making sure it's easy for folks to get insights, and understand their business in a way they weren't able to before. Steph, if you don't mind, let me go to the next slide. Yeah. This is that. I've held every not every, but most GTM roles in, mixed panel. So, leading SEs, leading AMs, implementation solutions, etcetera. And I've worked with a bunch of our customers and that's what my team does now. So my team's goal is to make sure that folks can get the insights and be successful with Mixpanel, to drive all the outcomes that you see here. So understanding how you acquire users, how they convert through your important funnels, how they retain, and making sure that folks can do that in a self-service way with data that they trust. Happy to talk offline about all the ways that we integrate, how we can make this easier for you. Feel free to drop me a message on LinkedIn, or throw some stuff in the chat that I can answer async. Thank you. Introduce itself. Yeah. Thank you. Exactly. Yeah. So I'll just start by saying, talking about sort of a little bit why our work is We exist to inspire better data cultures so teams can build better user experiences, which is a mission that is very aligned with mixed panels. So to help people make better decisions, ultimately. And, I say that because, while Abo exists or sort of our platform today helps people guarantee data quality upstream, meaning and we'll we'll go into detail about, like, what that means really and what we're trying to solve in this webinar. We ultimately care a lot about, how our customers and how our community actually is able to leverage their data. And that comes from myself, being having a thirteen, fourteen, fifteen something year background in data, all the way ranging from data engineering roles to the first analyst, to a product manager, to a revenue operations manager, to all of those different sort of stakeholder roles in data and ultimately, learning a lot about how important event based data is to drive business decisions, whether it's how to go to market, what product to build next, or, how to build recommendation engines, whatever you need. And also learning the very, very hard way how difficult it is to actually manage data quality at scale, for these fast moving event based analytics. And so that's a a little bit of the the backstory of of of Avo and myself. And we'll echo what what, Rich said. Please, if there's anything today that, that sort of sparks a question, please reach out via LinkedIn or the community Slack, which we have, I think, links to somewhere in this presentation. If you search for Avvo community or or mixed channel community, you will probably also find us there. But with that, I am going to dive into, I wanna mention this also. This is a personal passion of mine. I I like to say that Avvo both saves businesses and sanity. 2 of my favorite ways to frame that, 1 from a product leader who said, I would quit my job and switch careers if I could no longer use Avvo. And another 1 from, someone that's highlighting how important data is for revenue generation. So the quality of our data structures, things to have over the prerequisite for us being able to launch our generative AI solution months earlier and make millions of dollar more in revenue. And this is what we want to empower you to do today, Rich and I. So, that's our mission today. So let's take it to business impact. Why do data quality issues cost so much? Why are they so costly? Okay. I wanna paint up a few familiar, painful scenarios. Okay. The rich, prepare to be triggered, and everyone prepare to be triggered. That moment when someone tells you that something was really successful, in the launch, and it really made everything, all of the impact. And they saw a huge conversion conversion update. And the first thing you think is like, oh, no. What data did they use? I feel like that's like a scenario that many of us have been in if we are the data expert, and it's a terrible situation to be in. It's like a yeah. You you don't want to have that feeling. You don't you you want to have the feeling of trust that your business is able to rely on the data that you're helping them generate. The 1 the moment when the CEO says, let's do that AI thing everyone is doing, But you know it's not gonna work out because the data that is going to drive your AI thing is too messy. And yeah. The did you wanna add anything? I heard you said Yeah. We're in the process of doing this now, and our underlying data is needs some work. Yeah. Exactly. We're getting there. The this moment where a lot of people are sort of trying to use Mixpanel or or, look up some insights, and they don't really know the difference between sign up completed, signed up, and user sign up. And it causes them to either miss 1 of them or just sort of give up a little bit and not leverage it. And if they miss them, and then they don't actually end up with a holistic picture of the data, which is terrible. This moment, when you hear multiple teams talking about a conversion rate from, let's say, downloading the app to, creating an account or just the the lead conversion rate, and someone talks about about 40%, someone talks about 5%, and you don't really know what number is right. And that's because you don't have a documentation for what the metric definition literally is and what the data behind it is, and it's really painful. And then the sort of being stuck in reactive damage control is what I like to call it. So that's this is on us, the data professionals that may have been deep in SQL. And we've all of a sudden built this spaghetti spaghetti code SQL query, or tried to merge a bunch of mixed tunnel events in the lexicon UI to try to do some cosmetic fixes to make sure people are sort of using the same thing. But, ultimately, in the end, now we don't know, all of the sort of paths that the data has had to travel to reach its state, and we don't really know what the query means anymore. So these are all, I think, scenarios that probably many people on the call will, relate to. From there, I wanna go and to talk about and just highlight what is the actual business impact, of these issues. So we've talked about this. If you don't have high quality data, there is going to be no Gen AI for you. This reminds me a little bit of Seinfeld. No soup for you, my friend. And so but what this means effectively, there is, for example, a quote from an open AI engineer that talked about that. No matter what, how great your model is and and what you build in your model, the only differentiator, generative AI or AI products are going to have eventually is the data and the quality of the data that they put into it. So if you don't have reliable and good data, you are going to have worse products than whoever else is going to have reliable data. Lost revenue. And obviously, this is this is a major opportunity cost. This is when companies are are making choosing the wrong AB testing group from a result, for example, causing the last tens of millions of dollars in revenue over a few days because, they had the wrong result from an experimentation. This is also, because the event based data that we're sending into, push notification systems or something like that, are not converting people correctly because they're sending the the completely wrong message to the completely wrong people. So a lot of reasons why event based data actually costs lost revenue. Of course, this is, I think, maybe the most painful personal sort of the personal experience, because the other 2 are sort of business experiences. The waste of time and resources that we feel on our skins, and we see the people around us waste so much time and resources around the processes of getting to good data, or trying to leverage the data without really being effective in it. So that's what makes us pain feel the pain every day, really. And then this, when we sort of realize we've made a bad decision from bad data, that's so painful. I've been in that position, and so many people have been in that position. And when you have to report that back to the CEO or to the board of directors that you actually made the wrong decision based on wrong data, that undermines the entire ability to run a a database company or a data driven company, because from that moment, nobody's gonna trust anything that you say. And it's really painful, very terrible. And then, incredibly, we've just been seeing people churning from the data teams, or product teams because, they're not able to like, they they spend most of their time on fixing data or or hacking together some data stuff when they should be just making using their time on sort of providing proactive insights for the business. Do you wanna shed any light on any of these, Rich, before we move on? Just to kinda highlight, it's not uncommon for, some customers who do have bad data to not only make bad decisions but bad investments. They think there is a problem or an opportunity in the area, and because of the bad data, they misallocate, their bets for the year, which kind of points to your your points around lost revenue, waste of time and resources, and make the wrong opportunity cost trade off. Yeah. Exactly. It's all pretty interconnected, but yeah. Thank you. Exactly. And so but okay. Let's go from talking about, what this what these symptoms are. You know, these are symptoms, really, and those, like, shout outs that we were calling out in the familiar scenarios, those are sort of, like, what this looks like? What are what are we looking at, really? But let's switch gears a little bit to talk about the root causes, for the data quality issues. And, well, 1 of the things that, the conversations we've been having the last few months as we've been shaping this, both, you know, us too, Rich and I, as well as when we've been talking to, a bunch of really smart people that have been working with us to shape these guides, we can think about categorizing these root causes into 2, I would say. It's, organizational buy in, and it's processes, tools and processes. And the first three on this list that we're about to review are sort of the organizational buy in, and then the second one are tools and processes. And before I highlight them, I wanna say the real root, and this is to your your previous point, which you've talked a lot about this in in your your sessions, that you run for individual customers and that you also run at sort of like customer forums for Mixpanel. The real route is the organizational buy in, because if you don't have that, then you won't have the the drive to maintain the tools and processes. And, to 1 of our consultants, points that that that sort of, helped us prepare this, they often come in and they address the last 3 that we're about to display, like the the tools and processes. But then they say to whoever hired them, you are going to have to fix the route the top the cultural roots here at the top because otherwise, you'll you'll have to pay us again in one year to do the exact same work again. So okay. Let's go through this with that in mind. Okay. Sorry, Rich. Go ahead. Yeah. All good. Yeah, this is the first point you're about to talk about. But, at the end of the day, if you can fix every piece of data issue that you have today, if leadership isn't bought in on maintaining that and that data is a first class citizen at the organization, then things are gonna backslide. And for long term change, people have to be bought in and accountable that data matters at the org, and is something that everybody plays a role in. Nobody's going to own data from start to finish, but everybody has a role to play in how it's designed, how it's implemented, and how it stays clean. Yeah. Exactly. And we'll talk a little bit later just, to sneak preview that. We'll talk a little bit about later today about, what these root causes, when we've turned them into what good looks like, then how do they map onto that to to spoil that a little bit? This is gonna be the theme of what we're gonna be talking about today. So does data matter in the company? That's, like, a a really big, really, really important thing to to think about. Does anyone feel accountable for data quality? That's another 1. And that is very related to 1, to, like, to to to this point, to point number 1, because, it pertains a little bit to, like, how do you then who who does data matter to is, is also a really key question here. So okay. Do we think high level that it matters to the company? The CEO wants to be data driven, but is the rest of the company also empowered to be data driven in the sense that it also matters down to, like, the product is data driven. Like, all of the different aspects are data driven so that each part or domain in the organization is empowered to own their own data and thus own their own data quality. And that's something that we'll we'll talk a little bit about more today. Okay. Do we treat data as a cross functional team sport, or is it a chore that engineers do for an analyst somewhere in the business that they don't know? Do we treat it as a siloed thing, where people don't have insights into what the other teams are doing or what the other experts are doing? That is a root cause. That is a sort of, let's call it a red flag even. Do we have standards to follow when we're tracking new features? This means both, standards for the data structures that we have, but also for the processes. For every single time that we want to update our analytics release or update our analytics implementation, do we know what we're gonna do? Is it going to be a Jira ticket that links to a spreadsheet that has a standardized structure that, you know, then goes into they had to the developer? Just Whatever it is, it has to be standardized that everyone is doing the same thing. Are the events documented? Are the event schemas documented? This is this is this is a surprisingly, this is a surprisingly important aspect because it ties everything else together. You're it's more difficult to do cross functional team sports, where everyone is sort of referencing to something if you don't have any documentation to what it should look like. And then documentation is also key to having proper tools to implement and validate before you actually release. So this is what do the engineers, how do we support engineers in having some sort of a contract in place that they can, fulfill when they're releasing the updated analytics? Okay. Do you wanna add anything here, Rich, before we just move on to to see customer stories real quick? No. We can move on. I'll cover more right after this. Yeah. Okay. So we wanna talk a lot a little about, a a few examples, of of companies that are leveraging both Avvo and Mixpanel and the impact that that's had. Rich, I'll just hand it over to, how this happened with ADP. Yeah. So as I mentioned before, I was a leader at Mixpanel. I was an implementation manager and spent a lot of time working with customers to implement, Mixpanel and establish good data habits. ATB was 1 of my customers. And, luckily, they kinda knew early on that data quality and data governance was going to be important, and they didn't wanna get stuck in the the thrash and wash cycle of implementing data, having to go and fix it, playing whack a mole. They wanted good habits from the start. So early on, I recommended Albo to the ATB team, who's run with them. They've been our customer for years at this point and have had great outcomes. Yeah. And it's been a pleasure for us to work with them for sure. It was fantastic. And and and, there is a a a coming case study that sort of talks a little bit about how they migrated their processes, and from how they went from, let's just say, unreliable data, which is really painful, and manual processes to, and and they also had sort of, like, surprise data changes. All of a sudden, the product manager would see, oh, there's been a change, and I don't know what where it came from. Who who did that? What does it mean? To actually having a source of truth, reliable data, and eventually increase business outcomes. Right? Because you're working on the foundation because you're trying to drive good business outcomes. And that's what we always like to remind people. A couple of more examples, and you'll actually find these case studies online on the Avvo website. This is also well, this is from WALT, which is a DoorDash subsidiary. Yacopo, who we've been working with for a while as well. And, like, they have, for a really long time, sort of relied on Mixpanel for empowering distributed decision making. Right? So decentralized, not having to rely on the data team to help people make decisions. So the product managers or junior analysts or, like, whoever can just go and point and click and make decisions fast. That's what Mixpanel helps you do. It's a it's a it's a power tool that help you helps you decentralize how you get value from data in your business. A really important requirement for for for that is data literacy and data quality. And you don't have data literacy without data quality. And what really also empowers data literacy is when, when engineers, product managers, and analysts are involved in the process of defining the analytics events and the analytics updates, and then they are able to leverage it together. And so they were able to go from messy data, lack of data literacy, and sort of messy processes, to reliable insights and better decisions, which is really valuable and beautiful. And a fun detail also about their transition is they had already experimented with very rigorous documentation of the schemes. They, prior to Avo, had what, they had YAML files, on GitHub. So that's a technical sort of technical way to document event schemas, but it didn't scale because it wasn't helpful for business stakeholders. It wasn't helpful for sort of, planning. And they were able to really make a shift into the reliability and the data literacy, and sort of view Avel and Mixpanel as a sort of power combo. And then I'll quickly mention also 1 football, which I really love, working with as well, Alberto, our friend, where they have also relied on Mixpanel for a really long time. And they were able to go from messy processes to team alignment, where sort of data is a part of the release. And I see some people reacting to this funny quote here. Opened a channel between engineers and product people to finally talk without shouting at each other. So we can finally come up with a tracking plan that's really working. And I think that is sort of the that is the real life stories that we want you to feel. That's what we aspire for you. We know it's possible. And, we've seen so many companies do this successfully in really highly leveraged mixed panel. If they take a little bit of a breathing room to go from reactive damage control to proactive data management. Okay. So with that, let's talk about what good looks like. Rich, over to you. Yeah. So Hi, Andrew. Sorry. Andrew is an analyst at One Football. Good to see you. If you don't mind jumping to the next slide, jumping back to these root causes, I wanted to add a little bit of color, to what Steph talked about earlier. But a lot of this all stems from a kind of lack of accountability, a lack of process, a lack of just not having those conversations around, what does data mean for our organization. So if we jump to the next slide, good in this case, does data matter? Data is a product. Data is just as much a product of your org as the thing that you build and ship. So just the same way that you have, your platform, your app, whatever that is, the data that that platform, app, whatever it is, generates is just much a reflection of your team and the way that you think about data and how seriously you take that as it, as your product, I should say. And so if you're not bought in, if your leadership isn't bought in on good data, then the data that you have coming out of your product is also not going to be good. What does that mean for governance? For governance, that means there should be an owner of data or data quality, that doesn't necessarily have to be a single person, but it should mean that people are held accountable to their position in, kind of like the data life cycle. So as Steph was talking about treating data as like a cross functional team sport, data should just be part of your regular SDLC. So the same way that you create a story or a Jira ticket or whatever your process is for getting a new feature created, there should be corresponding Jira ticket stories, whatever they are, for your analytics implementation. Those those stories should follow standards. The standards that you sit down and hash out with your team to agree on what does our data look like? What counts as an event? Where is it coming from? Are there things that are should be on every single event as metadata? The user ID, the session ID, whatever that is. And without those conversations, you're gonna kind of end up with what Steph was talking about earlier, like a login login, for sign up, you know, sign up underscore login, etcetera. What that kind of looks like is that the product's not done until the analytics is implemented, and everybody plays the part of analysts are kind of telling the PMs and engineers what data they need to be successful in their role. The PMs or whoever's responsible for it are designing those events that are based on these agreed upon, standards. The engineers implement that data based on these agreed upon standards from the source of truth or wherever is acceptable at your organization, and the testers know exactly what data should look like when they're looking at it. So that by the time an event or any piece of data is at, the, at where it's going to be queried, it adheres to these standards. Then throughout that process, it should all be documented. After the engineer, whoever's going to implement this data, implements it, then it should also be added to your internal data dictionary so that somebody who's not familiar with, data or with not familiar with a particular event can go and reference that and say, where does it get triggered from? What does this mean? What is this actually measuring? So there's no question that they will draw any wrong conclusions when they query that event. And that all gets held together by the right processes and tools upstream. So that we're not relying on people being a % perfect to adhere to all the standards that we set out. But there's systems and tools in place to catch them when, mistakes do happen. At the end of the day, the, measure that I use and my team uses of whether an implementation is good or not and whether data is clean is if I know very little about your product, I can land in your analytics environment and really quickly orient myself, to your data. I shouldn't be guessing what does this mean, etcetera. Like, if you imagine you land on Amazon's or some other major retailer's website, you know, you'll see, like, a add to cart. You're probably pretty sure what that means. Even with, like, a base familiarity, if you can see the analytics, you should have very few questions about what am I actually querying if I use this event. Mhmm. I really love that framing. That's a great benchmark, and it sort of it goes back always to the data as a product. And This is just like for for those who are interested, we've written a bunch about this also because it pertains to the so called data mesh principles, which I won't scare people by talking about right now. But it's a framework to think about how to get, data quality at scale and data reliability at scale. And, we at Iowa have written a bunch about how to actually get the data mesh principles to work as, for event based data, specifically. 1 of the principles, 1 of there are 4. 1 of them is literally data as a product, and this readability of event names is exactly that. So when you are a data designer, which is a role that not a lot of people have as a hat, but it is a role that many people play, data designers. And when you're a data designer, what you're thinking about is when you're naming your event, you should be thinking about the person that opens up the Mixamo dashboard and tries to use it, use this event to sort of build a query and understand something. Yeah. This is great. And we'll go even deeper in these, both in a follow-up blog post, but also a little bit later today when we, set up some tactical tactical action items, really. So from here, let's go to the plan of action to better data. Okay. So we're going to run pretty quickly through some high level things, and it's going to be really exciting. But like I shared, we are going to release a longer guide on exactly this. So the blog post that we already showed you in the beginning gives a little bit of a high level picture, and then we are literally diving in very deep into your tactical guide for to actually fix your data. And if we go and zoom out a little bit, what the path to your plan of action, basically, to better data looks like. The themes that every single person that we talk to, and this is also our experience with working with like, I was a consultant before I started Avvo. I was a hands on data practitioner within a few companies before I was a consultant. And so I've I've seen a round of this. Rich has also seen a round of this, and we talked to a bunch of people that work, as consultants and just come into businesses and help them fix this. This was the red theme, like the the the theme of the steps that people, take always, when they are fixing their data. So it starts with an audit. It goes into preventing new issues, from surfacing, and you can start that work today. There's nothing that should stop you from starting today to prevent issues from, new issue from surfacing. And then going to sort of fixing potentially your historical data. I'm gonna dive deeper into each of this, but I wanna highlight this. So this is what how we map this out, and this is what what we're sharing also in a blog post, coming soon. The time to impact of the audit, some version of the audit, the 1 that allows you to sort of build your case internally for why you should spend more time on this. We're mapping out a process that could get you to a case, a built case, in ten minutes. Of course, there's there are layers to that, and, Richie will go into to this in a minute. Prevent, like I said, start today, and you can get impact with that in hours. We'll talk about what what this looks like, in a minute and then fix. You can get to impact on that in minutes, but there is short term impact in minutes, and then there are long term impact in in weeks, I would say. That's how I think about this. Rich, over to you for the audit. Are you ready? I'm thrilled. Every time before when I would pick up a new customer and now my team picks up a new customer, I asked the team to spend about ten or fifteen minutes in the customer's project looking at their data. Just to form an opinion on how good does this data look and where are there opportunities for improvement. And this doesn't have to be long, it doesn't have to be really in-depth, it doesn't have to be are we tracking everything we wanna track. It's more about does it look like they have the right processes in place to keep their data clean? So, things that we talked about earlier of, you know, are there 3 different types of login events or 3 different types of sign in events? Does the total number of users in that project kind of roughly map what we would expect it to look like, or are there more or less users that probably indicates that we're missing or over counting folks? Does it look like the same event is being tracked multiple different ways? Is there a standard naming convention, etcetera? So nothing super deep, but more just a quick diagnostic of does it look like there is robust data governance? And if there's not, then what are the kind of areas of opportunity for us to step in and help that customer implement those things to drive to better data in the long term? So this audit process isn't about do we have every event, are we measuring everything that we wanna measure, it is does our data look clean, where is our low hanging fruit, and where is our biggest impact to make sure that the data that we are gonna query is trustworthy. So the way I would stack rank that is if the number of users looks wrong, if we're missing or overcounting folks, that is kind of priority number 1 most times. If the data is, like, double counted, or there's multiple events of the same name, that kind of becomes priority number 2. And then things around naming conventions, you know, not being standardized wind up being number 3. The impact of multiple events, being implemented in the project, kind of to the example we've been using about 3 different types of sign ups or logins, means if I am new to the project and I don't have the tribal knowledge that there are 3 different sign up events or 3 different login events, then I'm going to go build a query and draw conclusions off of data that is incorrect. Right? And when we talk about building your case, that can be the kind of keystone in your case of, like, this data is messy. If I came in here, I wouldn't know what to look at. And if I tried to query it, I'm either gonna get frustrated and give up and not get the answer I need, or I'm gonna draw the wrong inclusion, and maybe make a bad business decision. Yes. Which is, it's just so hard to be feel accountable for that, which is, like, if you are a part of the central data team, you feel really accountable for it, but it's so out of your hands to impact it, which is terrible. And I guess that brings us maybe to yeah. Rich, are we ready for the next 1 really? Or Yeah. Yeah. Yeah. So in terms of prevention, like Steph talked about, this is something that can start today. If you do this audit and you look at, your data and realize, oh, it looks like we're double tracking or triple tracking things. Right? Then it's just getting everybody on the same page of really quick conversation of, like, how do we actually, implement new events and how do we make sure that the data that we're adding doesn't already exist. And the easiest way to do this is to just start small, on your pick 1 team, pick 1 flow, figure it out, implement those new, standards, workflows on that team, and spiral out from there. Yeah. Major plus 1 on that. Always start small. But how to think about what to do, though? Like, we always say this. We're always saying start small. That's always our recommendation. But, okay, if we get tactical, it always boils back down to the root causes that we just mapped out. And so, okay, first of all, does data matters matter to the business? Okay? And we touched on this a little bit in, like, what good looks like. What that looks like and what you want to strive for is you want to make data as part of the team rituals. I love this framing from this comes from you, Rich. Make it a part of the team rituals. How I've often recommended this in the past is number 1, you have a weekly metrics review. The reason why a weekly metrics review is valuable is because you if if you are 1 of the persons that are sort of try trying to drive business account or, like, data data as something that matters, then you can proactively come into a weekly metrics review with something interesting. And then you can ask people to ask questions in the weekly weekly, metrics review. And the more proactive insights you bring into it, the more people will ask questions, and then more people will get curious about how to how can they use data to make compatible decisions for themselves. This will also help people reveal how unreliable the data is as it stands, and that will drive them to want to get to better data quality, which is really magical. But what this also means is, like, well, what why is like, if the board is not if the board meeting isn't reporting on the product experience, why is that? Why why are we not seeing product, experience as part of the, the the core business outcome? 1 of my favorite examples of an Ava customer what included were, like, a financial team. The finance team used user retention as the predictor for their financial outcome for that year. That is how you properly leverage data to make, really good business outcomes. And So the finance team that typically lives somewhere in Power BI and when you're removed from Mixpanel, they were in Mixpanel as well. And so that's really huge. Okay. Data is part of rituals. Make sure you start to build types of rituals like that. Bring data in. Like, to, if you aren't already, if there isn't something there, then try to work on bringing that in, is basically our recommendation for that. Okay. The right person to facilitate change. Let's talk a little bit about that. Like we talked about earlier, this is often there's a person that sort of there's an individual that feels responsible for data and the quality in the in the company. There's an individual or team. This might be you because you own the Mixpanel account or something like that. It might be you because you own BigQuery. It might be you because, the CEO is on your back because you, have to make the right decisions about the product. There is some drive for you to, facilitate this change. And what you're striving towards is to help other people take ownership of their own data. Because if they take ownership of their data, this means anything from the product team really wanting to measure the success of every single product experience and relay their learning to leadership and all their product teams and the go to market functions so the company can iterate on who to promote which products to. But it can also be that salespeople feel accountable for documenting the origin of the leads in their pipeline, so that the marketing people and marketing team can do more of what works and less of what works, doesn't work in lead generation. Just driving this general accountability is what this sort of like, the person that facilitates change is supporting. But, ultimately, we're trying to create an environment where the domains themselves, the marketing domain, the sales domain, the product, different different prop teams, they own their own data. They own their their the accountability for their own data quality, but there is someone that's, like, sort of, hey. You know, this is what quality would look like. Do you do you feel accountable for this quality being there? And this, I guess, probably happens with a mix of carrot and stick, I would say. But ultimately, all of the like, a big part of the audit process, the build your case process, is make the problem very visible. And that's really what we're sort of recommending that helps you drive, the ability to prevent. Okay. Purpose meetings is a tool, for, like, the the cross functional team support thing. We have often recommended, purpose meetings. This is a concept that multiple companies have sort of, built really internally, but, we ended up calling them purpose meetings. It's a sit down where all of the different stakeholders, the product manager, the analyst, the engineers of all of the different platforms, they plan the analytics release together. This really drives change, and, we've written a bunch of blog posts on this. And if you're interested, reach out to us and we'll share it with you. This will really sort of drive awareness of what goes how the sausage is made. Let's put it that way. Make the change from, instead of the engineer seeing the, analytics implementation as a chore for the analyst, they start seeing it as 1 of their most fundamentally important tools to build better products. Really valuable, and I've seen this shift in sort of alignment and silo breaking, just with this tool again and again and again and again. And if we think about what that looks like a little bit, turning it into a really visibly cross functional team sport, this is what an example of a single team would look like. So this is what this process generally looks like. It's very hidden for most people. So there's maybe an onboarding team and, 1 of the product teams is is focuses on onboarding. They define a future goal. They should then define metrics. Then they should define like, the the metrics would be whatever you plan on visualizing in Mixpanel to help make the decision on whether this was a successful product release or not. Then you define the events that go into it, into the metrics. Like, what are the updates that you need to make to your data collection to have these dashboards available, to have whoever needs to sort of build them? Then you go into implementation. And if you have good tools, and you should, then implementation step has really good sort of validation. We'll cover that in a minute. Sift the feature and analyze and learn. And if you make this into a cross functional team sport, you'll see this here. You'll see a major leverage in the amount of people that are able to leverage Mixpanel on their day to day. So that's gonna drive adoption of Mixpanel as a tool, and that's gonna drive, adoption of sort of data as a decision making mechanism. And that is going to empower you and leverage you in your role so that you don't have to worry as much about sort of making sure everyone has the insights that they need, basically. Who touches this? These stats that differs a little bit on sort of like the the who is a part of the team? Sometimes product managers go all the way into defining events. Sometimes engineers go all the way up to defining metrics. Sometimes analysts cover much of this. Depends a little bit on the team structure and the prowess of each of the roles. Will you add anything to this, Rich? The only thing I'll mention is going back to your previous slide around or previous point around, the person to facilitate change, if you look at that map, not 1 person can own all of this and make sure everybody's doing their job. So being the change agent is much more about ensuring that ownership of the analytics pipeline in every step of the space is taken seriously, and the folks who do own each of those responsibilities are accountable to ensuring good data at the end. Mhmm. Great. Thank you. Yes. That's a great great call out. And I will also mention, Lynn, we've talked a bunch about this at Aravord. We'll be releasing a blog post on the Aravord blog and probably within the next month where we also talk about how does this process scale across the business when you have multiple team teams doing this also at the same time and how important data quality becomes when you just imagine this map growing, to multiple multiple teams doing this all the time because they're always updating their product releases, and the strain that you have if these people don't take accountability, for their for the data. That's a story for another time, but, make sure to reach out if you're interested in learning more about that. Okay. So, okay, so like we talked about, here, we're still talking about we're talking about a lot of, like, cultural shifts here. It's all really like, there are tools to to do these cultural shifts for sure. But here, we're also migrating over to the the literal tools and processes that sort of, having the cultural shift of the first three will enable you to have buy in for the tools and processes. But it's also like a, sort of like a bidirectional thing. So if you start working on the tools and processes that we're about to go into, that will also maybe sometimes, in many cases, drive sort of, people to become curious and you you get by. So you should be working on both the the cultural impact, but also the the schools. Okay. So let's talk about, standards. We talked both about standards for data structures, and we also talked about standards for processes. We talked a bunch about both the standards for processes and the standard standards for data structures. The this means data structure processes or data structure standards means you define very rigorously how your events should be named, how your property should be named, what flexibility the teams have in that, and we'll write more about that in the post. I see there are some questions about when this article will get out, so we'll get to that soon. I also love that we have so many questions that Kalina, is asking me to wrap up, and that's great. So we'll try to run quickly through the rest. So and then okay. We talked earlier about sort of are the event schemas documented? I wanna take that to the next level and say to you that spreadsheets are 1 thing, to document the schemas. But, to take it to the the the ultimate level is you have to think about it as you're you're actually source controlling them like you do your code. You need to be able to monitor how your schemas change over time. It is never just a current snapshot. It is also about how it's been changed and who contributes to that change. And so you need to be able to have, like, multiple drafts ideally going on, of your changes at the same time. So consider it just like you would code development. And then finally, proper tools and processes to implement. We have seen a lot of success when, engineers are equipped with direct translations from the definition of the events into what they should look like in code. And, we've seen a lot of success when companies build something like CodeGen. Ava also offers CodeGen. But, generally, like, translating the schemas to code is really valuable. And then having some way to validate, what the event actually looks like when you're triggering, the app and the code, instead of delaying it all the way until you have to wait for it to reach your your database or something like that. So have proper debugging tools to help engineers really confirm that they are implementing analytics events as according to the expectations and the plan. Otherwise, it's a way too too slow process, and this is especially a big problem for mobile where you would otherwise have to sort of, if you were not the engineer that yourself, then you actually have to, you know, obtain, an app build of the latest release show you that you're able to test it. It's just a really cumbersome process. So, really think about how you can empower your engineers to sort of work quicker and more efficiently. Yeah. So this was a sneak preview of what to what's to come. But if we quickly then also of in the prevent process. And then if we quickly go to fix, and I'm gonna run through this very, very quickly. There are 3 ways that you can go, and they have different amounts of leverage. The first one is learn to live with the issues. Put this in red because, okay, you can do this, but, it's it's it will leave you with all of your business problems still. Cosmetic fixes, you can sort of patch your data and then root fixes. Fix them at the source. Fix them at the the code where the events are originated. Quickly on pros and cons. If you learn if you go for this 1, then at least if you do the audit, then you're documenting the workarounds. You're documenting where the dragons lie in your data. The the the con, all of your business implications still, are present. And so you're not actually fixing the problem. You still have a lot of expensive, costly problems with data quality. Cosmetic fixes. I really like that framing from 1 of the consultants that we work with to prepare this. This is using something like lexicon bridge to merge events and rename them and things like that. The pro, mixed funnel users get good data. The the the con, it's just a single source fix. It will still require you to maintain really complicated SQL queries. It will still require you to have, like, a state machine of how you're mapping all of your events to, another event and how you're fixing all of your data. It's just it doesn't scale. Root fixes, fixing at the source, it is the only scalable way to fix your data. The the the con, you have to be proactive. But what we always say is, it is way more expensive to live in reactive damage control than to migrate over to proactive data management. You wanna add anything to that, Rich, before we No. I think that's that's alright, and this is the way that we think about it. You know, we will do cosmetic fixes on data, but we always tell folks the cosmetic fixes are just a Band Aid. Gotta fix stuff upstream. Mhmm. Yep. Upstream. Alright. Kalina, queuing you in. Yay. We did it, team. So we have about five minutes left, for q and a. I hope you all enjoyed that. It was super insightful. As Steph mentioned, there will be, more follow-up. You'll see in the chat. I also put the blog in, so if you wanna take a chance to read through that. But let's open it up for q and a. Feel free to put some items. I will be walking through them. When can we connect Mixpanel to inspector? I think that's a question for you, Rich. Yeah. Woo hoo. By Inspector, do you mean to invent Inspector or like an actual particular tool called Inspector? I know Oh, Inspector, Rich. Yeah. I'm hoping you mean that. Yeah. Let's assume. Yeah. What else is it? Yeah. Steph, Aavo does this today. Is that correct? Yes. Aavo has a product, which is a data quality of durability, product called Avvo Inspector. And you can turn it on as the destination from a CDP. So you're streaming your events directly into Avvo to build your current state of analytics and always see what's currently live and compare it with your schema. You can also set it up as a 1 time, SDK connection on your source. So it's just a 1 time setup to start streaming all of your event data into Avo so that we monitor the quality. And also the finally final final way is to to send us, your schemas via an API if you wanna do it from your own internally built CDP. So if you already use Segment, for example, then you can, go from 0 to a tracking plan with an Avvo in eighty seven seconds. We timed it, which is really fun. But I love that you're asking, that you want the destination from Mixpanel. That is actually something that we have talked to the Mixpanel product team about. And I think it's all about prioritization. So just, continue to being allowed to be allowed, Aaron, and, we'll get it done for you. Yeah. We're prioritizing data governance features on the roadmap this half. So hopefully I have an update for you soon. I like it. I'll also ping DJ. Yeah. I know there's also a question that came in in the Q and A about what are the best ways to get people in the organization excited or inspired to prioritize data quality. I think this is what Steph talked about a little bit of building it into the rituals, and making 1 team kind of like the shining beacon for all the other teams. So when 1 or when other folks in the org see 1 team using data really well, they tend to want to, mimic that. So what we do at our customers is we find 1 team who really cares about data, who has folks who were craving data, good quality data, and we work really closely with them. And then we use that to evangelize throughout the org. So doing things like the metric reviews that Seth was talking about, product retrospectives two weeks after a product ship, and then, like, making sure that metrics are, clean metrics are part of kind of the way that they're evaluated, keeps that team's data quality clean and then we use that throughout the rest of the org to say, like, look at all the value this team's getting with data. Look at how deep look at the impact that's been made. And what tends to happen is those folks wind up evangelizing to the rest of the org on their own. Mhmm. Yeah. I couldn't agree with that more. There are a few tactical tools that I like, there are there are a few tactical tools that I always just recommend. Make sure there's a data channel, where people can ask questions and try to post there regularly. Number 2, weekly metrics reviews. Try to get momentum for just reviewing. It's just fifteen minutes. Start with fifteen minutes if there's nothing on the box. We're just you're asking fun questions. And then 2 things that I really like, which are sort of, is the next level, but super fun. It's, mob programming on data. So have a few people that open up Mixpanel together, and only 1 of them can actually move the mouse and the keyboard, but that person can't decide what they do. And so all of the other people have to be the the question generators and the the people that actually tell you which buttons to click and things like that, because that will then have you narrate every single move that you make in Mixpanel, and it'll teach everyone in the session how to use Mixpanel. This is super, super powerful. Mod programming on Mixpanel. Really recommend that. And then the fourth one, if you are an analyst or a product manager or something and you're trying to make data driven, thoughts or insights or drive that as a part of the culture, Reserve. Start with thirty minutes and then go up to half a day on proactive insight hunting. And that's how you start finding insightful things and, not just be reactive in answering questions that are like, we have to find this out now. Like, you have to find proactive things that get people really excited. I really like that. Thank you, Ting. And it looks like we're at time. If you do have any additional questions, we will be sending out the recording, and you can reply to us with additional questions. We are so thankful for you taking the time to, share this with us. But without further ado, thank you so much. Thank you. Is it too late to add 1 more thing? 1 more thing. Don't worry. It looks like it We're off. Okay. It's ending. Okay.