Transcript for "Product Roadmap Review":
Hi, folks. I'm Brian. I'm VP of product here at Intercom. And, today, I wanna take, maybe twenty five minutes or so and kinda give you, a curated version of the road map, that we shared in at the November And maybe a slightly rawer version, and kinda share some of the thinking of, where my head's at, where the team's head is at, on some of the, items that we're working on now. So start at the start at the start. What we believe, we believe AI agents will do the vast majority of customer service work. We still have to figure out where that number is. Is it 80%? Is it 90%? For a lot of customers now, it's in that 60%. We do believe this number will keep increasing, but we don't think it will really get to a 100%, and that humans will be durably relevant. We believe you've gotta innovate at all layers of the technology stack, and we believe agents will just inevitably expand beyond customer service to do the sales and success jobs. So we do think there'll be a single AI customer agent category. So what this means for our investments is that kind of raw technology investment, bucket one, the thin flywheel, and that's what I'll focus most of, today talking about. And I won't actually talk about our investments in the customer agent. We'll leave that, for another time. And what I actually just going back, I'd also say we are investing more in the help desk. That's a decision we've just made recently there. So I won't go into details of that one. I'll just focus really on number two, on our fin flywheel. Now just framing the technology stack. The way we think about this is there's an app layer. That's what you use and and work with in intercom. The AI layer, which sits in between the models at the bottom and that app layer. And, really, what this looks like so I was mentioning this. This is the thin flywheel at the app layer. The AI layer is is easiest to think of as our RAG system. That's the system that goes, uses the model to figure out, what Finn should say, and then the raw model layer, which all those components in there. And the key point to make is that, we are still investing here at this model out to expand our footprint of where, we actually own that full technology. And the reason we do this is for latency improvements, for efficiency improvements, and for product quality improvements. We think the best product here ultimately will come from owning that full stack. So, and look looks like this number will increase a percent. We'll have to wait a few more days to lock this one in. All of that work is what really drives this continued performance improvement that we're able to give to all of our customers. Okay. So let's jump into to the flywheel. And I actually wanna start so the flywheel train, test, deploy, analyze the bedrock of how we framed our our fin product. And, I wanna start with, analyze, and I wanna start with automation rate. And, this is a a new metric that we're, gonna be bringing in very shortly, hopefully, in the next, couple weeks. But it's a simple metric. And let me explain why this metric is so important is that we have emphasized resolution rate, front and center, and people think, great. I've got, when Finn's involved, I got a 66% resolution rate. That sounds fantastic. I'm really, happy with that. But the reality is it enables people to ignore the fact that they're not letting Finn, have a chance to answer at a meaningful chunk of conversations. So you see here if someone's got, Finn not involved in 52% of it, that their actual automation rate, which is of all your inbound conversations, how many is Finn actually resolving? And that number is 32%. And we really wanna ensure customers aren't too anchoring to just resolution rate when, ultimately, automation rate is really the overall impact that Fin is having. And the result is that we think we've encouraged customers to not focus enough on involvement rate. They've just focused on resolution rate. So we're bringing that new number automation rate, which is just involvement rate times resolution rate, more front and center. Fin's performance and impact to your team, automation rate, the quality of Fin, that's the CX score. That's front and center. Now one of the big levers, this is not a roadmap one, this is now available for everyone, is Fin attributes. And this is one of the, big product improvements we tried to make to really encourage people to have that effectively approaching a 100% involvement rate is what we want. And and Fin attributes is really what we saw with so many customers, particularly who were using started with the help desk, is that Fin was kind of buried in complex workflows, and this was really constraining, Fin's involvement rate. And so Fin attributes is a way to use AI to do the job that people were using deterministic workflows, button based workflows before, which translated into, a less than ideal experience in the messenger, and to let AI do the work of those workflows for you. And then you can bring Fin up front and center and have that AI agent concierge experience so that when someone comes in, just tell us what you're coming here for. Fin will do the work to triage and route that, and you can still use, escalation guidance and even better escalation rules, which work more reliably when you wanna put attributes in, to ensure if there are conversations that you aren't comfortable with Fin answering, Fin can completely just skip those and say, I'm gonna hand you straight to the team. And so I'll actually just show you. This is this is literally designed from last night, draft design that someone shared. But, what I wanna highlight is this this part over here, and we'll try to make this more clear of, like, hey. So this number, we want to be almost a 100%, ideally a 100%, and that will show you more clearly, hey. Finn was involved, but some of these Finn escalated. Some because the user requested and said, no. You didn't answer my question. I need to talk to the team. Or some because you've set up your configuration rules and said, Finn, don't answer that. And kind of make that much more clear of what's going on. So we're really gonna try to expose a little more of the system at a little more complexity of this diagram so you can really get a sense of the levers and see, where your opportunities are, not just for resolution rate, but for involvement rate. Okay. So now let's move to resolution rate. Hopefully, this is, repetitive, and you you've heard this before, but it's just foundational to how we think about things is that conversational taxonomy break breakdown. What type of of question is your customer coming with? Is it and these map too. Informational maps to content can answer it. Personalized means there's some data, but, usually, you need to, like, read a database. For example, what's your account balance? Actions is usually multistep business logic or going and changing a value in a database. Hey. Can I cancel that order? And then investigative is where someone on your team had to go back and check with other people, and there it it was content alone couldn't answer it. So, the key point here is that, first of all, for, content, we've done a lot of work to real and this is more continued work from our insights product to really point you at content improvements to make to increase that resolution rate. So from these AI powered suggestions, like, here's the exact content that we saw from your human answering a question, and now you need to add this to your content so Finn can answer it, and then helping you clean up your content. Here's contradictory or duplicate suggestions. Here's actually suggestions by audience and brand to so you can prove that for that next level down, of complexity of getting that content right. So we're really trying to help you get that, informational resolution rate, maxing that out. And, actually, what we see in our customers look looking, in aggregate here of, like, where is that unresolved breakdown? As most customers are having the most success for informational, and that the the bigger opportunities more generally are in the actions in that personalized space. And, hopefully, everyone's aware of procedures. We've talked, a fair amount, about this, of this is our big product, effort to really tackle complex business processes, and when you actually have to change data. And procedures is where it's based on a a more straightforward doc format where we integrate deterministic controls inside of that. This is the huge design challenge of how do you balance generative AI with deterministic controls because you have to get these things right. So that's the modern natural language editor that incorporates those deterministic controls within it. So, we're making great progress. We're really excited with where we're seeing with, customers. This is still in in closed beta. You can see here is where you've got you can even incorporate code in there to ensure that Fin is doing exactly what you want, in certain situations. So this is progressing well. We're really excited about this. And what we've also talked about here is where we're actually using, an AI assistant to to help out as well because frequently that getting from zero to one, this AI assistant can get that first pass of that procedure way faster and really sets customers up well. But the key thing here and and just going back as this is, still in private beta, and our hope is to get this to manage availability, which is essentially work through sales, to get this into that's our hope is to get there in January. Our intent is to get this to general release so everyone can use these, but we are not yet sure if we can do that. And the reason why is because it's easy to go wrong here. It looks easy, but customers can actually set these up in a way that, isn't working, isn't successful. And so, we want to ensure we're not letting customers dig a hole and degrade the FIN quality, and the experience here unintentionally. So our our reason for holding this back to to that is just to ensure we have confidence that customers can successfully do this on their own. So we're working through that more. So, hopefully, this is in January. This gets to manage availability, which is working through sales. But what I wanna add, and and this is actually pretty fresh what we've realized, is us recognizing that data connectors are actually the the big unlock for customers and where almost all customers should put more emphasis first before thinking about procedures. So data connectors are there. They're available to everyone, today, and we know there is more work to do here. Where there's authentication improvements, other, technical improvements we're making here. We need a lot more UX improvements to make data connectors easier to use because, what we've learned is is that they're very effective for those personalized answers. And, actually, what we've learned sorry. To interrupt, what we're also gonna point to with insights is, hey. We're gonna point to here's improvements you can make with data connectors. Here's data that you can inject and try to point you how to do this. And this is the key point that is pretty fresh from where our thinking is is that, look, everyone should lean into content first. Why? It's easiest. It's fastest. Yes. Sometimes it's still a big lift in getting your content to the right place, but it doesn't require any technical, expertise at all. And you can get huge value out of from Fin so so fast, and there's so much opportunity there that, to go with. So content first, then go to data connectors, then bring in that data for the account balance for what type of account do you have that can really and what Fin does so well is combine those first two. So feed Fin the data, and you can give the like, going back and kinda control, give guidance on here's how to use this more. And and then Fin can combine content answers with personalized answers and really give this beautiful synthesized answer. And then the third one is when you've got complex business processes that need multistep laying out, that maybe you're going in and changing a database. That's the most complex. That's, like, the third bucket to go after, but it really should be that sequence. So one content, two data connectors available today. We are gonna make these easier. We're gonna make these more prominent in the product, but the k the raw capability is there today to go. So that's the thing I'd love for folks who are keen to get at procedures, but it's still restricted to, like, lean in to data connectors now. Fin works great for this, and we will make this easier, but you can this is unrestricted now. Okay. Next, I wanted to touch, quickly on, test. So, simulations, we talked a lot about these at Pioneer where, these are really, right now, framed as almost part of procedures. And that's how we're framing simulations, which is where you go in and describe an automated test and give the evaluation criteria, and then, we run that test, simulate an end user based on the criteria you said, and then evaluate if the procedure worked effect effectively. So simulations and procedures really go hand in hand, and that's what's innate enables you to confidently build out a procedure step by step and sequence your way to, like, yep. It's doing step one right, steps two, step three, and it's working for all the different, customer attributes and types and variations and edge cases to build that confidence there and to build the confidence so that it didn't break, unintentionally afterwards. So, really, procedures and simulations go, hand in hand to building confidence for for automating those complex queries. So, again, we're building in an assistant to help with these. Often getting the first pass is is is better. But what it so but those are right now, those simulations are are are tied directly with procedures. But what, where we're leaning more into, and this is pretty fresher thinking, is the importance of version control. So we're bringing this in sooner, across multiple parts of the product. And then that version control also tying that in with the preview capability, with that testing, capability, and these will be for procedures, but also beyond that. So, that's where we're leaning into now is better version control and better preview control so that, across the product across the Fin product space, you can have more confidence that changes you're implementing are right before going live. And we're also gonna, lean more into Fin QA, which is when it Fin is live, there may be certain areas you wanna ensure Fin is doing right on. Hey. I wanna just ensure for this specific area, say, this billing type query or or this thing. I wanna be sure I'm happy with how Fin is working or the improvements I made that I wanna check up and validate they're working well, as well as a broader just make it easier for humans to do the QA work here, not requiring them to build their own spreadsheets or or workarounds here. So we're gonna also this is coming soon. We'll share more details when we have more, confidence of the shape of what this product is. Now, from a Deploy point of view, voice, I hope everyone knows about this. It's just super exciting to see the progress here and and see this, landing in the real world and bringing procedures into voice. There's a big meaty road map here, but I'm I'm, I won't go into, the details, of voice for here. What I last one to jump down to is, is back to insights and and to CX score. So we launched, CX score, going back, I think, at the start of summer. And this is a really significant iteration that we've, made to the under the hood, how we're calculating it. And, really, what we've brought in is these additional attributes that we're evaluating Fin against, these score reasons. And these score reasons have multiple benefits. First off, we're actually able to increase the coverage of conversation that we can apply a CX score to because going into the details is a bit messy, but, basically, using these CX score reasons gives us more confidence that we can give an appropriate rating to a score, which previously, if there weren't many parts to it, we just couldn't rate it. So this has helped increase the coverage of, the score. It also will give you more confidence in the accuracy of that score because it's giving you the the specific reasons, not just in the the summary that we gave overall, but these specific reasons because they're attributes. It means you can go in and say, hey. I wanna find all the low AI agent answer quality ones where there was a negative emotion and then be able to dig into those conversations at scale. So these really unlock huge new power. They increase the quality of the score. They increase the coverage of the score, and they unlock huge new value of you really being able to dig in, understand what's going on, and improve, improve, improve. So we're really excited about this. We're gonna share, benchmarks so folks can have a sense of, how they map to other people in their industry for the CX score. And we really feel with this iteration of the CX score, it can now be your absolutely primary, metric for reporting on fin and and human quality. And there's a bunch that you can build off, now when it's built into the system there. So, those are the, key highlights that I wanted to touch on for for this road map. There's heaps more in there. I don't wanna bombard you with data. That gives you a sense of of where we're at, where we're thinking, and but, ultimately, the the biggest picture for us is thinking about, that automation rate for Fin and getting that automation rate, which is both involvement, get Fin on all the channels, and the resolution rate, get Fin answering all the question types, the query types, combining those together so that you're getting to the where the peak is in that 90% Fin automation rate. That is achievable today. We have customers getting that today, that's what we want everyone putting that deliberate effort in and setting that goal high. And we're trying to help you get there as much as possible, and we will keep doing that. So, I'll wrap there. Thanks, everyone. Hi. Good afternoon, everyone. It's Tim. I'm a program manager here at Intercom, and I'm just gonna be going through the remainder of the questions. So I encourage you to hop over to the q and a tab, about any questions that you find interesting or relevant to today's session, and then pose your own, I guess, if you if you don't see anything that, strikes your eye. So I'm gonna be going through these as best as I can. For anything that I'm unable to answer, what I'll do is I'll push it over to, Brian and his product team, and and, we'll be able to follow-up with you personally. But so far, I've been doing okay. So, let's see. Some of your competitors are looking at being able to handle much more complex customer support issues. Is there something in the road map that addresses these types of inquiries specifically? Yeah. So with procedures, I know that we in in the past, especially with our, webinars, we've tended to lean more towards, like, b to c examples of, like, refunds. But anything, that is able to be modified with API is gonna be able to be done with procedures as well provided that you have the right data connectors set up. So I'm not sure about how much more complex these customer support issues are, but I can confidently say that, like, as long as there are API endpoints, and data that that can, be updated, as long as it's connected with data connector or even, like, an MCP connection for FIN, these new procedures are gonna be able to conduct them, with a high degree of success, along with the procedure or sorry, the simulations as well, and that I'm really excited about. So Fin basically simulating all sorts of different, like, potential outcomes of the situation and almost ranking them together to see if it's able to successfully resolve the the query or not. So I feel confident, and the team feels confident about, its ability to handle, complex queries as well. Will there ever be a live overtest for QA? It's literally impossible to test the experience without an exact copy of the the database. I'm gonna need some more clarification on this question, if you don't mind. I I I wanna know basically what what what issues you have with, like, the current testing that we have available and then what we've announced for simulations as well. I think that's actually pretty thorough, and provides very, very actionable recommendations in terms of how you can continuously improve your AI agent, especially in bulk as well. So if you don't mind, I don't know, popping some some additional details in the chat. Jennifer would love to address this. And then another question. If you ever disagree with CX score, especially one that is ranked higher, maybe not as great. Is there a way to communicate this back to Fin? Yep. It's just something that's been identified by our product team as well. And so, I believe this is at a very, very high level testing. But they've discussed this thoroughly throughout the the testing of CX Core. But with the announcements that Brian has made to CX score, we've expanded the criteria that the the AI looks through in terms of applying the actual score, and we feel a lot more confident now more than ever on it. So, if you actually wanna learn more, we actually have a webinar coming up this Thursday on the new improvements that we've made to insights. And, Christine, actually, one of the directors of product management, is actually gonna walk through, the the the recent developments that they've made to the CX score and the improvements to it as well. Automation rate data. And a lot of this is already live already. So, take a look. And, again, it's it's simply involvement rate multiplied by resolution rate. That's the overall automation rate and, overall, how many of your overall queries are being handled successfully by a Fin AI agent at the moment. So for for beta releases, what I'm gonna do is I'm gonna recommend you to, join the Intercom community here. There should be a channel for you to request beta access. Otherwise, if you have, access to a relationship manager or account manager here at Intercom, you can always request through them as well. But if you don't, the Intercom community route is always available. Any updates on FIN for platforms? For example, FIN over API, still in development. Anything that you see on the FIN website announced there is is what channels that we have available at the moment. But last I heard, FIN over API was still undergoing internal testing. So very, very early, like, alpha testing stages at the moment. So I can't, comment on anything too confidently there, but it's been floating around. So I know that we're actively developing Fin over API. When will Fin include videos and screenshots from the knowledge base? I don't think that's being discussed right now in terms of the the the road map. Again, Thin is able to understand images that are sent to Thin, but not vice versa. And, like, I think we're talking very, very far from now or Fin's gonna be able to use LLM to to generate, media. But, again, we're talking a very, very long time from now. There are ways to circumvent this through custom answers, but that's the only way that you can have Finn kind of, like, reply with, rich media embedded within it. Otherwise, it it's purely text for now. So I think this might be an issue that we are going to have to think about offline. So, Kimberly, why don't we, speak after today's session? I know that you're in conversation with members of my team about this. So, yeah, I will make sure to follow-up with you after. This sounds like an issue related specifically to, the knowledge hub. The new changes might help with that, but QA does not have any of the workflows, knowledge, etcetera. So the only way to test live, is in live, like attributes, workflows, and guidance. I need the workflow established. It's the one we tested. Well, again, in the, preview tab from guidance or even from, like, your specific workflow. If you enable a conversation events, you should be able to see everything that's being applied. So attributes, workflows, guidance, and also, like, a brief explanation about why it was applied as well. So I I actually did screen. So, you know, there's a lot of ways to test without it being live, and that's through our, our testing tab. Now live testing is one of the many ways that you could conduct testing. But we also have, let's say, like, very close testing and also, batch testing, which I would heavily recommend. Is there a plan to easily be able to copy content settings from one workspace to another. I'd love to be able to have a one to one copy of my fin setup in my staging workspace so I can test changes to confidence guidance all at once. No. I don't think there's an easy way right now to, like, transfer information from one workspace to another. And it's a very cool idea that can definitely flow around with our, our product team. But, yeah, if you if you create, like, a testing or a staging workspace, and, unfortunately, you will have to, like, recreate workflows and have it ingest the same guidance as well. Where is the link for Thursday's call? So, great question. What I'm gonna do now is I'm gonna link our webinar hub. We're gonna be running this webinar across, multiple different time zones. So take a look and see which ones, will fit your schedule the best. Again, this one is on insights analysis, and that's where Christine from the product team, is going to be joining us to to lead us through her thoughts and then, like, a quick demo of everything that her team has been working on for the new insights tools. Anything on the road map for intercom phone, a stand alone app? So so, yeah, for FinVoice as a stand alone app, not so much. But FinVoice should be available for Fin standalone. So if you're using Fin on a different platform, it should still be available there. Do you know anything about development for hyperlinking to products instead of links? We run something very similar here, at Intercom through guidance. So in Fin's communication guidance, you can have it set so that it it it, it hyperlinks directly to specific things instead of pasting out links in full. So try that. We've done it successfully here at Intercom. For for any comments to Brian, I can definitely pass things on to him for sure after this. So I'm gonna try to blitz through these last two questions. And then anything else, you know, feel free to pause now or join us at the the webinar on Thursday if you have more questions, perhaps on, the insights tools. Is there anywhere that will show the hierarchy order in which things are applied, workflows, rules, guidance, procedures, etcetera. Yes, Kimberly. So this is gonna be done, and I'll I'll link it again here. This is explained very thoroughly on our page, which explains the AI engine. And so it walks you through, for example, the process of fin generating an answer and when, let's say, escalation guidance is has taken place, or when, for example, guidance is going to be used as well. So as a reference so I'll paste that there. You can read through, the phases there. I think I summarized it very thoroughly in a previous webinar that we've done on guidance and when that's applied within the engine process. So, take a look at the webinar landing page, and then you can navigate back to our session on guidance. I did it I think I personally think I really I did a really good job at, like, mapping out a process in in which Fin considers everything, that's set up from. Any work done to have email fully supported in multibrand workspace? Yeah. We've made a lot of developments, in our multibrand recently and, especially for multibranded Fin as well. So I don't know specifically what you're looking for there, but, we've we've made a lot of developments in terms of, like, switching domains, and and sender emails as well. So, folks, that's all the time that we have. I wanna be respectful. Thank you so much for taking the time to join us today. I will see you at the webinar on Thursday for insights. So make sure you register, And thank you all so much for coming out. I really appreciate it. Bye.