Name: 💡 Dataiku & NVIDIA: Scaling AI Agents in Financial Services
Uploaded: 2025-07-17T20:16:04.980Z
Duration: 1 h 12 s
Description: 💡 Dataiku & NVIDIA: Scaling AI Agents in Financial Services

Transcript for "💡 Dataiku & NVIDIA: Scaling AI Agents in Financial Services": In particular, we're going to be talking about generate.ai and at a scale successfully, facing the challenges that all of you will be well aware of. There may be a few more that we'll talk about so shortly. So I'm John McCambridge. I'm the global director for solutions for financial services at Dataiku. I'm joined by Malcolm and Shashank. Malcolm, please feel free to introduce yourself. Thanks, John. I'm Malcolm DeMayo. I'm the head of global banking here at NVIDIA. Shashank? Hey, everyone. Shashank. I'm a partner solution architect with Dataiku. Marcus. Thanks both. So, this conversation here today is gonna be, in particular, focused on generative AI and financial services and delivering real success at scale. In order to get ourselves into that conversation effectively, we're gonna start by discussing the state of the industry, some of the challenges and concepts that we think will be really illustrative and informative for you to have as you tackle this very real problem. We're gonna then shift briefly to discuss how our partnership between NVIDIA and Dataiku has been designed to solve the very challenges that we're seeing very clearly in the space. And most importantly, we'll then pivot to a very concrete demonstration of that partnership and those solutions in action led by Shashank. So in order for us to get that conversation started, Malcolm, let's talk initially about the trends and challenges that you're seeing in this space and how ultimately we're going to try to help folks succeed. Thanks, John. It's a great, great lead in. So, you know, banks have been at the forefront of tech adoption for decades. We've watched them implement ATMs all the way through full full online mobile banking platforms. And in the last last decade or so, big emphasis on digitization. And what's clear, what's become very clear is that AI has evolved or improved from kind of a tool that you could analyze vast amounts of data, identify patterns, make predictions. It's it's improved now to where it's making independent reason it's reasoning independently and able to take action. And if you think about how this can trans how this will transform banking, you could take what was an impersonal transaction to an end to end journey. And a great example of this, I think, is Capital One. Their recent chat concierge, which Capital One used to be a very big, financer of car loans. They still are. But CHAT concierge is now engaging client prospective car buyers in the entire journey end to end, and go check it out. And and so at our GTC Paris, Jensen in his keynote made the observation that AI factories are here. Sovereign AI factories are here to stay and, or or coming fast maybe the way to say it. And that every enterprise needs to transform from data centers, which is a place where we store and retrieve information traditionally, to AI factories where we generate intelligence and we and and basically, if you're able to generate intelligence, you're going to generate revenue, and that's the difference. So AI, it's very clear, is a transformational, power in, or catalyst in banking. But what's not clear is how to get there from here to a lot of enterprises. And and that really is the purpose of this discussion today is how Dataiku and NVIDIA have come together to provide a lot of the capabilities necessary to do so, to to to to join that transformational journey. And I suspect, Malcolm, given well, you've you've set up, I think, very nicely the, the successful opportunities here, the positive, returns. I think, obviously, folks in the industry are gonna be having the other side of that coin, which they always have even with all the previous successes, which is, alright. Well, innovation is great. So now talk to me about controls. How do we actually do this in a way that is effective and also could be done at a scale, and to continue to maintain that control even as things begin to build out. So we have this AI factory. We're gonna build one. How do we make sure that we keep that locked down nicely, make sure that everything's working well? So I think, you've got some great concepts here, I think, for this idea and thinking about controls in the space. So I'd I'd love for you to introduce those to the audience here, and then I think we'll get start getting into the, concreteness of it and to be able to show that, yeah, these are real challenges. Now let's talk about how indeed you can start knocking them off and and being successful at them. Well, it's really a it's a it's a it's a perfect lead in. I mean, great question, John. Because you think about machine learning has been in banking for a long time. And machine learning models tend to do very basic things and they tend to be very small and they tend to be able to run on CPU servers very nicely. But when you look at what's the if you look at the pace of change, AI models now are doubling in size every six to twelve months. And, this year alone, we're at 1,600,000,000,000 parameters. If that, trend continues by 02/1930, it would be 10,000 x bigger. And the laws of scaling essentially cite that, you know, the more parameters and the more compute you have, the smarter the models. If you think about accuracy, accuracy has made huge leaps, in the last couple of years. You know, you look at achieving that 60% accuracy threshold, I and on some of the benchmarks, I think the model size has dropped from you did two years ago, two years two, three years ago, and you had to have a 500,000,000,000 parameter model. Now you're achieving that with a 2,300,000,000 parameter model. So massive lease, but guess what? 60% is not good enough. And at the end of the day, because this this technology is able to act independently and reason, it introduces a whole bunch of new, challenges. The good news is banks have been, very good about implementing technology as I mentioned. They have, you know, four lines of of defense already in place. And so now it's a question of can you how do they quickly recalibrate that framework to deal with this new tech this new and fast paced evolving technology. And that's where we come in. That's where where Dataiku and NVIDIA come in to assist. You know, people love to talk about the data problem, data privacy requirements, the bias in data, etcetera. Data's been a problem forever. You know, the old saying is the best time to fix your data was twenty years ago and the next best time is now. So, data is something that we can help with, and without getting into great detail, from a from a monitoring and from a control perspective, having a unifying a unified platform that's available everywhere. One of the things Jensen talked about in his his, keynote in Paris was AI is gonna be everywhere. It's gonna be in public cloud. It's gonna be private cloud, it's gonna be in dedicated clouds. And it was fascinating that, the CTO of BNP Paribas, Jean Michel Garcia, was quoted, in a in a session saying, he basically said, look, we've been we we wanna control our destiny and this is sort of tapping into the sovereignty. We wanna be in control of our destiny which essentially means, there's sort of three pillars to this. One is independent and trustworthy infrastructure. The second is end to end data protection. And and end to end, he's talking about across, a dedicated, cloud, a private cloud, and a public cloud. So AI living everywhere, as Jensen pointed out, across that spectrum and then having reliable models that are aligned with the ethical and regulatory compliance standards of the day. So, but that sort of sets up why we're here today, I think. From a from a just keeping in mind from a, how do I connect to the rest of my enterprise? Banks banks are pretty much put together through acquisition over decades, many of them, and, you know, they have a lot of legacy systems. And some banks, some leading banks have already started replatforming using AI, these undocumented COBOL based systems, to something more modern like Python. But you have client server legacy systems, you have SAS legacy systems, you need to be able to connect AI to all of them, and what you don't want is a polyglot of AI. You don't want 45 to a 100 different tech stacks. You want a uniform tech stack that can live anywhere, to to be able to build, improve AIs, create and and today, we are seeing back to the accuracy challenge, we are seeing using NIM, NEEMO, data mesh, creating the data flywheel, the ability to drive accuracy to new levels, at the achievable levels that are going to enable putting these AIs into production in a trustworthy way. Absolutely. And so you're describing here a really, complex, situation. Right? Folks are having to grapple with the fact that they are going to need to have a rational output. They're gonna need to have a coherent space that they're working in that is a shared space, but, one that has a pretty consistent approach. But at the same time, they can't become overly rigid because they're definitely going to need to be connecting to multiple different data, sources, and those could be, like you say, anything from a legacy legacy system to multiple distinct cloud environments with different requirements. And likely, they're going to have to be dealing with multiple vendors. And with those vendors, perhaps, multiple variants of models. As you mentioned, there's likely going to be model situations in which you're gonna need to leverage whatever the cutting edge model is that you can get your hands on or other scenarios in which a distinct, perhaps fine tuned, or simply older but more, slightly targeted model is the correct one to go for. So having the this selection be sensible, so not everything out there in the world, but also not narrowly defined to say, like, one vendor or one locked in option that you chose a year ago. This is a different difficult balancing act. You're really trying to to strike it. And as you say, I I I always like this point. It's was true for machine learning. It's still true today with generative AI. The fundamental nature of these problems, if you boil them right down, are the same problems banks, insurers, and so on have been dealing with for decades and decades. It's the same type of attitude that needs to be taken, which is this balancing act between sensible, not trying to get something that is the perfect fit for everything in the moment and boy that brings it all down to one single point to retain the sense that you're going to have to be flexible, but you need to be efficient at the same time. And so I think that that's a a really challenging situation for for folks to to tackle. It's not an unfamiliar one, but but it is challenging. And I think, thankfully, as you were outlining, NVIDIA and Dataiku together both offer, opportunities to approach that problem much more effectively you were mentioning. And I think the efficiency point is so important because banks, John, are, you know, they're it's in their DNA, first of all, to always show a return on investment, but also to continually look for ways to deliver their services less expensively. And with together with that data, one of the things we're enabling is the ability to take a model that an open weight model now that's performing at a at a high accuracy level and distill that knowledge down to a smaller model. Say say you start with a 70,000,000,000 brother model, you distill that, create this flywheel where we're using fine tuning and chain of thought prompting and other techniques to, basically transfer that knowledge to, let's say, a 7,000,000,000 parameter model, which would now run at one tenth the cost. But, of course, your efficiency is going to drop dramatically because of the laws of scaling, more parameters, smarter model. But again, by training, fine tuning with your proprietary data, using, different techniques, we can get that that, level of accuracy back up and then do it again. Distill the knowledge down to a 1,000,000,000 parameter model and keep pushing this, as techno as the technology improves, as our capabilities evolve. So it's it's not just, having, you know, the the fact that we have created an ecosystem and that AI is a team sport. It's also being able to build the capabilities that will allow you to, deliver these services more efficiently, which I think was a great point that you just made. Yeah. Absolutely. And I think we keep coming back to this, and it's something that's, really important to this conversation is the realism behind all of this. I think, ultimately, that's one of the reasons that we wanted to make sure that this conversation is actually founded on a very concrete demonstration of everything that we're just discussing today. And so I think what we'll, start doing now is we'll move the conversation over towards Shashank because I think what we really wanna, make sure is clear in this conversation is that these are concepts, of course. These are ways we think the world is today and how it's going to unfold. But most importantly, for those, that perspective, we have a very concrete idea of what we think, will help. And so this, solution that you're about to see here was built, using Dataiku and NVIDIA as a partner, together, and it was, recently kinda launched at the GTC Paris as a an AML blueprint with, NVIDIA. This is part of an overall blueprint concept that NVIDIA has and one that we're delighted to be, a partner of. And this is one example of these, blueprints that could be built out in Dataiku and, of course, in other systems as well. The demonstration here is an anti money laundering agent assistant. So it is leveraging the most powerful capabilities of generative AI today, which is not simply the processing and interpretation of information, but the structuring of that into an agentic flow in which the agent is making decisions, is following multiple steps, and is then bringing that information back in order to iteratively move towards a solution. Just as importantly, though, as Malcolm was saying earlier, banks have their existing processes to manage risk, and they need to be leveraging those to the maximum. So this exact same process in which the agent is successfully delivering value is very carefully encapsulated using a human on the loop. So there is critically not a replacement of any human agent here. Instead, the human agent remains and is massively more effective, much more efficient, and much more capable of making the critical decisions that they are ultimately there to make to be making. So I'll move it the conversation over now to Shashank, who will demonstrate this in action. What you're about to see is a fully developed, FSI blueprint, running on Dataiku, NVIDIA, in which you will see exactly this framing, a very realistic approach to a very real problem in which the human is dramatically augmented by AgenTek AI and is able to perform their most critical functions ever more efficiently. So, Shashank, please take it away. Yeah. Thanks, John, and thanks, Malcolm, for sharing all the challenges that these organizations are facing. And I'll be switching to show this blueprint solution that, John was talking about. So start with, the solution has a a notification mechanism built in. So as an investigator, I'll be receiving notification, and this is something you can schedule to run on a particular schedule or on a particular specific trigger. So what you're seeing here is the agent at the end as an output was able to generate, a small summary for each of the alerts it has received. So as an, as an investigator, I can look into these and I can also use, a dashboard link which comes within the email to further deep dive into these alerts. When I click into this link, it will open up, a dashboard in Dataiku. Now what you're seeing here, it has for every alert, it has a tab, and within each tab, you will see these two views. On the left hand side, you have a a detailed summary of that particular other and this summary is generated by, this agent. And on the right hand side, you have the chat interface, the same agent. And it has all the context of this, somebody that it has generated. So you can directly start it asking question, in this check interface using, the summary that you have on the left hand side. Now first, let's look into this. So each for each of this risk, it, creates an initial summary, like a very high level overview. What's the risk score? What's the associated, values like target account number, transaction ID, and give you a quick summary of, this alert. Then based on the type of risk it has identified, it creates different sections. For example, for this alert, it has identified, shell complete detection as well as, intersected transaction risk. So you will see a separate section for each of this in this report. So for example, here in the shell company detection alert, you have company profile, unusual findings that agents identified. What are the red flags that it has observed? What investigation methods the agent has used to come up with this, summary? And it's also followed up by, some tips on what is the further investigation that as an investigator I can do, to look into it. Alert. Now here you see it out it automatically gives me all these data points that as an investigator, I have to go and look into these datasets and summarizing in for into certain key points, which as an investigator, I can focus into instead of looking all the associated data points. Similarly, it creates a section for another alert. You can see, the subsection, varies. It also depends on what type of risk it has identified on this transaction. This one also has a quick view as a a graphical interface. So based on all the entities associated, for this alert, it has able to create this view. It is using a Dataiku, plugin, which is a graph database, visual plugin. And you can even take this Cypher query that that agent has used to come up with that directly into, this graphical tool, paste it, and then look into the dataset. In this view, you can define your own nodes and edges based on the relationship that you want to analyze on. And if you quickly zoom out, you can see that these analysis can become very complex. And nowadays, agents are very capable of going into different, graphical directions, assess all these, and then came up with a small set of key relationship that as an investigator, I can look into it and completely remove the additional noise that is coming from the data. So going back to our dashboard, so, yeah, similar to that, another section providing recommendation on what as an investigator I can do, for this kind of risk. Now to use the chat interface, like, the things I can do is, suppose as an investigator, I found this person, Steve Monroe, person of interest. I wanted to see how this person is related to, this alert and what is the risk that this individual has on this. So I have already have the preloaded check. You can see, I can ask questions directly to the same agent who generated this report and get the findings, get the associated source that on what basis, this, this assessment has been generated. And then this whole chat will keep the contact so you can keep asking more and more question based on what information you have received to get further deep dive into, any specific topic of that. Now you see for a different alert, the sections may vary. So the initial summary remains, the initial summary will be available for each one of them. But based on what kind of risk it has identified, you'll see different subsections available. Like in this particular alert, we have shell company structure analysis, cross border, cross currency. So it depends on how many risks are identified. So sections would keep changing, and then based on what type of risk it has seen, it would create somebody accordingly. Now so this is the final deliverable of the solution. Now let's see what how the solution has been powered in the back end. So overall, we have right now five agents that are running this solution. We have our master agent, AML investigation agent, and all these agents are powered with model from NVIDIA. We initially tested this with Lama, 70 b instruct, and then I think recently NIM, we have Lama four Maverick available, so we switched to that. We're also using Mistral NIM for certain, tasks that will, come up in later. So we have this AML investigation engine. This is a master agent that's orchestrating all the process. It has built using Dataiku visual agent capability. So in Dataiku, you have both visually you can create agents without, like, a no code interface and also you can fully customize an agent using a code interface by using coding language like Python. Similar to that, you can also use capability to create, visual tools as well as create your own custom tools using a coding language. So in this view, what you're seeing here, I can select a particular model. Here, I'm using LAMA for Maverick. I just need to make sure that model supports tool calling. Create a detailed prompt on what this agent has supposed to do, providing the information, what tools it has access to. So what you're seeing here is all the sub agents are available to this agent as a tool. So you have PEP agent, AML data agent, SAP preparation, and the graph exploration. Similarly, you can add more and more tools to this agent with, like, few clicks. Based on what are the tools available in this project, you can directly, add to this, agent. Using this quick stack quick test interface, you can also quickly run these agents here and too because this is a more an iterative process over time. So you keep doing making sure that you have the right prompt to generate the output. Now to support it, we have, four agents. So let's look into that. We have AML internal data agents. It it is also a visual agent using, a new model, and it has access to further sub tools. So this particular agent is used to scan through all the datasets that you have available in the project. It can search all of them using some key, columns like alert ID or or a particular name of a person and then get you all the information. We have another agent called PEP agent and this is a code agent. So here you can see it has been built using, Langchain and and the Python technology. It has two tools. One is to search in a PEP database as well as search for a particular entity over web. And we are also you can use the Dataiku LLM mesh SDK to point it to any of the model attached to Dataiku LLM. Here you can see we are using LLAMA again. Similar to that, we have two more agents. One is the graph exploration. This is actually connecting to, a Dataiku graph plug in, to access the dataset in a in a graphical, representation. And then we have a SAR preparation agent, which is used to generate a remote, generate a report and then send us as a final email notification. Now look into the batch process that has had been pros performed when the alert has been received. It run this whole batch process. It should look into it and then finally send that email. Here in this, we have this agent and it attached to the final dataset and we're running a prompt recipe. In the initial prompt, we are just sending asking the agent to take this alert ID which is coming from the dataset and summarize its findings. Now here you see we don't need to actually put in all the dataset to this agent because it has access to this tool based just based on the alert ID, it can identify what sub agents can get in that data. Like, in this case, it will be an AML internal data agent, and then get all the information and then use it to further reasoning on that. Once we have that, our next step is to using a general purpose LLM, in Mistral. And here, what we are doing is we are using this, LLM model and providing the information that here is the summary of the report, and then your job is to create questions based on all the sub agents available to our master agent to generate prompts. The reason we are using a general purpose LLM here and not the agent because we want to make sure that the prompts are created independently without the prior information. So it's similar to an investigator asking following up questions and then following up on those data points. It generate those queries and then force the agent to use these queries against its sub agents, to generate a summary. Now this is, this is a final output of, like, when you complete the prompt engineering. You do the prompt engineering in Dataiku. You can you have something called prompt studios. This is where the whole prompt engineering happens. It's an iterative process. You run multiple prompts, choose different models, you provide the prompt, we can add examples to support that. You can add datasets to feed in columns so that it can run on a as a batch process. And here you can see the results compared different model results in a in a prompt output compared competitive view. And then once you're satisfied that this one looks good, you export it as a recipe and that's what you see it here in the, in the flow. So it's it's really helpful when you have to do prompt engineering for, like, multiple prompts and then multiple models that you want to compare. Now going back to our, flow. So once we identify that, we create these prompts and then we do some data engineering. We break it down all these into one single record. So, like, we have four transaction and then so five transaction and after that, when we create prompt for each of the sub agents, which has four agents, so now it has 20 records. So we have one prompt and then one output, for each of this, alerts prompt. So you will have, like, a five outcome of, each of the other. And this is where we use the agent again because now we created the prompt using a general purpose element, and now we are asking agent again to use these prompts against its sub agents to get more details. Once you do that and there's a steps we generate titles. As you see, every alert has a title, in the in the summary that the agent generated. Furthermore, cleaning, there are steps, there are steps, to, like, generate further summary, convert it into HTML, then we're using a Python recipe to create a custom template. And this is what you see, what you get in the email as well as what has been shown in the dashboard. And then as a final output, we generate, HTML files, which we can attach it over email. Sorry. Focus. Yeah. We can attach it over the email and send it to the users. I think I'm zoomed in. Yeah. So here, what you're seeing in this whole process, we are using some LLM powered recipes, some general data preparation, some of the visual recipes like no code interfaces and couple places, a code where we need to further customize it. And that's how, right now in today's date, the agentic flows looks like. It's not just putting prompt to an agent, but it also combination of lot of your business logic, customizing the final outcome, and then taking advantages of, LLM powered recipes as well as just normal data cleaning and data preparation. So once you have this flow generated, I think the the final step is to automate this flow. So in DataIQ, you have, what they call scenarios, which can be used to automate workforce. You can define your trigger or, like, if you want to send these alerts to investigation over, like, on a weekly basis or monthly basis or even on a, like, a a particular dataset change, like, it triggers a dataset change, it automatically triggers this flow. And then you have reporter, so you can define how you want to notify the end user once this flow is completed. So here you see we are using an email notification. We have multiple options like Slack notification on Google chat, and then we are using a custom template which actually, takes the HTML report that we generated, convert it into a text, send it over email, and then also attach a dashboard link. And the third thing is the steps. So what exactly step this alert, this workflow needs to perform? So we're building our final data set. So it automatically runs the whole flow. And then once we have that, we are converting that HTML file into a variable, which is set up as an email template, in the notification. So that concludes the solution, like, for how this has been built. There are, other than there are some supportive, application that you can also help developer while building this tool. For example, in Dataiku, we have this, Trace Explorer. It's a visually built in plug in, and it automatically scans all your LLM request. For every prompt, you can actually see, what level of reasoning that has gone through, what how many LLM calls has been made, how many tool calls has been made, did we made any call to a guardrail, LLM to scan the response or the query before generating the output. So you can follow all these. This can help in multiple ways. First of all, you can see the reasoning happening in this right way that, you are expecting. Apart from that, you can also see which steps is actually taking more time slowing down the response of, the agent, how much cost is it getting for each of these steps to run, because it's very easy to calculate a a cost for a particular one LLM call. But when it comes to tool calling, it becomes very complex because you cannot predict how many calls the agent will make based on the prompt that user has provided. So these things will help you to do this process iteratively and identify what agents to use, what tools to use, what are the key points we need to further optimize, and then before coming up with a solution. The last thing I have is as I mentioned, we are using, NVIDIA name models in Dataiku. You have both the option. You can either connect to NVIDIA and host if you already have host of these models. You can connect to it through LLM mesh, and take advantage of all the LLM mesh feature as well as the scalability of, NVIDIA and the models. Or if you choose to deploy all these model in Dataiku, you also have an option. You can just provide your authentication key, choose the model, and Dataiku will deploy this model locally whether it's on prem or on cloud. It can deploy and generate an endpoint which can be used to again feed into the tile to a mesh, and then it automatically enables to all the visual and code based, LLM recipes. So yeah. So that's all on my side from the demo. I hope, you guys have enjoyed it. I'll hand it back over to John. Thank you so much, Shashank. So I hope as folks have seen that we've delivered on our promise of very concretely tying everything we discussed there into an actual realizable project that could be achieved today within your firm. So as you can see here, we are tackling both problems of scale, but also problems of control and and allowing that level of control that you need to scale effectively as you build out your ambitions. Malcolm, anything you'd like to add, to what was just shown here? Yep. Shashank, great job, by the way. And, John, thanks for all of that. I I would just say that, I hope everyone out there is realizing that this platform, what we've just demonstrated, these NIMs are Lego blocks that can quickly be redeployed for other blueprint automation. So this was just to give you the art of the possible. We we are showing you an AML investigator, but this could be any workflow. And, very excited to be here with you, John and Shashank. Absolutely. And I think that's a great setup as well to be talking about how this is one of could ultimately be an infinite number of different blueprints. I have both blueprints deployed by, NVIDIA and Dataiku, and, of course, work that the firms themselves can build internally. Remember, everything you saw here is achievable with your own effort internally or with external partners if that's how, you want to move forward. So as Malcolm said, one example of all of the possibilities, open to you. So thank you so much for taking time with us. There'll be contact information provided as follow-up. So if you'd like to reach out to myself, to Malcolm, to Shashank, or anyone else at, NVIDIA, you'll have that information. And I'll just emphasize again that what you saw here is a realistic demonstration of how you can achieve this work today. So I'd really encourage you to move forward with confidence, around these challenges and to know with Dataiku, NVIDIA, and your own capabilities combined, you will be able to succeed in this space. Thank you all so much. Malcolm, thank you so much for your time. Shashank, thank you so much for the excellent demonstration. Take care. Thanks, John. Thanks, John.