Transcript for "Agent Evals 101":
Hi, everyone. Thank you for joining us today for this session on agent evals There's a poll question up, so feel free to let us know where you are in your agent eval journey as you come in. I'm Jezell, the education marketing manager here at LangChain, and I'm here with Divya, one of our amazing education engineers who will be walking you through today's workshop. So you're in really good hands. I'll let Divya take it from here. Alright. Thank you so much, Jezell, for that intro. I'm gonna go ahead and share my screen, and we can get started. Alright. Okay. Alright. So echoing Jezell welcome message, welcome to agent evals one zero one. My name is Divya, and I'm an education engineer here at LangChain. If you have any questions during the session, feel free to use the q and a tab or drop them in the chat. We have some excellent mods behind the scenes who will be monitoring live, and links to everything I show today are in the docs tab on the right, including the course repo or the demo repo we'll use today. Alright. We released a poll question at the top of the hour. Looks like most people are slowly entering in their answers. Beginners are very welcome here, and wherever you are in your journey, I think you'll get something out of the session and hopefully learn something new about evals. Alright. So I'm gonna quickly go over the agenda for today's session. By the end of the session, you should know what agent evals are and how to start evaluating your own agent. The first part is learning the fundamentals, why agents are hard to test, what an eval is, agent traces, offline and online evals, the different kinds of evaluators, and a quick look at benchmarks. Then I'll switch to a live demo and finish with a take home assignment. Everything in the demo works on the free LangChain developer tier, so you can reproduce this at home later. Alright. So why are agents hard to test? Well, for one, they're non deterministic. What does that mean? Well, if you ask an agent the same question twice, you'll very likely get two different answers. Sometimes, it's just the wording. Sometimes, it's the answer itself. With agents, there's an infinite input space and output space. Also, there's often many right answers an agent can have. Say you ask it to summarize a document. A bullet list, a paragraph, and a chart could all be correct. There's no single string to compare against. And finally, agents choose their own steps. With agents, you don't write the control flow. The model does. It might call a tool it doesn't need, loop on the same tool, or skip a tool and guess. In this example on the screen, the agent is skipping a lookup and answering anyway. A unit test checks for one exact answer. Say we wanted to evaluate an agent's output for summarizing some notes. In this unit test, we're asserting that the output is exactly the phrase, the meeting moved to Friday. Run one of the agent happens to produce that exact phrasing and passes. But run two and run three say the same thing in different words. Both are correct and yet both failed the assertion test. So we need checks that score what matters about the answer, not necessarily an exact string. That's what agent evals are for. So what is an eval? An eval scores a run of your agent against a metric that you define. That metric could be something like agent helpfulness or the agent using the correct tools. It tells you which runs are failing, how often, and whether a change to your agent fixed the problem or broke something else. There are three core pieces to Evals, datasets, evaluators, and experiments, and we're gonna talk about all three. Let's start with datasets. A dataset is a set of examples. Each example has an input and sometimes a reference output. The input is generally a realistic input your agent would receive. The optional reference output can be anything from a status check to a complete written response. I'll be using a support agent as an example for the next few slides. In this dataset, for the support agent, users ask about their orders and the reference output is the order status. Here we can say, we can see the status is shipped. An evaluator is the check that scores an agent run. It can evaluate the run's input, the output, and anything in between, like tool calls or model calls. Here, we have two evaluators. With the input, where's order number 4821, one eval confirms the agent called the lookup order tool. Another eval reads the reply and decides whether it answered the question. We'll come back to some of the different kinds of evaluators in a bit. Now that last component, experiments. An experiment is a batch of agent runs over a dataset plus the evaluator scores. You get a score per example in the dataset for each evaluator you use. Based on the evaluator scores, it looks like the agent called lookup order 90% of the time and answered the question 75% of the time. After seeing these results in an experiment for your agent's performance, your next step would be adding a fix. That could be changing your agent's system prompt, for example. Then you would run another experiment with the same examples and the same evaluators and compare your agent's performance after the fix with its performance before the fix. So to recap, this is how everything comes together. The dataset sends inputs to the agent. The agent's outputs go to the evaluators. The evaluator scores make up the experiment. Alright. Now before you actually start writing any eval... Any evals, I would recommend that you read through some agent traces. A trace is a record of one run of your agent. It shows the input, the output, and every model and tool call in between. Reading even 10 agent traces tells you what real requests look like for your agent, and it may even tell you where the agent is going wrong. You can also turn your traces into dataset examples, and we're actually gonna do that in the demo later in the session. Here's an example of an agent trace from the support agent. In Linksmith, you can view this as a run tree in the UI. For this trace, a user asks, where's order number 4821? The first thing we can see in the trace is the input. In a real app, the input can be more than just the user's message, like a user ID or data that the app passes along. We can also see model calls in a trace. Each model call shows up as its own step in the run tree. In this example, the first call decides to use lookup order with the order ID 4821. The second call writes the reply. In the trace, you can click into these calls to see exactly what the model was given and what it returned. Here's the tool call. We can see what went into the tool and what came back. A lot of bugs tend to live in tool calls. That's a bit of foreshadowing for our demo. The output is the final reply that the user saw. In this case, it's your order shipped and arrives Friday. Traces in LangChain also show you data related to latency and token count. Every step shows how long it took and how many tokens it used. That's useful for cost and speed, and you can evaluate on those factors as well. Alright. Now we've talked about a lot of components to eval so far. So if you're feeling a little overwhelmed and you're not really sure where to start, here's one way you can do it. First, pick one behavior. Don't try to evaluate everything with your agent at once. Pick one thing that matters to your users or a feature you've already seen break. Next, build a small set of examples from your traces. 10 to 20 is enough when you're getting started. Real traces are better than examples you make up because they typically contain real user inputs. Next, run a baseline experiment. This is an assessment of your agent's performance before writing a fix. Then fix the agent and rerun the experiment. Check that the fix worked and that nothing else broke. And finally, as you move towards production, add new production failures as examples to your dataset. Your dataset grows from real problems over time. There are two times to check your agent. Before you ship, let's say you changed a prompt, a model, or a tool. You wanna know if your agent improved and if anything broke. We call those offline evals. After you ship, real users ask things you never tested. Now you want to know if the agent's quality is holding up in production. Those are online evals. Offline evals run your agent on a dataset you control. You pick the inputs and you can optionally add reference outputs. Here, we ran two versions of the support agent on the same 20 examples. V two looks better overall based on the evaluator's average score. We can see it went from 75% with v one to 90% with v two. But one example that passed in v one now fails. It's important to keep an eye out for these types of regressions when you're testing and improving your agent so that you don't overwrite good behavior. Online evals score real traces as they come in. Live requests don't come with a reference output, so online evaluators have to grade a run on their own. This eval checks whether each reply answered the question. The two fails are an address change and a missing package. Those are inputs our dataset never covered, and so we were only able to flag this behavior in production. That's why online evaluation is so important. When an online eval flags a bad run, you should add that trace to your offline dataset. The next version gets tested against it before it ships. Offline and online evals feed into each other through that loop. Now zooming out a little bit, this is how we think about building agents. We build, test, deploy, monitor, and build again. Offline evals are covered in the test stage of this loop. Online evals are covered in the monitor stage. Evals are what drive this agent development life cycle. Now there are three kinds of evaluators you'll use most, whether you're working with offline or online evaluation. Use code based when there's a deterministic check you can perform. For example, did the agent call lookup order with the correct order ID? Code checks are fast, cheap, and give the same answer every time. Use an LLM as judge when it's a judgment call that needs a rubric. For example, was the reply polite and on brand? An LLM will read the output and grade it. Code based evals and LLM judges are what you'll encounter most often. There's also a type of evaluator called pairwise. Pairwise evaluators are used when better only exists in comparison. For example, which of two replies would a customer prefer? This helps when you're choosing between two prompts or two models. There's also human review. It's slow, but it's the best signal for quality and it's how you ensure that your other evaluators are aligned to your judgment. An evaluator doesn't have to look only at the final reply. Let's go back to the support agent. The evaluator can evaluate one step, like was lookup order called with the right order ID. It can also evaluate the final response, like does the reply give the right delivery date? It can evaluate the trajectory like did it look up the order before answering? And it can evaluate the end state. If the user asked for a refund, does the refund exist in the system? Most people start with the final response. But once you have traces, you can check any of these components. Now if you're unsure when to use a code eval or an LLM judge, here's a general rule of thumb. If you can check it with an if statement, use code. If it takes reading and judgment, use an LLM judge. Start with code wherever you can. Alright. Now Jezell will be launching a poll. This is a multi select poll for which types of evaluators you check with code over an LLM judge. Take a look at the poll. It should be on the right hand side of your screen. Take, take a look and give your answers. We'll do maybe about sixty seconds. Alright. Responses are coming in. That's great. Remember, it's multi select, so you could have several correct answers on this one. Alright. Great job, everyone. Alright. So we're contrasting code and LLM judge evaluators and when it's better to use which. Do another thirty seconds. Alright. Looking good. Alright. Jezell, could you give me a quick update on how the poll's doing? Looking okay. Alright. Cool. Alright. So let's talk about the answer to this, to this question. Alright. So the question says, which of these scenarios would you use a code check over an LLM check? The reply includes the order number, the agent called lookup order before replying, the reply is polite, and the reply answers what the user asked. The correct answer for this one is the first two options. The reply includes the order number and the agent called lookup order before replying. Those are both deterministic checks that you can do with a code eval. The other two options, you probably use an LLM judge. Alright. Great job, everyone. Alright. Now can you trust an LLM judge? Fair question. Maybe the only AI you really trust is Wall E. But there are a few ways we can make our LLM judges more reliable and predictable in practice. A risk of LLM judges is that your judge can drift from your team's judgment. Three things help rein it in. First, ask narrow questions. Does the reply give the delivery date works better than is this a good reply? Second, use a binary pass fail score. It's hard for a judge or a person to distinguish between a seven and an eight on a one to 10 scale the same way every time. Third, align your judge against human feedback. Have your team grade a sample, then adjust the judge until it agrees with your judgment. Now the last topic before we jump into the demo. Agents are becoming increasingly capable and complicated. Agents now run code, edit files, and call services. One task can take dozens of tool calls. A coding agent could say, I fixed a bug, but you only know for sure once you run the tests. Here, two of three tests failed. For agents like this, you have to check the result of the work, not just a string output. That's what a benchmark does. It's an offline eval made of longer tasks. Each task has instructions like fix the login test. It has a sandbox, which is an isolated container with the repo, files, and tools the agent needs. And it has a test script that checks the state after the run. Every run gets a fresh sandbox, so one run can't affect the next. Most agent benchmarks like Terminal Bench work this way. Harbor is an open source harness for running benchmarks. Terminal Bench runs on it. It builds a sandbox for each task and runs your agent inside it. With the Linksmith integration, each run shows up as an experiment with every trace next to its score. You can also turn failures from your own traces into harbor tasks. We use it at LangChain to improve deep agents and Linksmith engine. We won't cover benchmarks in the demo, but it's important to know about benchmarking since it's widely used for agent evals today. Okay. That's enough context for now. So let's look at a real agent. We're going to work through a demo with DinnerBot. The agent we'll be working through for this demo is called DinnerBot. When you ask DinnerBot, what can I make for dinner? It suggests a recipe. DinnerBot is part of an app that keeps track of your kitchen. The app sends your saved kitchen along with every request. It has one tool, GetPantry, which returns your kitchen plus a built in staples list. It replies with a dish, an ingredient list, and instructions. Now together, we'll walk through some of the code at a higher level so we can get a little bit more familiar with this agent. Alright. I'm going to stop my share and then share some code. Alright. Alright. Hopefully, everyone can see this. This is the implementation of our dinner bot. Now in this file, this is agent dot p y and it's the main implementation. We can see the system prompt for our agent up at the top. We can also see how ingredients and recipes are formatted. An ingredient has a name and a quantity, and a recipe has a dish with ingredients and steps or instructions. If we scroll down a little bit, we can see our tool, get pantry. This is what returns the user's kitchen. Here, we actually create our agent and we can see it has the get pantry tool and that system prompt from earlier. If we scroll down, this is how we actually run our agent. Here, we're using an input, what can I make for dinner with the kitchen pasta, eggs, Parmesan, and spinach? It calls the ask function, which is defined over here. And we also have a traceable decorator so we can see this in the trace in LangChain. Now let's try an actual sample run of our agent. We're going to use this input that we see right here. Alright. Now I'm going to drag this into the screen here. This is how we run our agent. I'm gonna go ahead and run it. And as it runs, it's going to take in that user message. It's going to take in the kitchen and try to suggest a recipe that we can use. And here we see the recipes. It suggested creamy spinach pasta. We can see the ingredients it listed, pasta, spinach, Parmesan, garlic, butter, milk, salt, pepper, and oil. And we see the steps. Alright. So think about this output. I'm gonna switch back over to the slides really quick. Okay. And while I'm doing that, Jezell, if you wanna launch the next poll. Thank you. Let's see. Now we saw the agent implementation and we saw an example input into our agent. Now I wanted... I want you to think about, what this actually looks like, if we saw an output that looks like this. Whisk two eggs with Parmesan, toss with hot pasta, and finish with butter and garlic. The user's kitchen is pasta, eggs, Parmesan, and spinach. Just based on this output and what you know about the ingredients in the user's kitchen, would you, as an evaluator, say this is a pass or a fail? You can submit your answer in the poll on the right hand side of the screen. I'll give you about sixty seconds. Alright. So think about what this means. Keep in mind, the ingredients that are in the user's kitchen and the ingredients listed in the output. Alright. Answers are coming in. Alright. Great job, everyone. Alright. I'm seeing some messages in the chat as well. It looks like you guys are on the same page as me. I would call this a fail. The user doesn't have butter or garlic in their kitchen. Even though the reply reads well, it makes the mistake easy to miss. And if you were skimming through a 100 of these traces, you'd probably let it through. So let's find out why this is happening and how often. Alright. Gonna stop the share. And we're gonna open up Linksys. Alright. Alright. Thank you for bearing with me there. Alright. So if we go over to LangChain, we can see that that agent run we just did created a trace in our LangChain project dinner bot example. Now I'm gonna go ahead and click on this trace so we can get a little bit more context into what's happening. If you remember the run tree we saw earlier in the presentation, this looks really similar. We can see the input and output as well as model and tool calls in between. Now here we can see the input. What can I make for dinner is the message and the kitchen has pasta, eggs, Parmesan, and spinach? The output is a recipe. It's listed here as well as the ingredients. Now we said that this is a fail because it's including some ingredients that the user doesn't have like butter and milk and garlic. Those all look incorrect to me. Now if we actually click on the tool, we can get a little bit more context into what's happening. In the tool output, we can see the kitchen, pasta, eggs, Parmesan, and spinach, and we can see the staples list. Now every person has some staples in their kitchen. We could probably safely assume that everyone has salt, pepper, oil, water most likely. But we also see butter, garlic, onion, soy sauce and milk listed as staples. That's definitely a bug. Now, we can see that based on what kitchen and staples that the agent got, it actually performed correctly. So the bug is something to do with the tool. Now, I also have another tracing project here and this one has several traces, 25 to be specific. We mentioned earlier in the presentation that we can take traces and turn them into a dataset. That's what we're gonna do with these traces right here. When we add all of these traces to a dataset, and I have one already loaded for us, we have something called dinnerbot dataset. In this dataset, we can see all the inputs, which are the same inputs that were in the traces in the previous tracing project. We can also see reference outputs. These got added automatically and these reference outputs are the real outputs we saw from the traces. It's up to us whether we wanna actually use these in practice when we're running an evaluator. You might be wondering, if the trace already shows the bug, why run an eval? Well, there are two reasons. For one, it tells you how often this happens. Is it one kitchen in 20 or is it most of the kitchens? And after we fix the bug, we can compare the agent's performance to the baseline to check whether the fix really worked. Alright. So now that we have our dataset, let's look at the evaluator that we're going to use. I'm going to switch back over to the code. And first, we'll take a look at kitchen evaluator. Now this evaluator looks at what's in the kitchen and the assumed staples and it looks like it looks at the output's ingredients. If there are extra ingredients, then we're gonna score this as a zero. This evaluator is used in our eval script which is over here. Here we use the LangChain SDK function evaluate and we pass it the evaluator that we'd like to use. Now using this evaluator, we can run an experiment in LangChain and those are logged right here. Now if we click on the results of the experiment, we can see all of the scores using our kitchen evaluator. For most of these recipes, it looks like it scored zero. It also includes a comment if you hover over it. For example, in this recipe, it says that the reasoning for scoring zero is that it used garlic and onion which are not in the kitchen. We can click on this to confirm. In the output, it used garlic and onion even though it's not in the input ingredients. Oh, over here. In the input ingredients. Alright. Our next step would be fixing this bug. We want to make sure that the staples list isn't too generous. If I go back over to the code and look at the agent implementation, I'm gonna scroll up a little bit, and we can see where this bug is. The staples list has way too many ingredients. So my next step would be fixing this and then jumping back over to LangChain to run another experiment with that fix implemented. We can see that all of these recipes now pass after fixing staples. An additional feature is that you can directly compare the experiments. If I select both and hit compare, we can see tons of information comparing feedback scores, latency, token count, and cost. We can also directly compare the outputs. If I click on this example and click on outputs, we can see this one used these ingredients while this one used too many. Alright. Now I'm gonna jump back over to the slides. Alright. Alright. So that code eval that we just wrote performs a deterministic check for the ingredients that are used in the outputted recipe. What it can't tell us is the quality of the recipe. Here's a reply that passes the code check. Every ingredient in the recipe is in the kitchen, But is it a good recipe? Many things could go wrong in a recipe whether it's ingredients, bad instructions, or inaccurate cook times. That takes reading and judgment so we use an LLM judge. Earlier, I mentioned that an LLM judge should always align with your team's judgment. So before trusting a judge blindly, we need some human labels. Our team reviewed these eight recipes that DinnerBot created and scored the recipe as either good or bad considering factors like food safety and whether the recipe is a real dish. In practice, you'd collect these labels in an annotation queue. This is our LLM judge prompt. It simply asks, is this a good recipe? The prompt never defines what good means, so the judge chooses its own definition. Looking at a LangChain experiment with the broad evaluator, the human label and the evaluator score are not in full agreement. Specifically, we can see two examples where the LLM judge scored one while the human scored zero. We can investigate this further by hovering over some of those examples. It looks like our team failed this recipe because the same knife and board are used for raw chicken and the salad. The judge passed it because it sounds like a nice balanced meal. The problem is that the broad judge never checked for safety. When our team evaluated the recipes, we didn't just ask, is it good? We had specific factors in mind like food safety and realistic dishes. So let's give the judge those same two questions and see what changes. The first narrow judge asks one question. Can someone follow these steps without risking food poisoning? It spells out what fails like undercooked meat and contamination. And the last line tells it what not to judge. That keeps the question narrow. The second judge asks, is this a real dinner a home cook would make? Again, it says what passes. These are two short examples at the... There are two short examples at the bottom. They're different from the recipes in the dataset, so the judge isn't just overfitting and matching examples. Here, we can see the example say garlic butter pasta written casually should still be a pass, but a bowl of plain ketchup should probably be a fail. This is the same dataset with the same outputted recipes. The only difference is that now we have two narrow judges instead of one broad judge. That same lemon chicken is now realistic but not food safe, which matches our team's judgment exactly. Here's the summary of what we just saw on that slide. The broad judge agreed with our team on six of eight examples. It passed the lemon chicken and the turkey meatballs. Both have an unsafe step that the judge missed. The narrow judges, on the other hand, agreed on all 16 labels. So what changed? Two things changed between these experiments. We split one broad question into two narrow ones. The results are also more useful now. Some recipes only fail safety and some only fail realism, so we know what to fix. For example, we could add food safety guidance to dinner bot's prompt. As we've seen, it's important to align your judges with your team's judgment. If Gordon Ramsay were on your team, his idea of a good recipe would probably be stricter than the LLMs. Your judge should match your team. Lanesmith also has a dedicated workflow for this called align evals. Now for the last part of the demo. Offline evaluators can run online too as long as they don't need a reference output or a human label. Our food safety judge only needs the recipe so it can score real traces as they come in. The kitchen code check could run online too since the kitchen is in the inputs. Now, once again, I'm gonna switch over to LangChain. Alright. Change my screen. Alright. So here we are back in our tracing project. Now those same offline evaluators we created earlier, we can use them as online evaluators. If I go over to the online evaluators tab, you can see that there are plenty of LLM judge templates that you can work from. You could create it from scratch or you could use a decision model evaluator or a code evaluator from scratch as well. And there are plenty of templates like security, safety, quality, conversation, trajectory, image evaluations, and more. There's plenty of options to choose from. In our case, we're demoing how you can reuse an offline evaluator for online evaluation. If I hit plus evaluator, I can see down here, I already have three existing evaluators. These are the ones we used in the demo part we just saw, Realistic Judge, Food Safe Judge, and that Broad Good Judge, which we probably wouldn't wanna reuse. I can hit food safe judge, and we can see how this is actually configured. Scrolling down a little bit, this is the judge prompt. We saw this in the slides. It's the same thing, just in the links with UI. We also configure the feedback. In this case, we're using a Boolean response format since we're doing pass fail. We can also see a sample input and a sample output. I could create this evaluator. And now this is attached as an online eval for my project. So every time a new trace comes in, it'll grade it automatically and assign a score on whether that, trace or whether that recipe is food safe. Alright. Now I will switch back one last time to the slides. Okay. Alright. There we go. Alright. So now we've talked about a lot of different evaluator concepts and we've gotten to see them, ourselves. Alright. So now it's your turn. The take home assignment uses a new agent in the same kitchen. Recipe remixer takes a recipe and a request like make this vegetarian. The agent uses a get substitute tool to find swaps like chicken to tofu. Then a rewrite step updates the instructions. When you run it, you might notice something's off in the result. Your job is to find out what, build a dataset, write evaluators that catch it, fix it, and show the fix worked. There are plenty of hints in the read me of the repo and an answer key in the recipe remixer folder. You can find all of this in the repo in the docs tab on the right of your screen. Alright. Now where to go next? So like I said, check out the docs tab on the right. You can see the repo and plenty of other resources. We also have the LangSmith agent evaluation docs listed on the right. We have a free course on LangSmith essentials. And if you fill out a form, you can get a $100 in free LangSmith credits. So definitely check that out. Also, keep an eye out for a full agent Evals course on Linkchain Academy in early December. Alright. Thank you so much for joining. I hope you enjoyed the session, and good luck on the take home assignment.,