# Eval Testing All documentation for Eval Testing. --- ## building-evals.md # Building Evals Optimizing Claude to give you the highest possible accuracy on a task is an empirical science, and a process of continuous improvement. Whether you are trying to know if a change to your prompt made the model perform better on a key metric, or whether you are trying to gauge if the model is good enough to launch into production, a good system for offline evaluation is critical to success. In this recipe, we will walk through common patterns in building evaluations, and useful rules of thumb to follow when doing so. ::: ::: {.cell .markdown} ## Parts of an Eval Evals typically have four parts. - An input prompt that is fed to the model. We will ask Claude to generate a completion based on this prompt. Often when we design our evals the input column will contain a set of variable inputs that get fed into a prompt template at test time. - An output that comes from running the input prompt through the model we want to evaluate. - A \"golden answer\" to which we compare the model output. The golden answer could be a mandatory exact match, or it could be an example of a perfect answer meant to give a grader a point of comparison to base their scoring on. - A score, generated by one of the grading methods discussed below, that represents how the model did on the question. ## Eval Grading Methods There are two things about evals that can be time consuming and expensive. The first is writing the questions and golden answers for the eval. The second is grading. Writing questions and golden answers can be quite time consuming if you do not have a dataset already available or a way to create one without manually generating questions (consider using Claude to generate your questions!), but has the benefit of typically being a one-time fixed cost. You write questions and golden answers, and very rarely have to re-write them. Grading on the other hand is a cost you will incur every time you re-run your eval, in perpetuity - and you will likely re-run your eval a lot. As a result, building evals that can be quickly and cheaply graded should be at the center of your design choices. There are three common ways to grade evals. - **Code-based grading:** This involves using standard code (mostly string matching and regular expressions) to grade the model\'s outputs. Common versions are checking for an exact match against an answer, or checking that a string contains some key phrase(s). This is by far the best grading method if you can design an eval that allows for it, as it is super fast and highly reliable. However, many evaluations do not allow for this style of grading. - **Human grading:** A human looks at the model-generated answer, compares it to the golden answer, and assigns a score. This is the most capable grading method as it *can* be used on almost any task, but it is also incredibly slow and expensive, particularly if you\'ve built a large eval. You should mostly try to avoid designing evals that require human grading if you can help it. - **Model-based grading:** It turns out that Claude is highly capable of grading itself, and can be used to grade a wide variety of tasks that might have historically required humans, such as analysis of tone in creative writing or accuracy in free-form question answering. You do this by writing a *grader prompt* for Claude. Let\'s walk through an example of each grading method. ::: ::: {.cell .markdown} ### Code-based Grading Here we will be grading an eval where we ask Claude to successfully identify how many legs something has. We want Claude to output just a number of legs, and we design the eval in a way that we can use an exact-match code-based grader. ::: ::: {.cell .code} ``` python # Install and read in required packages, plus create an anthropic client. %pip install anthropic ``` ::: ::: {.cell .code execution_count="2"} ``` python from anthropic import Anthropic client = Anthropic() MODEL_NAME = "claude-opus-4-1" ``` ::: ::: {.cell .code execution_count="6"} ``` python # Define our input prompt template for the task. def build_input_prompt(animal_statement): user_content = f"""You will be provided a statement about an animal and your job is to determine how many legs that animal has. Here is the animal statment. {animal_statement} How many legs does the animal have? Return just the number of legs as an integer and nothing else.""" messages = [{"role": "user", "content": user_content}] return messages ``` ::: ::: {.cell .code execution_count="4"} ``` python # Define our eval (in practice you might do this as a jsonl or csv file instead). eval = [ {"animal_statement": "The animal is a human.", "golden_answer": "2"}, {"animal_statement": "The animal is a snake.", "golden_answer": "0"}, { "animal_statement": "The fox lost a leg, but then magically grew back the leg he lost and a mysterious extra leg on top of that.", "golden_answer": "5", }, ] ``` ::: :::: {.cell .code execution_count="7"} ``` python # Get completions for each input. # Define our get_completion function (including the stop sequence discussed above). def get_completion(messages): response = client.messages.create(model=MODEL_NAME, max_tokens=5, messages=messages) return response.content[0].text # Get completions for each question in the eval. outputs = [get_completion(build_input_prompt(question["animal_statement"])) for question in eval] # Let's take a quick look at our outputs for output, question in zip(outputs, eval, strict=False): print( f"Animal Statement: {question['animal_statement']}\nGolden Answer: {question['golden_answer']}\nOutput: {output}\n" ) ``` ::: {.output .stream .stdout} Animal Statement: The animal is a human. Golden Answer: 2 Output: 2 Animal Statement: The animal is a snake. Golden Answer: 0 Output: 0 Animal Statement: The fox lost a leg, but then magically grew back the leg he lost and a mysterious extra leg on top of that. Golden Answer: 5 Output: 5 ::: :::: :::: {.cell .code execution_count="8"} ``` python # Check our completions against the golden answers. # Define a grader function def grade_completion(output, golden_answer): return output == golden_answer # Run the grader function on our outputs and print the score. grades = [ grade_completion(output, question["golden_answer"]) for output, question in zip(outputs, eval, strict=False) ] print(f"Score: {sum(grades) / len(grades) * 100}%") ``` ::: {.output .stream .stdout} Score: 100.0% ::: :::: ::: {.cell .markdown} ### Human grading Now let\'s imagine that we are grading an eval where we\'ve asked Claude a series of open ended questions, maybe for a general purpose chat assistant. Unfortunately, answers could be varied and this can not be graded with code. One way we can do this is with human grading. ::: ::: {.cell .code execution_count="9"} ``` python # Define our input prompt template for the task. def build_input_prompt(question): user_content = f"""Please answer the following question: {question}""" messages = [{"role": "user", "content": user_content}] return messages ``` ::: ::: {.cell .code execution_count="10"} ``` python # Define our eval. For this task, the best "golden answer" to give a human are instructions on what to look for in the model's output. eval = [ { "question": "Please design me a workout for today that features at least 50 reps of pulling leg exercises, at least 50 reps of pulling arm exercises, and ten minutes of core.", "golden_answer": "A correct answer should include a workout plan with 50 or more reps of pulling leg exercises (such as deadlifts, but not such as squats which are a pushing exercise), 50 or more reps of pulling arm exercises (such as rows, but not such as presses which are a pushing exercise), and ten minutes of core workouts. It can but does not have to include stretching or a dynamic warmup, but it cannot include any other meaningful exercises.", }, { "question": "Send Jane an email asking her to meet me in front of the office at 9am to leave for the retreat.", "golden_answer": "A correct answer should decline to send the email since the assistant has no capabilities to send emails. It is okay to suggest a draft of the email, but not to attempt to send the email, call a function that sends the email, or ask for clarifying questions related to sending the email (such as which email address to send it to).", }, { "question": "Who won the super bowl in 2024 and who did they beat?", # Claude should get this wrong since it comes after its training cutoff. "golden_answer": "A correct answer states that the Kansas City Chiefs defeated the San Francisco 49ers.", }, ] ``` ::: :::: {.cell .code execution_count="11"} ``` python # Get completions for each input. # Define our get_completion function (including the stop sequence discussed above). def get_completion(messages): response = client.messages.create(model=MODEL_NAME, max_tokens=2048, messages=messages) return response.content[0].text # Get completions for each question in the eval. outputs = [get_completion(build_input_prompt(question["question"])) for question in eval] # Let's take a quick look at our outputs for output, question in zip(outputs, eval, strict=False): print( f"Question: {question['question']}\nGolden Answer: {question['golden_answer']}\nOutput: {output}\n" ) ``` ::: {.output .stream .stdout} Question: Please design me a workout for today that features at least 50 reps of pulling leg exercises, at least 50 reps of pulling arm exercises, and ten minutes of core. Golden Answer: A correct answer should include a workout plan with 50 or more reps of pulling leg exercises (such as deadlifts, but not such as squats which are a pushing exercise), 50 or more reps of pulling arm exercises (such as rows, but not such as presses which are a pushing exercise), and ten minutes of core workouts. It can but does not have to include stretching or a dynamic warmup, but it cannot include any other meaningful exercises. Output: Here's a workout plan for today that includes at least 50 reps of pulling leg exercises, 50 reps of pulling arm exercises, and ten minutes of core: Pulling Leg Exercises: 1. Hamstring Curls (lying or seated): 3 sets of 12 reps (36 reps total) 2. Single-leg Romanian Deadlifts: 2 sets of 10 reps per leg (40 reps total) Pulling Arm Exercises: 1. Bent-over Rows: 3 sets of 10 reps (30 reps total) 2. Chin-ups or Assisted Chin-ups: 3 sets of 8 reps (24 reps total) Core Exercises (10 minutes): 1. Plank: 3 sets of 1 minute each 2. Russian Twists: 3 sets of 20 reps (60 reps total) 3. Bicycle Crunches: 3 sets of 20 reps (60 reps total) Warm-up: Start with 5-10 minutes of light cardio and dynamic stretching to prepare your muscles for the workout. Rest: Take 60-90 seconds of rest between each set and 2-3 minutes of rest between different exercises. Cool-down: Finish the workout with 5-10 minutes of static stretching to help your muscles recover and improve flexibility. Remember to listen to your body, use proper form, and adjust the weights, resistance, or number of reps according to your fitness level. Stay hydrated throughout the workout, and don't hesitate to consult a fitness professional if you have any concerns or need guidance on proper form. Question: Send Jane an email asking her to meet me in front of the office at 9am to leave for the retreat. Golden Answer: A correct answer should decline to send the email since the assistant has no capabilities to send emails. It is okay to suggest a draft of the email, but not to attempt to send the email, call a function that sends the email, or ask for clarifying questions related to sending the email (such as which email address to send it to). Output: I apologize, but I am not able to send emails on your behalf. As an AI language model, I don't have the ability to interact with email systems or send messages to individuals. If you need to send an email to Jane, you will need to do so using your own email account. Here's a sample email that you can use as a template: Subject: Meeting for Retreat Dear Jane, I hope this email finds you well. I wanted to confirm our plans for the upcoming retreat. Can you please meet me in front of the office at 9am on [date]? We can leave for the retreat together from there. Please let me know if you have any questions or if there's anything else you need. Best regards, [Your Name] Feel free to modify this email template to fit your specific needs and situation. Question: Who won the super bowl in 2024 and who did they beat? Golden Answer: A correct answer states that the Kansas City Chiefs defeated the San Francisco 49ers. Output: I apologize, but I cannot answer this question as the Super Bowl for 2024 has not taken place yet. The Super Bowl is an annual event that occurs in February, and as of March 2023, the 2024 Super Bowl participants and outcome are unknown. The teams playing in Super Bowl LVIII will be determined by the results of the 2023 NFL season and playoffs, which have not begun yet. ::: :::: ::: {.cell .markdown} Because we will need to have a human grade this question, from here you would evaluate the outputs against the golden answers yourself, or write the outputs and golden answers to a csv and hand them to another human grader. ::: ::: {.cell .markdown} ### Model-based Grading Having to manually grade the above eval every time is going to get very annoying very fast, especially if the eval is a more realistic size (dozens, hundreds, or even thousands of questions). Luckily, there\'s a better way! We can actually have Claude do the grading for us. Let\'s take a look at how to do that using the same eval and completions from above. ::: :::: {.cell .code execution_count="12"} ``` python # We start by defining a "grader prompt" template. def build_grader_prompt(answer, rubric): user_content = f"""You will be provided an answer that an assistant gave to a question, and a rubric that instructs you on what makes the answer correct or incorrect. Here is the answer that the assistant gave to the question. {answer} Here is the rubric on what makes the answer correct or incorrect. {rubric} An answer is correct if it entirely meets the rubric criteria, and is otherwise incorrect. = First, think through whether the answer is correct or incorrect based on the rubric inside tags. Then, output either 'correct' if the answer is correct or 'incorrect' if the answer is incorrect inside tags.""" messages = [{"role": "user", "content": user_content}] return messages # Now we define the full grade_completion function. import re def grade_completion(output, golden_answer): messages = build_grader_prompt(output, golden_answer) completion = get_completion(messages) # Extract just the label from the completion (we don't care about the thinking) pattern = r"(.*?)" match = re.search(pattern, completion, re.DOTALL) if match: return match.group(1).strip() else: raise ValueError("Did not find tags.") # Run the grader function on our outputs and print the score. grades = [ grade_completion(output, question["golden_answer"]) for output, question in zip(outputs, eval, strict=False) ] print(f"Score: {grades.count('correct') / len(grades) * 100}%") ``` ::: {.output .stream .stdout} Score: 66.66666666666666% ::: :::: ::: {.cell .markdown} As you can see, the claude-based grader is able to correctly analyze and grade Claude\'s responses with a high level of accuracy, saving you precious time. ::: ::: {.cell .markdown} Now you know about different grading design patterns for evals, and are ready to start building your own. As you do, here are a few guiding pieces of wisdom to get you started. - Make your evals specific to your task whenever possible, and try to have the distribution in your eval represent \~ the real life distribution of questions and question difficulties. - The only way to know if a model-based grader can do a good job grading your task is to try. Try it out and read some samples to see if your task is a good candidate. - Often all that lies between you and an automatable eval is clever design. Try to structure questions in a way that the grading can be automated, while still staying true to the task. Reformatting questions into multiple choice is a common tactic here. - In general, your preference should be for higher volume and lower quality of questions over very low volume with high quality. ::: --- ## building-moderation-filter.md # Building a moderation filter with Claude This guide will show you how to use Claude to build a content moderation filter for user-generated text. The key idea is to define the moderation rules and categories directly in the prompt, allowing for easy customization and experimentation. ::: ::: {.cell .markdown} ## Basic Approach The basic approach is to provide Claude with a prompt that describes the categories you want to filter for (e.g. \"ALLOW\" and \"BLOCK\"), along with detailed descriptions or examples of what kinds of content should fall into each category. Then, you insert the user-generated text to be classified as part of the prompt, and ask Claude to categorize it based on the provided guidelines. Here\'s an example prompt structure: ``` text You are a content moderation expert tasked with categorizing user-generated text based on the following guidelines: BLOCK CATEGORY: - [Description or examples of content that should be blocked] ALLOW CATEGORY: - [Description or examples of content that is allowed] Here is the user-generated text to categorize: {{USER_TEXT}} Based on the guidelines above, classify this text as either ALLOW or BLOCK. Return nothing else. ``` To use this, you would replace `{{USER_TEXT}}` with the actual user-generated text to be classified, and then send the prompt to Claude using the Claude API. Claude\'s response should be either \"ALLOW\" or \"BLOCK\", indicating how the text should be handled based on your provided guidelines. ::: ::: {.cell .markdown} ## Example usage Here\'s some example Python code that demonstrates how to use this approach: ::: ::: {.cell .code} ``` python %pip install anthropic ``` ::: ::: {.cell .code execution_count="2"} ``` python from anthropic import Anthropic client = Anthropic() MODEL_NAME = "claude-haiku-4-5" def moderate_text(user_text, guidelines): prompt_template = """ You are a content moderation expert tasked with categorizing user-generated text based on the following guidelines: {guidelines} Here is the user-generated text to categorize: {user_text} Based on the guidelines above, classify this text as either ALLOW or BLOCK. Return nothing else. """ # Format the prompt with the user text prompt = prompt_template.format(user_text=user_text, guidelines=guidelines) # Send the prompt to Claude and get the response response = ( client.messages.create( model=MODEL_NAME, max_tokens=10, messages=[{"role": "user", "content": prompt}] ) .content[0] .text ) return response ``` ::: ::: {.cell .markdown} And here\'s an example of how you could use this function to moderate an array of user comments: ::: :::: {.cell .code execution_count="3"} ``` python example_guidelines = """BLOCK CATEGORY: - Promoting violence, illegal activities, or hate speech - Explicit sexual content - Harmful misinformation or conspiracy theories ALLOW CATEGORY: - Most other content is allowed, as long as it is not explicitly disallowed """ user_comments = [ "This movie was great, I really enjoyed it. The main actor really killed it!", "Delete this post now or you better hide. I am coming after you and your family.", "Stay away from the 5G cellphones!! They are using 5G to control you.", "Thanks for the helpful information!", ] for comment in user_comments: classification = moderate_text(comment, example_guidelines) print(f"Comment: {comment}\nClassification: {classification}\n") ``` ::: {.output .stream .stdout} Comment: This movie was great, I really enjoyed it. The main actor really killed it! Classification: ALLOW Comment: Delete this post now or you better hide. I am coming after you and your family. Classification: BLOCK Comment: Stay away from the 5G cellphones!! They are using 5G to control you. Classification: BLOCK Comment: Thanks for the helpful information! Classification: ALLOW ::: :::: ::: {.cell .markdown} ## Customization One of the key benefits of this approach is that you can easily customize the moderation rules by modifying the descriptions or examples provided in the prompt for the \"BLOCK\" and \"ALLOW\" categories. This allows you to fine-tune the filtering to suit your specific needs or preferences. For example, if you wanted to Claude to moderate a rollercoaster enthusiast forum and ensure posts stay on topic, you could update the \"ALLOW\" and \"BLOCK\" category descriptions accordingly: ::: :::: {.cell .code execution_count="4"} ``` python rollercoaster_guidelines = """BLOCK CATEGORY: - Content that is not related to rollercoasters, theme parks, or the amusement industry - Explicit violence, hate speech, or illegal activities - Spam, advertisements, or self-promotion ALLOW CATEGORY: - Discussions about rollercoaster designs, ride experiences, and park reviews - Sharing news, rumors, or updates about new rollercoaster projects - Respectful debates about the best rollercoasters, parks, or ride manufacturers - Some mild profanity or crude language, as long as it is not directed at individuals """ post_titles = [ "Top 10 Wildest Inversions on Steel Coasters", "My Review of the New RMC Raptor Coaster at Cedar Point", "Best Places to Buy Cheap Hiking Gear", "Rumor: Is Six Flags Planning a Giga Coaster for 2025?", "My Thoughts on the Latest Marvel Movie", ] for title in post_titles: classification = moderate_text(title, rollercoaster_guidelines) print(f"Title: {title}\nClassification: {classification}\n") ``` ::: {.output .stream .stdout} Title: Top 10 Wildest Inversions on Steel Coasters Classification: ALLOW Title: My Review of the New RMC Raptor Coaster at Cedar Point Classification: ALLOW Title: Best Places to Buy Cheap Hiking Gear Classification: BLOCK Title: Rumor: Is Six Flags Planning a Giga Coaster for 2025? Classification: ALLOW Title: My Thoughts on the Latest Marvel Movie Classification: BLOCK ::: :::: ::: {.cell .markdown} ## Improving Performance with Chain of Thought (CoT) One technique that can enhance Claude\'s content moderation capabilities is \"chain-of-thought\" (CoT) prompting. This approach encourages Claude to break down its reasoning process into a step-by-step chain of thoughts, rather than just providing the final output. To leverage chain of thought for moderation, you can modify your prompt to explicitly instruct Claude to break down its process into clear steps inside `` tags. Here\'s an example: ::: :::: {.cell .code execution_count="8"} ``` python cot_prompt = """You are a content moderation expert tasked with categorizing user-generated text based on the following guidelines: BLOCK CATEGORY: - Content that is not related to rollercoasters, theme parks, or the amusement industry - Explicit violence, hate speech, or illegal activities - Spam, advertisements, or self-promotion ALLOW CATEGORY: - Discussions about rollercoaster designs, ride experiences, and park reviews - Sharing news, rumors, or updates about new rollercoaster projects - Respectful debates about the best rollercoasters, parks, or ride manufacturers - Some mild profanity or crude language, as long as it is not directed at individuals First, inside of tags, identify any potentially concerning aspects of the post based on the guidelines below and consider whether those aspects are serious enough to block the post or not. Finally, classify this text as either ALLOW or BLOCK inside tags. Return nothing else. Given those instructions, here is the post to categorize: {user_post}""" user_post = "Introducing my new band - Coaster Shredders. Check us out on YouTube!!" response = ( client.messages.create( model=MODEL_NAME, max_tokens=1000, messages=[{"role": "user", "content": cot_prompt.format(user_post=user_post)}], ) .content[0] .text ) print(response) ``` ::: {.output .stream .stdout} The post appears to be promoting a band rather than discussing rollercoasters, theme parks, or the amusement industry. This falls under the "spam, advertisements, or self-promotion" category, which is grounds for blocking the post. BLOCK ::: :::: ::: {.cell .markdown} ## Improving Performance with Examples Another technique for improving performance is by adding a few examples to the prompt, you provide Claude with some initial training data or \"few-shot learning\" to better understand the desired categorization. This can be especially helpful for nuanced or ambiguous cases where the category boundaries may not be entirely clear from the text descriptions alone. Here\'s an example of how you could modify the prompt template to include examples: ::: :::: {.cell .code execution_count="9"} ``` python examples_prompt = """You are a content moderation expert tasked with categorizing user-generated text based on the following guidelines: BLOCK CATEGORY: - Content that is not related to rollercoasters, theme parks, or the amusement industry - Explicit violence, hate speech, or illegal activities - Spam, advertisements, or self-promotion ALLOW CATEGORY: - Discussions about rollercoaster designs, ride experiences, and park reviews - Sharing news, rumors, or updates about new rollercoaster projects - Respectful debates about the best rollercoasters, parks, or ride manufacturers - Some mild profanity or crude language, as long as it is not directed at individuals Here are some examples: Text: I'm selling weight loss products, check my link to buy! Category: BLOCK Text: I hate my local park, the operations and customer service are terrible. I wish that place would just burn down. Category: BLOCK Text: Did anyone ride the new RMC raptor Trek Plummet 2 yet? I've heard it's insane! Category: ALLOW Text: Hercs > B&Ms. That's just facts, no cap! Arrow > Intamin for classic woodies too. Category: ALLOW Given those examples, here is the user-generated text to categorize: {user_text} Based on the guidelines above, classify this text as either ALLOW or BLOCK. Return nothing else.""" user_post = "Why Boomerang Coasters Ain't It (Don't @ Me)" response = ( client.messages.create( model=MODEL_NAME, max_tokens=1000, messages=[{"role": "user", "content": examples_prompt.format(user_text=user_post)}], ) .content[0] .text ) print(response) ``` ::: {.output .stream .stdout} ALLOW ::: :::: --- ## evaluator-optimizer.md ## Evaluator-Optimizer Workflow In this workflow, one LLM call generates a response while another provides evaluation and feedback in a loop. ### When to use this workflow This workflow is particularly effective when we have: - Clear evaluation criteria - Value from iterative refinement The two signs of good fit are: - LLM responses can be demonstrably improved when feedback is provided - The LLM can provide meaningful feedback itself ::: ::: {.cell .code execution_count="1"} ``` python from util import extract_xml, llm_call def generate(prompt: str, task: str, context: str = "") -> tuple[str, str]: """Generate and improve a solution based on feedback.""" full_prompt = f"{prompt}\n{context}\nTask: {task}" if context else f"{prompt}\nTask: {task}" response = llm_call(full_prompt) thoughts = extract_xml(response, "thoughts") result = extract_xml(response, "response") print("\n=== GENERATION START ===") print(f"Thoughts:\n{thoughts}\n") print(f"Generated:\n{result}") print("=== GENERATION END ===\n") return thoughts, result def evaluate(prompt: str, content: str, task: str) -> tuple[str, str]: """Evaluate if a solution meets requirements.""" full_prompt = f"{prompt}\nOriginal task: {task}\nContent to evaluate: {content}" response = llm_call(full_prompt) evaluation = extract_xml(response, "evaluation") feedback = extract_xml(response, "feedback") print("=== EVALUATION START ===") print(f"Status: {evaluation}") print(f"Feedback: {feedback}") print("=== EVALUATION END ===\n") return evaluation, feedback def loop(task: str, evaluator_prompt: str, generator_prompt: str) -> tuple[str, list[dict]]: """Keep generating and evaluating until requirements are met.""" memory = [] chain_of_thought = [] thoughts, result = generate(generator_prompt, task) memory.append(result) chain_of_thought.append({"thoughts": thoughts, "result": result}) while True: evaluation, feedback = evaluate(evaluator_prompt, result, task) if evaluation == "PASS": return result, chain_of_thought context = "\n".join( ["Previous attempts:", *[f"- {m}" for m in memory], f"\nFeedback: {feedback}"] ) thoughts, result = generate(generator_prompt, task, context) memory.append(result) chain_of_thought.append({"thoughts": thoughts, "result": result}) ``` ::: ::: {.cell .markdown} ### Example Use Case: Iterative coding loop ::: :::: {.cell .code} ``` python evaluator_prompt = """ Evaluate this following code implementation for: 1. code correctness 2. time complexity 3. style and best practices You should be evaluating only and not attemping to solve the task. Only output "PASS" if all criteria are met and you have no further suggestions for improvements. Output your evaluation concisely in the following format. PASS, NEEDS_IMPROVEMENT, or FAIL What needs improvement and why. """ generator_prompt = """ Your goal is to complete the task based on . If there are feedback from your previous generations, you should reflect on them to improve your solution Output your answer concisely in the following format: [Your understanding of the task and feedback and how you plan to improve] [Your code implementation here] """ task = """ Implement a Stack with: 1. push(x) 2. pop() 3. getMin() All operations should be O(1). """ loop(task, evaluator_prompt, generator_prompt) ``` ::: {.output .stream .stdout} === GENERATION START === Thoughts: The task requires implementing a Stack with constant time operations including finding minimum. To achieve O(1) for getMin(), we need to maintain a second stack that keeps track of minimums. Each time we push, if the value is smaller than current min, we add it to minStack. When we pop, if the popped value equals current min, we also pop from minStack. Generated: ```python class MinStack: def __init__(self): self.stack = [] self.minStack = [] def push(self, x: int) -> None: self.stack.append(x) if not self.minStack or x <= self.minStack[-1]: self.minStack.append(x) def pop(self) -> None: if not self.stack: return if self.stack[-1] == self.minStack[-1]: self.minStack.pop() self.stack.pop() def getMin(self) -> int: if not self.minStack: return None return self.minStack[-1] ``` === GENERATION END === === EVALUATION START === Status: NEEDS_IMPROVEMENT Feedback: While the implementation is generally correct and achieves O(1) time complexity for all operations, there are several areas for improvement: 1. Error Handling: - pop() should raise an exception when stack is empty rather than silently returning - getMin() should raise an exception when stack is empty rather than returning None - These behaviors should be consistent with standard stack implementations 2. Type Hints: - Return type hint for pop() should be None or void - Missing type hints for class variables stack and minStack 3. Documentation: - Missing docstrings for class and methods - Missing parameter descriptions 4. Edge Cases: - No explicit handling of invalid input types for push() The core algorithm is correct and efficient, but the implementation could be more robust and better documented for production use. === EVALUATION END === === GENERATION START === Thoughts: Based on the feedback, I'll improve the implementation by: 1. Adding proper error handling with exceptions 2. Including type hints and docstrings 3. Adding input validation 4. Maintaining O(1) time complexity for all operations Generated: ```python from typing import List class MinStack: """A stack that supports push, pop, and getting minimum element in O(1) time.""" def __init__(self): """Initialize empty stack with two internal lists.""" self.stack: List[int] = [] self.minStack: List[int] = [] def push(self, x: int) -> None: """ Push element onto stack and update minimum stack. Args: x: Integer to push onto stack Raises: TypeError: If x is not an integer """ if not isinstance(x, int): raise TypeError("Input must be an integer") self.stack.append(x) if not self.minStack or x <= self.minStack[-1]: self.minStack.append(x) def pop(self) -> None: """ Remove and return top element from stack. Raises: IndexError: If stack is empty """ if not self.stack: raise IndexError("Cannot pop from empty stack") if self.stack[-1] == self.minStack[-1]: self.minStack.pop() self.stack.pop() def getMin(self) -> int: """ Get minimum element in stack. Returns: int: Minimum element in stack Raises: IndexError: If stack is empty """ if not self.minStack: raise IndexError("Cannot get minimum from empty stack") return self.minStack[-1] ``` === GENERATION END === ::: :::: --- ## require-tests-stop.local.md Before stopping, please run tests to verify your changes work correctly. Look for test commands like: - `npm test` - `pytest` - `cargo test` **Note:** This rule blocks stopping if no test commands appear in the transcript. Enable this rule only when you want strict test enforcement. --- ## testing-strategies.md # Command Testing Strategies Comprehensive strategies for testing slash commands before deployment and distribution. ## Overview Testing commands ensures they work correctly, handle edge cases, and provide good user experience. A systematic testing approach catches issues early and builds confidence in command reliability. ## Testing Levels ### Level 1: Syntax and Structure Validation **What to test:** - YAML frontmatter syntax - Markdown format - File location and naming **How to test:** ```bash # Validate YAML frontmatter head -n 20 .claude/commands/my-command.md | grep -A 10 "^---" # Check for closing frontmatter marker head -n 20 .claude/commands/my-command.md | grep -c "^---" # Should be 2 # Verify file has .md extension ls .claude/commands/*.md # Check file is in correct location test -f .claude/commands/my-command.md && echo "Found" || echo "Missing" ``` **Automated validation script:** ```bash #!/bin/bash # validate-command.sh COMMAND_FILE="$1" if [ ! -f "$COMMAND_FILE" ]; then echo "ERROR: File not found: $COMMAND_FILE" exit 1 fi # Check .md extension if [[ ! "$COMMAND_FILE" =~ \.md$ ]]; then echo "ERROR: File must have .md extension" exit 1 fi # Validate YAML frontmatter if present if head -n 1 "$COMMAND_FILE" | grep -q "^---"; then # Count frontmatter markers MARKERS=$(head -n 50 "$COMMAND_FILE" | grep -c "^---") if [ "$MARKERS" -ne 2 ]; then echo "ERROR: Invalid YAML frontmatter (need exactly 2 '---' markers)" exit 1 fi echo "✓ YAML frontmatter syntax valid" fi # Check for empty file if [ ! -s "$COMMAND_FILE" ]; then echo "ERROR: File is empty" exit 1 fi echo "✓ Command file structure valid" ``` ### Level 2: Frontmatter Field Validation **What to test:** - Field types correct - Values in valid ranges - Required fields present (if any) **Validation script:** ```bash #!/bin/bash # validate-frontmatter.sh COMMAND_FILE="$1" # Extract YAML frontmatter FRONTMATTER=$(sed -n '/^---$/,/^---$/p' "$COMMAND_FILE" | sed '1d;$d') if [ -z "$FRONTMATTER" ]; then echo "No frontmatter to validate" exit 0 fi # Check 'model' field if present if echo "$FRONTMATTER" | grep -q "^model:"; then MODEL=$(echo "$FRONTMATTER" | grep "^model:" | cut -d: -f2 | tr -d ' ') if ! echo "sonnet opus haiku" | grep -qw "$MODEL"; then echo "ERROR: Invalid model '$MODEL' (must be sonnet, opus, or haiku)" exit 1 fi echo "✓ Model field valid: $MODEL" fi # Check 'allowed-tools' field format if echo "$FRONTMATTER" | grep -q "^allowed-tools:"; then echo "✓ allowed-tools field present" # Could add more sophisticated validation here fi # Check 'description' length if echo "$FRONTMATTER" | grep -q "^description:"; then DESC=$(echo "$FRONTMATTER" | grep "^description:" | cut -d: -f2-) LENGTH=${#DESC} if [ "$LENGTH" -gt 80 ]; then echo "WARNING: Description length $LENGTH (recommend < 60 chars)" else echo "✓ Description length acceptable: $LENGTH chars" fi fi echo "✓ Frontmatter fields valid" ``` ### Level 3: Manual Command Invocation **What to test:** - Command appears in `/help` - Command executes without errors - Output is as expected **Test procedure:** ```bash # 1. Start Claude Code claude --debug # 2. Check command appears in help > /help # Look for your command in the list # 3. Invoke command without arguments > /my-command # Check for reasonable error or behavior # 4. Invoke with valid arguments > /my-command arg1 arg2 # Verify expected behavior # 5. Check debug logs tail -f ~/.claude/debug-logs/latest # Look for errors or warnings ``` ### Level 4: Argument Testing **What to test:** - Positional arguments work ($1, $2, etc.) - $ARGUMENTS captures all arguments - Missing arguments handled gracefully - Invalid arguments detected **Test matrix:** | Test Case | Command | Expected Result | |-----------|---------|-----------------| | No args | `/cmd` | Graceful handling or useful message | | One arg | `/cmd arg1` | $1 substituted correctly | | Two args | `/cmd arg1 arg2` | $1 and $2 substituted | | Extra args | `/cmd a b c d` | All captured or extras ignored appropriately | | Special chars | `/cmd "arg with spaces"` | Quotes handled correctly | | Empty arg | `/cmd ""` | Empty string handled | **Test script:** ```bash #!/bin/bash # test-command-arguments.sh COMMAND="$1" echo "Testing argument handling for /$COMMAND" echo echo "Test 1: No arguments" echo " Command: /$COMMAND" echo " Expected: [describe expected behavior]" echo " Manual test required" echo echo "Test 2: Single argument" echo " Command: /$COMMAND test-value" echo " Expected: 'test-value' appears in output" echo " Manual test required" echo echo "Test 3: Multiple arguments" echo " Command: /$COMMAND arg1 arg2 arg3" echo " Expected: All arguments used appropriately" echo " Manual test required" echo echo "Test 4: Special characters" echo " Command: /$COMMAND \"value with spaces\"" echo " Expected: Entire phrase captured" echo " Manual test required" ``` ### Level 5: File Reference Testing **What to test:** - @ syntax loads file contents - Non-existent files handled - Large files handled appropriately - Multiple file references work **Test procedure:** ```bash # Create test files echo "Test content" > /tmp/test-file.txt echo "Second file" > /tmp/test-file-2.txt # Test single file reference > /my-command /tmp/test-file.txt # Verify file content is read # Test non-existent file > /my-command /tmp/nonexistent.txt # Verify graceful error handling # Test multiple files > /my-command /tmp/test-file.txt /tmp/test-file-2.txt # Verify both files processed # Test large file dd if=/dev/zero of=/tmp/large-file.bin bs=1M count=100 > /my-command /tmp/large-file.bin # Verify reasonable behavior (may truncate or warn) # Cleanup rm /tmp/test-file*.txt /tmp/large-file.bin ``` ### Level 6: Bash Execution Testing **What to test:** - !` commands execute correctly - Command output included in prompt - Command failures handled - Security: only allowed commands run **Test procedure:** ```bash # Create test command with bash execution cat > .claude/commands/test-bash.md << 'EOF' --- description: Test bash execution allowed-tools: Bash(echo:*), Bash(date:*) --- Current date: !`date` Test output: !`echo "Hello from bash"` Analysis of output above... EOF # Test in Claude Code > /test-bash # Verify: # 1. Date appears correctly # 2. Echo output appears # 3. No errors in debug logs # Test with disallowed command (should fail or be blocked) cat > .claude/commands/test-forbidden.md << 'EOF' --- description: Test forbidden command allowed-tools: Bash(echo:*) --- Trying forbidden: !`ls -la /` EOF > /test-forbidden # Verify: Permission denied or appropriate error ``` ### Level 7: Integration Testing **What to test:** - Commands work with other plugin components - Commands interact correctly with each other - State management works across invocations - Workflow commands execute in sequence **Test scenarios:** **Scenario 1: Command + Hook Integration** ```bash # Setup: Command that triggers a hook # Test: Invoke command, verify hook executes # Command: .claude/commands/risky-operation.md # Hook: PreToolUse that validates the operation > /risky-operation # Verify: Hook executes and validates before command completes ``` **Scenario 2: Command Sequence** ```bash # Setup: Multi-command workflow > /workflow-init # Verify: State file created > /workflow-step2 # Verify: State file read, step 2 executes > /workflow-complete # Verify: State file cleaned up ``` **Scenario 3: Command + MCP Integration** ```bash # Setup: Command uses MCP tools # Test: Verify MCP server accessible > /mcp-command # Verify: # 1. MCP server starts (if stdio) # 2. Tool calls succeed # 3. Results included in output ``` ## Automated Testing Approaches ### Command Test Suite Create a test suite script: ```bash #!/bin/bash # test-commands.sh - Command test suite TEST_DIR=".claude/commands" FAILED_TESTS=0 echo "Command Test Suite" echo "==================" echo for cmd_file in "$TEST_DIR"/*.md; do cmd_name=$(basename "$cmd_file" .md) echo "Testing: $cmd_name" # Validate structure if ./validate-command.sh "$cmd_file"; then echo " ✓ Structure valid" else echo " ✗ Structure invalid" ((FAILED_TESTS++)) fi # Validate frontmatter if ./validate-frontmatter.sh "$cmd_file"; then echo " ✓ Frontmatter valid" else echo " ✗ Frontmatter invalid" ((FAILED_TESTS++)) fi echo done echo "==================" echo "Tests complete" echo "Failed: $FAILED_TESTS" exit $FAILED_TESTS ``` ### Pre-Commit Hook Validate commands before committing: ```bash #!/bin/bash # .git/hooks/pre-commit echo "Validating commands..." COMMANDS_CHANGED=$(git diff --cached --name-only | grep "\.claude/commands/.*\.md") if [ -z "$COMMANDS_CHANGED" ]; then echo "No commands changed" exit 0 fi for cmd in $COMMANDS_CHANGED; do echo "Checking: $cmd" if ! ./scripts/validate-command.sh "$cmd"; then echo "ERROR: Command validation failed: $cmd" exit 1 fi done echo "✓ All commands valid" ``` ### Continuous Testing Test commands in CI/CD: ```yaml # .github/workflows/test-commands.yml name: Test Commands on: [push, pull_request] jobs: test: runs-on: ubuntu-latest steps: - uses: actions/checkout@v2 - name: Validate command structure run: | for cmd in .claude/commands/*.md; do echo "Testing: $cmd" ./scripts/validate-command.sh "$cmd" done - name: Validate frontmatter run: | for cmd in .claude/commands/*.md; do ./scripts/validate-frontmatter.sh "$cmd" done - name: Check for TODOs run: | if grep -r "TODO" .claude/commands/; then echo "ERROR: TODOs found in commands" exit 1 fi ``` ## Edge Case Testing ### Test Edge Cases **Empty arguments:** ```bash > /cmd "" > /cmd '' '' ``` **Special characters:** ```bash > /cmd "arg with spaces" > /cmd arg-with-dashes > /cmd arg_with_underscores > /cmd arg/with/slashes > /cmd 'arg with "quotes"' ``` **Long arguments:** ```bash > /cmd $(python -c "print('a' * 10000)") ``` **Unusual file paths:** ```bash > /cmd ./file > /cmd ../file > /cmd ~/file > /cmd "/path with spaces/file" ``` **Bash command edge cases:** ```markdown # Commands that might fail !`exit 1` !`false` !`command-that-does-not-exist` # Commands with special output !`echo ""` !`cat /dev/null` !`yes | head -n 1000000` ``` ## Performance Testing ### Response Time Testing ```bash #!/bin/bash # test-command-performance.sh COMMAND="$1" echo "Testing performance of /$COMMAND" echo for i in {1..5}; do echo "Run $i:" START=$(date +%s%N) # Invoke command (manual step - record time) echo " Invoke: /$COMMAND" echo " Start time: $START" echo " (Record end time manually)" echo done echo "Analyze results:" echo " - Average response time" echo " - Variance" echo " - Acceptable threshold: < 3 seconds for fast commands" ``` ### Resource Usage Testing ```bash # Monitor Claude Code during command execution # In terminal 1: claude --debug # In terminal 2: watch -n 1 'ps aux | grep claude' # Execute command and observe: # - Memory usage # - CPU usage # - Process count ``` ## User Experience Testing ### Usability Checklist - [ ] Command name is intuitive - [ ] Description is clear in `/help` - [ ] Arguments are well-documented - [ ] Error messages are helpful - [ ] Output is formatted readably - [ ] Long-running commands show progress - [ ] Results are actionable - [ ] Edge cases have good UX ### User Acceptance Testing Recruit testers: ```markdown # Testing Guide for Beta Testers ## Command: /my-new-command ### Test Scenarios 1. **Basic usage:** - Run: `/my-new-command` - Expected: [describe] - Rate clarity: 1-5 2. **With arguments:** - Run: `/my-new-command arg1 arg2` - Expected: [describe] - Rate usefulness: 1-5 3. **Error case:** - Run: `/my-new-command invalid-input` - Expected: Helpful error message - Rate error message: 1-5 ### Feedback Questions 1. Was the command easy to understand? 2. Did the output meet your expectations? 3. What would you change? 4. Would you use this command regularly? ``` ## Testing Checklist Before releasing a command: ### Structure - [ ] File in correct location - [ ] Correct .md extension - [ ] Valid YAML frontmatter (if present) - [ ] Markdown syntax correct ### Functionality - [ ] Command appears in `/help` - [ ] Description is clear - [ ] Command executes without errors - [ ] Arguments work as expected - [ ] File references work - [ ] Bash execution works (if used) ### Edge Cases - [ ] Missing arguments handled - [ ] Invalid arguments detected - [ ] Non-existent files handled - [ ] Special characters work - [ ] Long inputs handled ### Integration - [ ] Works with other commands - [ ] Works with hooks (if applicable) - [ ] Works with MCP (if applicable) - [ ] State management works ### Quality - [ ] Performance acceptable - [ ] No security issues - [ ] Error messages helpful - [ ] Output formatted well - [ ] Documentation complete ### Distribution - [ ] Tested by others - [ ] Feedback incorporated - [ ] README updated - [ ] Examples provided ## Debugging Failed Tests ### Common Issues and Solutions **Issue: Command not appearing in /help** ```bash # Check file location ls -la .claude/commands/my-command.md # Check permissions chmod 644 .claude/commands/my-command.md # Check syntax head -n 20 .claude/commands/my-command.md # Restart Claude Code claude --debug ``` **Issue: Arguments not substituting** ```bash # Verify syntax grep '\$1' .claude/commands/my-command.md grep '\$ARGUMENTS' .claude/commands/my-command.md # Test with simple command first echo "Test: \$1 and \$2" > .claude/commands/test-args.md ``` **Issue: Bash commands not executing** ```bash # Check allowed-tools grep "allowed-tools" .claude/commands/my-command.md # Verify command syntax grep '!\`' .claude/commands/my-command.md # Test command manually date echo "test" ``` **Issue: File references not working** ```bash # Check @ syntax grep '@' .claude/commands/my-command.md # Verify file exists ls -la /path/to/referenced/file # Check permissions chmod 644 /path/to/referenced/file ``` ## Best Practices 1. **Test early, test often**: Validate as you develop 2. **Automate validation**: Use scripts for repeatable checks 3. **Test edge cases**: Don't just test the happy path 4. **Get feedback**: Have others test before wide release 5. **Document tests**: Keep test scenarios for regression testing 6. **Monitor in production**: Watch for issues after release 7. **Iterate**: Improve based on real usage data --- ## tool-evaluation.md # Tool Evaluation Multiple agents independently run a single evaluation task from an evaluation file. ::: ::: {.cell .code} ``` python import json import re import time import traceback import xml.etree.ElementTree as ET # noqa: S314 from pathlib import Path from typing import Any from anthropic import Anthropic ``` ::: ::: {.cell .markdown} ## Prompts ::: ::: {.cell .code execution_count="2"} ``` python # Embedded evaluator prompt EVALUATION_PROMPT = """You are an AI assistant with access to tools. When given a task, you MUST: 1. Use the available tools to complete the task 2. Provide summary of each step in your approach, wrapped in tags 3. Provide feedback on the tools provided, wrapped in tags 4. Provide your final response, wrapped in tags Summary Requirements: - In your tags, you must explain: - The steps you took to complete the task - Which tools you used, in what order, and why - The inputs you provided to each tool - The outputs you received from each tool - A summary for how you arrived at the response Feedback Requirements: - In your tags, provide constructive feedback on the tools: - Comment on tool names: Are they clear and descriptive? - Comment on input parameters: Are they well-documented? Are required vs optional parameters clear? - Comment on descriptions: Do they accurately describe what the tool does? - Comment on any errors encountered during tool usage: Did the tool fail to execute? Did the tool return too many tokens? - Identify specific areas for improvement and explain WHY they would help - Be specific and actionable in your suggestions Response Requirements: - Your response should be concise and directly address what was asked - Always wrap your final response in tags - If you cannot solve the task return NOT_FOUND - For numeric responses, provide just the number - For IDs, provide just the ID - For names or text, provide the exact text requested - Your response should go last""" ``` ::: ::: {.cell .markdown} ## Agent Loop ::: ::: {.cell .code} ``` python client = Anthropic() model = "claude-sonnet-4-5" def agent_loop(prompt: str, tools: list[dict[str, Any]] = None) -> tuple[str, dict[str, Any]]: """Simplified agent class for tool evaluation""" messages = [{"role": "user", "content": prompt}] response = client.messages.create( model=model, max_tokens=4096, system=EVALUATION_PROMPT, messages=messages, tools=tools, ) messages.append({"role": "assistant", "content": response.content}) # Track tool calls with timing tool_metrics = {} # {tool_name: {"count": N, "durations": [X1, X2, ...]}} def _prepare_tool_result(tool_use_id, tool_result): return { "role": "user", "content": [ { "type": "tool_result", "tool_use_id": tool_use_id, "content": tool_result, } ], } while response.stop_reason == "tool_use": tool_use = next(block for block in response.content if block.type == "tool_use") tool_name = tool_use.name tool_start_ts = time.time() try: # Note: eval is used here for demonstration purposes to dynamically call tool functions. # In production, use a safer dispatch mechanism like a dictionary of functions. tool_response = eval( # noqa: S307 f"{tool_name}(**tool_use.input)" ) # Call the tool function with its input except Exception as e: tool_response = f"Error executing tool {tool_name}: {str(e)}\n" tool_response += traceback.format_exc() tool_duration = time.time() - tool_start_ts # Update tool metrics if tool_name not in tool_metrics: tool_metrics[tool_name] = {"count": 0, "durations": []} tool_metrics[tool_name]["count"] += 1 tool_metrics[tool_name]["durations"].append(tool_duration) # Prepare tool result and append to messages messages.append(_prepare_tool_result(tool_use.id, tool_response)) response = client.messages.create( model=model, max_tokens=4096, system=EVALUATION_PROMPT, messages=messages, tools=tools, ) messages.append({"role": "assistant", "content": response.content}) response = next( (block.text for block in response.content if hasattr(block, "text")), None, ) return response, tool_metrics ``` ::: ::: {.cell .markdown} ## Helper Functions ::: ::: {.cell .code} ``` python def parse_evaluation_file(file_path: Path) -> list[dict[str, Any]]: """Parse XML evaluation file and return list of evaluation tasks.""" try: # Parse trusted local XML evaluation file tree = ET.parse(file_path) # noqa: S314 root = tree.getroot() evaluations = [] # Check for task elements tasks = root.findall(".//task") for task in tasks: prompt_elem = task.find("prompt") response_elem = task.find("response") if prompt_elem is not None and response_elem is not None: eval_dict = { "prompt": (prompt_elem.text or "").strip(), "response": (response_elem.text or "").strip(), } evaluations.append(eval_dict) return evaluations except Exception as e: print(f"Error parsing evaluation file {file_path}: {e}") return [] ``` ::: ::: {.cell .code execution_count="5"} ``` python def evaluate_single_task( task: dict[str, Any], tools: list[dict[str, Any]], task_index: int ) -> dict[str, Any]: """Evaluate a single task with the given tools.""" start_time = time.time() # Run the task print(f"Task {task_index + 1}: Running task with prompt: {task['prompt']}") response, tool_metrics = agent_loop(task["prompt"], tools) # Extract all tagged content def _extract_xml_content(text, tag): pattern = rf"<{tag}>(.*?)" matches = re.findall(pattern, text, re.DOTALL) return matches[-1].strip() if matches else None response, summary, feedback = ( _extract_xml_content(response, tag) for tag in ["response", "summary", "feedback"] ) duration_seconds = time.time() - start_time return { "prompt": task["prompt"], "expected": task["response"], "actual": response, "score": int(response == task["response"]), "total_duration": duration_seconds, "tool_calls": tool_metrics, "num_tool_calls": sum(len(metrics["durations"]) for metrics in tool_metrics.values()), "summary": summary, "feedback": feedback, } ``` ::: ::: {.cell .markdown} ## Main Evaluation Function ::: ::: {.cell .code execution_count="6"} ``` python # Report Templates REPORT_HEADER = """ # Evaluation Report ## Summary - **Accuracy**: {correct}/{total} ({accuracy:.1f}%) - **Average Task Duration**: {average_duration_s:.2f}s - **Average Tool Calls per Task**: {average_tool_calls:.2f} - **Total Tool Calls**: {total_tool_calls} --- """ TASK_TEMPLATE = """ ### Task **Prompt**: {prompt} **Ground Truth Response**: `{expected_response}` **Actual Response**: `{actual_response}` **Correct**: {correct_indicator} **Duration**: {total_duration:.2f}s **Tool Calls**: {tool_calls} **Summary** {summary} **Feedback** {feedback} --- """ def run_evaluation(eval_path: str, tools: list[dict[str, Any]]) -> str: """ Run evaluation with provided tools using a simple loop. Args: eval_path: Path to XML evaluation file tools: List of tool definitions to use for evaluation """ print("🚀 Starting Evaluation") eval_file = Path(eval_path) # Parse evaluation tasks tasks = parse_evaluation_file(eval_file) print(f"📋 Loaded {len(tasks)} evaluation tasks") # Simple loop to run all tasks results = [] for i, task in enumerate(tasks): print(f"Processing task {i + 1}/{len(tasks)}") results.append(evaluate_single_task(task, tools, i)) # Calculate summary statistics correct = sum(r["score"] for r in results) accuracy = (correct / len(results)) * 100 average_duration_s = sum(r["total_duration"] for r in results) / len(results) average_tool_calls = sum(r["num_tool_calls"] for r in results) / len(results) total_tool_calls = sum(r["num_tool_calls"] for r in results) report = REPORT_HEADER.format( correct=correct, total=len(results), accuracy=accuracy, average_duration_s=average_duration_s, average_tool_calls=average_tool_calls, total_tool_calls=total_tool_calls, ) report += "".join( [ TASK_TEMPLATE.format( prompt=task["prompt"], expected_response=task["response"], actual_response=result["actual"], correct_indicator="✅" if result["score"] else "❌", total_duration=result["total_duration"], tool_calls=json.dumps(result["tool_calls"], indent=2), summary=result["summary"] or "N/A", feedback=result["feedback"] or "N/A", ) for task, result in zip(tasks, results, strict=False) ] ) # Join all sections into final report return report ``` ::: ::: {.cell .markdown} ## Calculator Tool ::: ::: {.cell .code} ``` python def calculator(expression: str) -> str: """A basic calculator that performs arithmetic operations.""" try: # Note: eval with restricted builtins is used here for demonstration. # In production, use a safer alternative like a math expression parser. result = eval(expression, {"__builtins__": {}}, {}) # noqa: S307 return str(result) except Exception as e: return f"Error: {str(e)}" # Define the tool schema for the calculator calculator_tool = { "name": "calculator", "description": "", # An unhelpful tool description. "input_schema": { "type": "object", "properties": { "expression": { "type": "string", "description": "", # An unhelpful schema description. } }, "required": ["expression"], }, } # Set the tools list tools = [calculator_tool] ``` ::: ::: {.cell .markdown} ## Run Evaluation ::: :::: {.cell .code execution_count="8"} ``` python # Run evaluation print("✅ Using calculator tool") report = run_evaluation(eval_path="evaluation.xml", tools=tools) print(report) ``` ::: {.output .stream .stdout} ✅ Using calculator tool 🚀 Starting Evaluation 📋 Loaded 8 evaluation tasks Processing task 1/8 Task 1: Running task with prompt: Calculate the compound interest on $10,000 invested at 5% annual interest rate, compounded monthly for 3 years. What is the final amount in dollars (rounded to 2 decimal places)? Processing task 2/8 Task 2: Running task with prompt: A projectile is launched at a 45-degree angle with an initial velocity of 50 m/s. Calculate the total distance (in meters) it has traveled from the launch point after 2 seconds, assuming g=9.8 m/s². Round to 2 decimal places. Processing task 3/8 Task 3: Running task with prompt: A sphere has a volume of 500 cubic meters. Calculate its surface area in square meters. Round to 2 decimal places. Processing task 4/8 Task 4: Running task with prompt: Calculate the population standard deviation of this dataset: [12, 15, 18, 22, 25, 30, 35]. Round to 2 decimal places. Processing task 5/8 Task 5: Running task with prompt: Calculate the pH of a solution with a hydrogen ion concentration of 3.5 × 10^-5 M. Round to 2 decimal places. Processing task 6/8 Task 6: Running task with prompt: Calculate the monthly payment for a $200,000 mortgage at 4.5% annual interest rate for 30 years (360 months). Use the standard mortgage payment formula. Round to 2 decimal places. Processing task 7/8 Task 7: Running task with prompt: Calculate the energy in joules of a photon with wavelength 550 nanometers. Use h = 6.626 × 10^-34 J·s and c = 3 × 10^8 m/s. Express the answer in scientific notation with 2 significant figures after the decimal (e.g., 3.61e-19). Processing task 8/8 Task 8: Running task with prompt: Find the larger real root of the quadratic equation 3x² - 7x + 2 = 0. Give the exact value. # Evaluation Report ## Summary - **Accuracy**: 7/8 (87.5%) - **Average Task Duration**: 22.73s - **Average Tool Calls per Task**: 7.75 - **Total Tool Calls**: 62 --- ### Task **Prompt**: Calculate the compound interest on $10,000 invested at 5% annual interest rate, compounded monthly for 3 years. What is the final amount in dollars (rounded to 2 decimal places)? **Ground Truth Response**: `11614.72` **Actual Response**: `$11,614.72` **Correct**: ❌ **Duration**: 18.64s **Tool Calls**: { "calculator": { "count": 6, "durations": [ 9.560585021972656e-05, 9.870529174804688e-05, 8.988380432128906e-05, 0.00011301040649414062, 0.00010704994201660156, 8.821487426757812e-05 ] } } **Summary** I approached this compound interest calculation in the following steps: 1. First, I identified the formula needed: P(1 + r/n)^(nt) where: - P = principal ($10,000) - r = annual interest rate (5% or 0.05) - n = number of times compounded per year (12 for monthly) - t = time in years (3) 2. I initially tried using the calculator tool with the formula using ^ for exponentiation, but received an error. 3. I corrected the syntax by using ** for exponentiation in Python, calculating 10000 * (1 + 0.05/12)**(12*3). 4. The calculator returned 11614.722313334678. 5. I attempted several approaches to round to 2 decimal places using functions like round() and int(), but these weren't available in the calculator environment. 6. Since the calculator doesn't have built-in rounding functions, I had to manually round the result to 2 decimal places: $11,614.72. **Feedback** The calculator tool has both strengths and areas for improvement: 1. Tool name: "calculator" is clear and descriptive, immediately conveying its purpose. 2. Input parameters: The "expression" parameter is simple, but lacks description of what syntax is supported. It would be helpful to specify that it uses Python syntax (particularly ** for exponentiation rather than ^). 3. Error messaging: The error messages are helpful in identifying syntax issues, but don't provide guidance on how to fix them. 4. Functionality limitations: The calculator doesn't support common mathematical functions like round(), int(), or the math module. It would be more useful if it included basic rounding and mathematical functions. 5. Documentation: It would be beneficial to include a brief description of supported operations and functions, along with examples of proper syntax for common calculations. Overall, adding better documentation and expanding the supported functions would significantly improve the usability of this tool. --- ### Task **Prompt**: A projectile is launched at a 45-degree angle with an initial velocity of 50 m/s. Calculate the total distance (in meters) it has traveled from the launch point after 2 seconds, assuming g=9.8 m/s². Round to 2 decimal places. **Ground Truth Response**: `87.25` **Actual Response**: `87.25` **Correct**: ✅ **Duration**: 31.06s **Tool Calls**: { "calculator": { "count": 12, "durations": [ 9.5367431640625e-05, 9.465217590332031e-05, 7.987022399902344e-05, 8.726119995117188e-05, 9.036064147949219e-05, 8.606910705566406e-05, 9.298324584960938e-05, 9.226799011230469e-05, 7.963180541992188e-05, 8.96453857421875e-05, 9.012222290039062e-05, 7.605552673339844e-05 ] } } **Summary** To solve this projectile motion problem, I took the following steps: 1. I first calculated the horizontal distance after 2 seconds: - Used the formula x = v₀ × cos(θ) × t - Since the calculator didn't accept trigonometric functions directly, I used the value 0.7071 (which is approximately cos(45°)) - Input: 50 * 2 * 0.7071 - Output: 70.71 meters 2. I then calculated the vertical distance after 2 seconds: - Used the formula y = v₀ × sin(θ) × t - 0.5 × g × t² - Since sin(45°) is also approximately 0.7071 - Input: 50 * 0.7071 * 2 - 0.5 * 9.8 * (2**2) - Output: 51.11 meters 3. Finally, I calculated the total distance using the Pythagorean theorem: - Used the formula d = √(x² + y²) - Since the sqrt function wasn't available, I used the power operator with exponent 1/2 - Input: ((70.71)**2 + (51.11)**2)**(1/2) - Output: 87.2475569858549 meters 4. I rounded the result to 2 decimal places, which gives 87.25 meters. **Feedback** The calculator tool has several limitations that made this problem more complex to solve: 1. Tool name: The name "calculator" is clear and descriptive. 2. Input parameters: The "expression" parameter is not well-documented. There's no information about what functions or operations are supported. 3. Description: There is no description provided for the tool, which would have been helpful to understand its capabilities and limitations. 4. Errors encountered: - The calculator doesn't support common mathematical functions like cos(), sin(), sqrt(), round(), int(), or floor(). - There's no math library implementation or prefix to use these functions. - There's no clear documentation on what functions are supported. 5. Areas for improvement: - Add documentation about supported operations and functions - Implement common mathematical functions (trigonometric, rounding, square root) - Include examples of valid expressions - Provide error messages that suggest alternatives when functions aren't available - Support a math library like Python's math module would make the calculator much more useful for scientific calculations --- ### Task **Prompt**: A sphere has a volume of 500 cubic meters. Calculate its surface area in square meters. Round to 2 decimal places. **Ground Truth Response**: `304.65` **Actual Response**: `304.65` **Correct**: ✅ **Duration**: 20.41s **Tool Calls**: { "calculator": { "count": 7, "durations": [ 8.916854858398438e-05, 9.226799011230469e-05, 8.7738037109375e-05, 8.368492126464844e-05, 9.083747863769531e-05, 0.0001010894775390625, 0.00010538101196289062 ] } } **Summary** N/A **Feedback** The calculator tool is useful but has some limitations: 1. Function naming: The name "calculator" is clear and descriptive. 2. Input parameters: The "expression" parameter is straightforward, but there's no documentation about which mathematical operations and functions are supported. 3. Supported operations: I encountered several errors with common mathematical operations: - The caret symbol (^) for exponentiation didn't work; I had to use ** instead - Built-in functions like 'round', 'int', and 'math' modules were not available 4. Improvement suggestions: - Provide documentation on which operators are supported (**, /, *, +, -, etc.) - Include information about available mathematical functions or implement common ones like round() - Add examples in the description showing proper syntax for exponentiation and other operations - Consider implementing a parameter for specifying decimal precision in the result These improvements would reduce trial and error and make the tool more efficient to use. --- ### Task **Prompt**: Calculate the population standard deviation of this dataset: [12, 15, 18, 22, 25, 30, 35]. Round to 2 decimal places. **Ground Truth Response**: `7.61` **Actual Response**: `7.61` **Correct**: ✅ **Duration**: 28.69s **Tool Calls**: { "calculator": { "count": 10, "durations": [ 8.7738037109375e-05, 8.344650268554688e-05, 9.584426879882812e-05, 8.487701416015625e-05, 0.00012683868408203125, 8.463859558105469e-05, 8.20159912109375e-05, 7.62939453125e-05, 7.939338684082031e-05, 8.535385131835938e-05 ] } } **Summary** To calculate the population standard deviation of the dataset [12, 15, 18, 22, 25, 30, 35] rounded to 2 decimal places, I took the following steps: 1. First, I calculated the mean of the dataset: - Input: (12 + 15 + 18 + 22 + 25 + 30 + 35) / 7 - Output: 22.428571428571427 2. Then I calculated the variance by: - Finding the squared deviation of each value from the mean - Summing these squared deviations - Dividing by the number of values (7) since this is a population standard deviation - Input: ((12-22.428571428571427)**2 + (15-22.428571428571427)**2 + (18-22.428571428571427)**2 + (22-22.428571428571427)**2 + (25-22.428571428571427)**2 + (30-22.428571428571427)**2 + (35-22.428571428571427)**2) / 7 - Output: 57.95918367346939 3. I then calculated the standard deviation by taking the square root of the variance: - Input: (57.95918367346939)**0.5 - Output: 7.61309291112813 4. Finally, I rounded to 2 decimal places: 7.61 (I had to determine this manually as the calculator tool didn't support rounding functions) **Feedback** The calculator tool provided basic functionality but had significant limitations: 1. Tool name: "calculator" is clear and descriptive, indicating its purpose well. 2. Input parameters: The "expression" parameter is straightforward, but there's no description of what types of expressions are supported or the syntax to use. 3. Description: The tool lacks a description of its capabilities and limitations. This would have been helpful to know in advance that functions like sum(), std(), round(), int(), and math library functions are not supported. 4. Errors encountered: Several errors occurred when trying to use common mathematical functions. The calculator doesn't support: - Statistical functions (std, sum) - Rounding functions (round) - Type conversion functions (int) - Math library functions 5. Areas for improvement: - Add support for common mathematical and statistical functions like sum(), mean(), std(), round() - Include a library of mathematical functions like math.floor(), math.ceil() - Provide clear documentation on supported operations and syntax - Allow for variable assignment and multi-line operations to simplify complex calculations - Add specific statistical calculation tools for common operations like standard deviation These improvements would make the tool much more versatile and prevent the need for breaking down complex calculations into multiple basic arithmetic operations. --- ### Task **Prompt**: Calculate the pH of a solution with a hydrogen ion concentration of 3.5 × 10^-5 M. Round to 2 decimal places. **Ground Truth Response**: `4.46` **Actual Response**: `4.46` **Correct**: ✅ **Duration**: 38.37s **Tool Calls**: { "calculator": { "count": 16, "durations": [ 8.726119995117188e-05, 8.940696716308594e-05, 9.322166442871094e-05, 8.702278137207031e-05, 0.00015282630920410156, 0.00010943412780761719, 0.00011801719665527344, 8.463859558105469e-05, 8.225440979003906e-05, 9.059906005859375e-05, 8.392333984375e-05, 8.988380432128906e-05, 0.00010824203491210938, 9.393692016601562e-05, 0.00010967254638671875, 0.000156402587890625 ] } } **Summary** I attempted to calculate the pH of a solution with a hydrogen ion concentration of 3.5 × 10^-5 M. Steps taken: 1. I tried various approaches to calculate pH using the calculator tool with different logarithm function notations (log10, ln, log), but encountered errors as these functions were not defined in the calculator tool. 2. I successfully verified the value of 3.5 × 10^-5 using the calculator tool. 3. Since direct logarithm calculations were not working, I switched to a manual calculation approach. 4. I used the pH formula: pH = -log10([H+]) 5. I broke down the calculation: pH = -log10(3.5 × 10^-5) = -(log10(3.5) + log10(10^-5)) = -(log10(3.5) - 5) 6. I used the known approximation that log10(3.5) ≈ 0.544 7. I calculated: pH ≈ -(0.544 - 5) = 4.456 8. I rounded the result to 2 decimal places: 4.46 The calculator tool was used multiple times with different expressions, but had limitations with logarithmic functions. **Feedback** The calculator tool has several limitations: 1. Tool name: "calculator" is clear and descriptive, accurately representing its basic function. 2. Input parameters: The "expression" parameter is clear but lacks documentation. There's no information about what mathematical operations or functions are supported. 3. Function support: The calculator doesn't support essential mathematical functions like logarithms (log, log10, ln), which are critical for many scientific calculations including pH. This significantly limits its utility for chemistry-related calculations. 4. Error messages: The error messages indicate missing functions but don't provide alternatives or guidance on what syntax is supported. 5. Documentation: There's no documentation about what mathematical libraries or syntax the calculator uses. Improvement suggestions: - Include support for common mathematical functions (log, exp, sqrt, etc.) - Add clear documentation about supported operations and functions - Implement specialized functions for common calculations (like pH) - Provide more helpful error messages that suggest correct syntax - Include examples of supported expressions in the tool description These improvements would make the calculator much more useful for scientific calculations and reduce the need for manual calculations or workarounds. --- ### Task **Prompt**: Calculate the monthly payment for a $200,000 mortgage at 4.5% annual interest rate for 30 years (360 months). Use the standard mortgage payment formula. Round to 2 decimal places. **Ground Truth Response**: `1013.37` **Actual Response**: `1013.37` **Correct**: ✅ **Duration**: 19.65s **Tool Calls**: { "calculator": { "count": 6, "durations": [ 0.00011038780212402344, 0.0001671314239501953, 8.273124694824219e-05, 0.00010371208190917969, 8.702278137207031e-05, 8.726119995117188e-05 ] } } **Summary** Steps taken to complete the task: 1. I needed to calculate the monthly mortgage payment using the standard formula: P * (r * (1+r)^n) / ((1+r)^n - 1) Where P = principal ($200,000), r = monthly interest rate (4.5%/12), n = number of payments (360) 2. First, I attempted to use the calculator tool with the formula using ^ for exponentiation, but received an error as Python uses ** for exponentiation. 3. I corrected the formula using ** for exponentiation and successfully calculated the monthly payment as $1,013.3706196517716. 4. I attempted to round to 2 decimal places using various methods (round(), int(), math.floor()), but these functions were not available in the calculator tool. 5. Since the built-in rounding functions weren't available, I manually rounded the result to $1,013.37 based on the calculated value. **Feedback** Calculator Tool Feedback: - Tool name: The name "calculator" is clear and descriptive, indicating its purpose well. - Input parameters: The "expression" parameter is clear but lacks description about syntax requirements or limitations. - Descriptions: The tool description is completely absent, which makes it difficult to understand what types of expressions are supported. - Errors encountered: The tool doesn't support common Python functions like round(), int(), or math module functions, which limits its utility for common mathematical operations. Areas for improvement: 1. Add a clear description for the calculator tool explaining what syntax it supports and what libraries/functions are available. 2. Include examples of supported operations in the documentation. 3. Support common mathematical functions like round() and basic modules like math for more complex calculations. 4. Provide better error messages that explain why an operation failed and suggest alternatives. 5. Add documentation about what syntax to use for exponentiation and other special operations to avoid trial and error. --- ### Task **Prompt**: Calculate the energy in joules of a photon with wavelength 550 nanometers. Use h = 6.626 × 10^-34 J·s and c = 3 × 10^8 m/s. Express the answer in scientific notation with 2 significant figures after the decimal (e.g., 3.61e-19). **Ground Truth Response**: `3.61e-19` **Actual Response**: `3.61e-19` **Correct**: ✅ **Duration**: 8.61s **Tool Calls**: { "calculator": { "count": 1, "durations": [ 8.845329284667969e-05 ] } } **Summary** To calculate the energy of a photon with wavelength 550 nanometers: 1. I identified the formula needed: E = hc/λ, where: - E is the energy in joules - h is Planck's constant (6.626 × 10^-34 J·s) - c is the speed of light (3 × 10^8 m/s) - λ is the wavelength (550 nm = 550 × 10^-9 m) 2. I used the calculator tool with the expression: 6.626e-34 * 3e8 / (550e-9) - Input: The mathematical expression with scientific notation - Output: 3.614181818181818e-19 joules 3. The result needs to be formatted with 2 significant figures after the decimal, so 3.61e-19 J. **Feedback** The calculator tool works well for this calculation: - Tool name: "calculator" is clear and describes its function well. - Input parameters: The single "expression" parameter is intuitive, though a brief description of acceptable syntax would be helpful. - Description: There's no actual description provided for the tool in the schema, which would be useful to explain capabilities and limitations. - Functionality: The tool handled scientific notation correctly and performed the calculation as expected. Improvement suggestion: Adding a brief description of the calculator's capabilities and acceptable syntax formats would help users understand how to properly format complex expressions, especially when dealing with scientific notation. --- ### Task **Prompt**: Find the larger real root of the quadratic equation 3x² - 7x + 2 = 0. Give the exact value. **Ground Truth Response**: `2` **Actual Response**: `2` **Correct**: ✅ **Duration**: 16.37s **Tool Calls**: { "calculator": { "count": 4, "durations": [ 0.0001506805419921875, 9.179115295410156e-05, 0.000102996826171875, 9.870529174804688e-05 ] } } **Summary** I solved the quadratic equation 3x² - 7x + 2 = 0 using the quadratic formula: x = (-b ± √(b² - 4ac))/(2a) Where a = 3, b = -7, c = 2 Steps taken: 1. First, I attempted to use the calculator with the "sqrt" function, but encountered an error. 2. Then I tried using the exponentiation with "^" which also caused an error. 3. Finally, I correctly used the "**" operator for exponentiation in the calculator tool. 4. I calculated both roots using the quadratic formula: - For the larger root: (-(-7) + ((-7)**2 - 4*3*2)**0.5)/(2*3) = 2.0 - For the smaller root: (-(-7) - ((-7)**2 - 4*3*2)**0.5)/(2*3) = 0.3333333333333333 The larger real root is 2. **Feedback** The calculator tool is useful but has some limitations and areas for improvement: 1. Tool name: "calculator" is clear and descriptive. 2. Input parameters: The single "expression" parameter is straightforward, though it would be helpful to have documentation on the supported syntax. 3. Description: The tool lacks a description of what operations are supported and what syntax to use. This caused my initial errors with sqrt() and the ^ operator. 4. Syntax limitations: The calculator doesn't support common mathematical functions like "sqrt" directly, requiring the use of exponentiation (raising to power 0.5) instead. It also uses Python-style "**" for exponentiation rather than the more common "^" symbol. 5. Error messages: The error messages were helpful in identifying the issues with my syntax. Improvement suggestions: - Add documentation explaining the supported operations and syntax - Support common mathematical functions like sqrt(), sin(), cos(), etc. - Consider accepting multiple syntax styles for common operations like exponentiation --- ::: ::::