Typewriter: Single Tool#

In this task, an agent is given access to a single tool called “type_letter”. This tool takes one argument called “letter” which is expected to be a character.

The agent must repeat the input string from the user, printing one character a time on a piece of virtual paper.

The agent is evaluated based on its ability to print the correct string using the “type_letter” tool.

from langchain_benchmarks import registry

task = registry["Tool Usage - Typewriter (1 tool)"]
task

Name	Tool Usage - Typewriter (1 tool)
Type	ToolUsageTask
Dataset ID	59577193-8938-4ccf-92a7-e8a96bcf4f86
Description	Environment with a single tool that accepts a single letter as input, and prints it on a piece of virtual paper. The objective of this task is to evaluate the ability of the model to use the provided tools to repeat a given input string. For example, if the string is 'abc', the tools 'a', 'b', and 'c' must be invoked in that order. The dataset includes examples of varying difficulty. The difficulty is measured by the length of the string.

The Environment#

The environment consists of a single tool and a virtual paper.

The tool accepts a single letter as an input and prints the leter on the virtual paper. If successful, the tool returns the output “OK”.

To determine what’s written on the paper, one needs to read the environment state.

env = task.create_environment()

env.tools

[StructuredTool(name='type_letter', description='type_letter(letter: str) -> str - Print the given letter on the paper.', args_schema=<class 'pydantic.v1.main.type_letterSchema'>, func=<function create_typer.<locals>.type_letter at 0x73a65909ee80>)]

tool = env.tools[0]

tool.invoke({"letter": "a"})

'OK'

tool.invoke({"letter": "b"})

'OK'

env.read_state()

'ab'

Explore the task#

For evaluation, we need an agent factory that will create a new instance of an agent executor for every evaluation run.

We’ll use the StandardAgentFactory – look at the intro for more information about what it does and/or how to create a custom one.

from langchain_core.prompts import ChatPromptTemplate
from langchain_openai.chat_models import ChatOpenAI

from langchain_benchmarks.tool_usage.agents import StandardAgentFactory

model = ChatOpenAI(temperature=0)
prompt = ChatPromptTemplate.from_messages(
    [
        ("system", "{instructions}"),  # Populated from task.instructions automatically
        ("human", "{question}"),  # Populated from the test data
        (
            "placeholder",
            "{agent_scratchpad}",
        ),  # Work where the agent can do its work (e.g., call multiple tools)
    ]
)

agent_factory = StandardAgentFactory(task, model, prompt)

from langchain import globals

globals.set_verbose(True)

agent = agent_factory()
agent.invoke({"question": "abc"})

> Entering new AgentExecutor chain...

Invoking: `type_letter` with `{'letter': 'a'}`

OK
Invoking: `type_letter` with `{'letter': 'b'}`

OK
Invoking: `type_letter` with `{'letter': 'c'}`

OKabc

> Finished chain.

{'question': 'abc',
 'output': 'abc',
 'intermediate_steps': [(ToolAgentAction(tool='type_letter', tool_input={'letter': 'a'}, log="\nInvoking: `type_letter` with `{'letter': 'a'}`\n\n\n", message_log=[AIMessageChunk(content='', additional_kwargs={'tool_calls': [{'index': 0, 'id': 'call_f4exPQMfz4VWxFJw4LhyMc80', 'function': {'arguments': '{"letter": "a"}', 'name': 'type_letter'}, 'type': 'function'}, {'index': 1, 'id': 'call_DHOJfLJEKuOKdzBa8ZLRYJZq', 'function': {'arguments': '{"letter": "b"}', 'name': 'type_letter'}, 'type': 'function'}, {'index': 2, 'id': 'call_EziJvB6jtUEg3CmXSsQ7OWBj', 'function': {'arguments': '{"letter": "c"}', 'name': 'type_letter'}, 'type': 'function'}]}, response_metadata={'finish_reason': 'tool_calls'}, id='run-7d6be045-b9e2-4f24-991c-8e34ccd53b98', tool_calls=[{'name': 'type_letter', 'args': {'letter': 'a'}, 'id': 'call_f4exPQMfz4VWxFJw4LhyMc80'}, {'name': 'type_letter', 'args': {'letter': 'b'}, 'id': 'call_DHOJfLJEKuOKdzBa8ZLRYJZq'}, {'name': 'type_letter', 'args': {'letter': 'c'}, 'id': 'call_EziJvB6jtUEg3CmXSsQ7OWBj'}], tool_call_chunks=[{'name': 'type_letter', 'args': '{"letter": "a"}', 'id': 'call_f4exPQMfz4VWxFJw4LhyMc80', 'index': 0}, {'name': 'type_letter', 'args': '{"letter": "b"}', 'id': 'call_DHOJfLJEKuOKdzBa8ZLRYJZq', 'index': 1}, {'name': 'type_letter', 'args': '{"letter": "c"}', 'id': 'call_EziJvB6jtUEg3CmXSsQ7OWBj', 'index': 2}])], tool_call_id='call_f4exPQMfz4VWxFJw4LhyMc80'),
   'OK'),
  (ToolAgentAction(tool='type_letter', tool_input={'letter': 'b'}, log="\nInvoking: `type_letter` with `{'letter': 'b'}`\n\n\n", message_log=[AIMessageChunk(content='', additional_kwargs={'tool_calls': [{'index': 0, 'id': 'call_f4exPQMfz4VWxFJw4LhyMc80', 'function': {'arguments': '{"letter": "a"}', 'name': 'type_letter'}, 'type': 'function'}, {'index': 1, 'id': 'call_DHOJfLJEKuOKdzBa8ZLRYJZq', 'function': {'arguments': '{"letter": "b"}', 'name': 'type_letter'}, 'type': 'function'}, {'index': 2, 'id': 'call_EziJvB6jtUEg3CmXSsQ7OWBj', 'function': {'arguments': '{"letter": "c"}', 'name': 'type_letter'}, 'type': 'function'}]}, response_metadata={'finish_reason': 'tool_calls'}, id='run-7d6be045-b9e2-4f24-991c-8e34ccd53b98', tool_calls=[{'name': 'type_letter', 'args': {'letter': 'a'}, 'id': 'call_f4exPQMfz4VWxFJw4LhyMc80'}, {'name': 'type_letter', 'args': {'letter': 'b'}, 'id': 'call_DHOJfLJEKuOKdzBa8ZLRYJZq'}, {'name': 'type_letter', 'args': {'letter': 'c'}, 'id': 'call_EziJvB6jtUEg3CmXSsQ7OWBj'}], tool_call_chunks=[{'name': 'type_letter', 'args': '{"letter": "a"}', 'id': 'call_f4exPQMfz4VWxFJw4LhyMc80', 'index': 0}, {'name': 'type_letter', 'args': '{"letter": "b"}', 'id': 'call_DHOJfLJEKuOKdzBa8ZLRYJZq', 'index': 1}, {'name': 'type_letter', 'args': '{"letter": "c"}', 'id': 'call_EziJvB6jtUEg3CmXSsQ7OWBj', 'index': 2}])], tool_call_id='call_DHOJfLJEKuOKdzBa8ZLRYJZq'),
   'OK'),
  (ToolAgentAction(tool='type_letter', tool_input={'letter': 'c'}, log="\nInvoking: `type_letter` with `{'letter': 'c'}`\n\n\n", message_log=[AIMessageChunk(content='', additional_kwargs={'tool_calls': [{'index': 0, 'id': 'call_f4exPQMfz4VWxFJw4LhyMc80', 'function': {'arguments': '{"letter": "a"}', 'name': 'type_letter'}, 'type': 'function'}, {'index': 1, 'id': 'call_DHOJfLJEKuOKdzBa8ZLRYJZq', 'function': {'arguments': '{"letter": "b"}', 'name': 'type_letter'}, 'type': 'function'}, {'index': 2, 'id': 'call_EziJvB6jtUEg3CmXSsQ7OWBj', 'function': {'arguments': '{"letter": "c"}', 'name': 'type_letter'}, 'type': 'function'}]}, response_metadata={'finish_reason': 'tool_calls'}, id='run-7d6be045-b9e2-4f24-991c-8e34ccd53b98', tool_calls=[{'name': 'type_letter', 'args': {'letter': 'a'}, 'id': 'call_f4exPQMfz4VWxFJw4LhyMc80'}, {'name': 'type_letter', 'args': {'letter': 'b'}, 'id': 'call_DHOJfLJEKuOKdzBa8ZLRYJZq'}, {'name': 'type_letter', 'args': {'letter': 'c'}, 'id': 'call_EziJvB6jtUEg3CmXSsQ7OWBj'}], tool_call_chunks=[{'name': 'type_letter', 'args': '{"letter": "a"}', 'id': 'call_f4exPQMfz4VWxFJw4LhyMc80', 'index': 0}, {'name': 'type_letter', 'args': '{"letter": "b"}', 'id': 'call_DHOJfLJEKuOKdzBa8ZLRYJZq', 'index': 1}, {'name': 'type_letter', 'args': '{"letter": "c"}', 'id': 'call_EziJvB6jtUEg3CmXSsQ7OWBj', 'index': 2}])], tool_call_id='call_EziJvB6jtUEg3CmXSsQ7OWBj'),
   'OK')],
 'state': 'abc'}

Benchmarking#

See introduction and benchmark all for information on how to run benchmarks. This notebook is just to here to explain and explore the task.

Typewriter: Single Tool

Contents

Typewriter: Single Tool#

The Environment#

Explore the task#

Benchmarking#