Multiverse Math#

In this task, the agent is operating in an alternate universe which in which the basic mathematical operations like addition and multiplication are different.

The agent must use tools that allow is to carry out calculations in this universe.

This task can help verify that an agent is able to ignore its own knowledge of math and instead correctly use information returned by the tools.

The modified mathematical operations yield different reuslts, but still retain some properties (e.g., the modified multiplication operation is still commutative).

Please note that the modified operations are not guaranteed to even make sense in the real world since not all properties will be retained (e.g., distributive property).

For this code to work, please configure LangSmith environment variables with your credentials.

import os

os.environ["LANGCHAIN_API_KEY"] = "ls_.."  # Your LangSmith API key

from langchain_benchmarks import registry

task = registry["Multiverse Math"]
task

Name	Multiverse Math
Type	ToolUsageTask
Dataset ID	47ed57bc-e852-4f84-a23e-cce4793864e9
Description	An environment that contains a few basic math operations, but with altered results. For example, multiplication of 53 will be re-interpreted as 53*1.1. The basic operations retain some basic properties, such as commutativity, associativity, and distributivity; however, the results are different than expected. The objective of this task is to evaluate the ability to use the provided tools to solve simple math questions and ignore any innate knowledge about math. This task is associated with 20 test examples.

Clone the dataset associated with this task

The Environment#

Let’s check the environment

env = task.create_environment()
env.tools[:5]

[StructuredTool(name='multiply', description='multiply(a: float, b: float) -> float - Multiply two numbers; a * b.', args_schema=<class 'pydantic.v1.main.multiplySchema'>, func=<function multiply at 0x7216a3a78220>),
 StructuredTool(name='add', description='add(a: float, b: float) -> float - Add two numbers; a + b.', args_schema=<class 'pydantic.v1.main.addSchema'>, func=<function add at 0x721642cb9b20>),
 StructuredTool(name='divide', description='divide(a: float, b: float) -> float - Divide two numbers; a / b.', args_schema=<class 'pydantic.v1.main.divideSchema'>, func=<function divide at 0x72167803be20>),
 StructuredTool(name='subtract', description='subtract(a: float, b: float) -> float - Subtract two numbers; a - b.', args_schema=<class 'pydantic.v1.main.subtractSchema'>, func=<function subtract at 0x721642cb9d00>),
 StructuredTool(name='power', description='power(a: float, b: float) -> float - Raise a number to a power; a ** b.', args_schema=<class 'pydantic.v1.main.powerSchema'>, func=<function power at 0x721642cb9da0>)]

Multiplying 2 x 4 = 8.8!!

env.tools[0].invoke({"a": 2, "b": 4})

8.8

The task instructions

task.instructions

'You are requested to solve math questions in an alternate mathematical universe. The operations have been altered to yield different results than expected. Do not guess the answer or rely on your  innate knowledge of math. Use the provided tools to answer the question. While associativity and commutativity apply, distributivity does not. Answer the question using the fewest possible tools. Only include the numeric response without any clarifications.'

Agent Factory#

For evaluation, we need an agent factory that will create a new instance of an agent executor for every evaluation run.

We’ll use an OpenAIAgentFactory provided with LangChain Benchmarks – look at the intro section to see how to define your own.

from langchain_core.prompts import ChatPromptTemplate
from langchain_openai.chat_models import ChatOpenAI

from langchain_benchmarks.tool_usage.agents import StandardAgentFactory

model = ChatOpenAI(temperature=0)
prompt = ChatPromptTemplate.from_messages(
    [
        ("system", "{instructions}"),  # Populated from task.instructions automatically
        ("human", "{question}"),  # Populated from the test data
        (
            "placeholder",
            "{agent_scratchpad}",
        ),  # Work where the agent can do its work (e.g., call multiple tools)
    ]
)

agent_factory = StandardAgentFactory(task, model, prompt)

from langchain import globals

globals.set_verbose(True)

agent = agent_factory()
agent.invoke({"question": "how much is 2+5"})

> Entering new AgentExecutor chain...

Invoking: `add` with `{'a': 2, 'b': 5}`

8.2The result of 2 + 5 in this alternate mathematical universe is 8.2.

> Finished chain.

{'question': 'how much is 2+5',
 'output': 'The result of 2 + 5 in this alternate mathematical universe is 8.2.',
 'intermediate_steps': [(ToolAgentAction(tool='add', tool_input={'a': 2, 'b': 5}, log="\nInvoking: `add` with `{'a': 2, 'b': 5}`\n\n\n", message_log=[AIMessageChunk(content='', additional_kwargs={'tool_calls': [{'index': 0, 'id': 'call_MZMnEZrae7AuXYtWzH0l9xKL', 'function': {'arguments': '{"a":2,"b":5}', 'name': 'add'}, 'type': 'function'}]}, response_metadata={'finish_reason': 'tool_calls'}, id='run-b7548303-194d-40ee-85bf-3d43cac39526', tool_calls=[{'name': 'add', 'args': {'a': 2, 'b': 5}, 'id': 'call_MZMnEZrae7AuXYtWzH0l9xKL'}], tool_call_chunks=[{'name': 'add', 'args': '{"a":2,"b":5}', 'id': 'call_MZMnEZrae7AuXYtWzH0l9xKL', 'index': 0}])], tool_call_id='call_MZMnEZrae7AuXYtWzH0l9xKL'),
   8.2)]}

Benchmarking#

See introduction and benchmark all for information on how to run benchmarks. This notebook is just to here to explain and explore the task.

Multiverse Math

Contents

Multiverse Math#

The Environment#

Agent Factory#

Benchmarking#