Relational Data#

In this task, an agent is given access to a set of tools that can be used to make queries across 3 relational tables.

The tables contain information about users, locations and foods. The agent must answer questions about the data using the provided tools.

The underlying data looks like this (showing first 2 records)

User data:

id	name	email	location	favorite_color	favorite_foods
1	Alice	alice@gmail.com	1	red	[1, 2, 3]
21	Bob	bob@hotmail.com	2	orange	[4, 5, 6]

Location Data:

id	city	current_time	current_weather
1	New York	2023-11-14 10:30 AM	Partly Cloudy, Temperature: 68°F
2	Los Angeles	2023-11-14 7:45 AM	Sunny, Temperature: 75°F

Food data:

id	name	calories	allergic_ingredients
1	Pizza	285	[“Gluten”, “Dairy”]
2	Chocolate	50	[“Milk”, “Soy”]

The tools allow to look up information based on ids (e.g., get_user_email takes a user id and returns the email), and to search (e.g., find_foods_by_name takes a food name and returns a list of results).

from langchain_benchmarks import registry

For this code to work, please configure LangSmith environment variables with your credentials.

task = registry["Tool Usage - Relational Data"]

The Environment#

Let’s check the environment

env = task.create_environment()
env.tools[:5]

[StructuredTool(name='get_user_name', description="get_user_name(user_id: int) -> str - Get the name of the user with the given user ID.\n\n        Args:\n            user_id: The user's ID.\n\n        Returns:\n            The user's name.", args_schema=<class 'pydantic.v1.main.get_user_nameSchema'>, handle_tool_error=True, func=<function get_available_functions.<locals>.get_user_name at 0x78f30602fec0>),
 StructuredTool(name='list_user_ids', description='list_user_ids() -> List[str] - List all the user IDs.', args_schema=<class 'pydantic.v1.main.list_user_idsSchema'>, handle_tool_error=True, func=<function get_available_functions.<locals>.list_user_ids at 0x78f30602fe20>),
 StructuredTool(name='find_users_by_name', description='find_users_by_name(name: str) -> List[langchain_benchmarks.tool_usage.tasks.relational_data.SearchHit] - Find users with the given name.\n\n        Args:\n            name: The name to search for.\n\n        Returns:\n            The list of matching users.', args_schema=<class 'pydantic.v1.main.find_users_by_nameSchema'>, handle_tool_error=True, func=<function get_available_functions.<locals>.find_users_by_name at 0x78f306058040>),
 StructuredTool(name='find_locations_by_name', description='find_locations_by_name(city: str) -> List[langchain_benchmarks.tool_usage.tasks.relational_data.SearchHit] - Find locations with the given city name.', args_schema=<class 'pydantic.v1.main.find_locations_by_nameSchema'>, handle_tool_error=True, func=<function get_available_functions.<locals>.find_locations_by_name at 0x78f3060580e0>),
 StructuredTool(name='find_foods_by_name', description='find_foods_by_name(food: str) -> List[langchain_benchmarks.tool_usage.tasks.relational_data.SearchHit] - Find foods with the given name.', args_schema=<class 'pydantic.v1.main.find_foods_by_nameSchema'>, handle_tool_error=True, func=<function get_available_functions.<locals>.find_foods_by_name at 0x78f306058180>)]

env.tools[0].invoke({"user_id": 21})

'Bob'

env.tools[3].invoke({"city": "LA"})

[{'id': 2, 'city': 'Los Angeles'},
 {'id': 1, 'city': 'New York'},
 {'id': 3, 'city': 'Chicago'},
 {'id': 4, 'city': 'Houston'},
 {'id': 5, 'city': 'Miami'}]

Explore the task#

For evaluation, we need an agent factory that will create a new instance of an agent executor for every evaluation run.

We’ll use the StandardAgentFactory – look at the intro for more information about what it does and/or how to create a custom one.

from langchain_core.prompts import ChatPromptTemplate
from langchain_openai.chat_models import ChatOpenAI

from langchain_benchmarks.tool_usage.agents import StandardAgentFactory

model = ChatOpenAI(temperature=0)
prompt = ChatPromptTemplate.from_messages(
    [
        ("system", "{instructions}"),  # Populated from task.instructions automatically
        ("human", "{question}"),  # Populated from the test data
        (
            "placeholder",
            "{agent_scratchpad}",
        ),  # Work where the agent can do its work (e.g., call multiple tools)
    ]
)

agent_factory = StandardAgentFactory(task, model, prompt)

from langchain import globals

globals.set_verbose(True)

agent = agent_factory()
agent.invoke({"question": "what is the weather in LA"})

> Entering new AgentExecutor chain...

Invoking: `find_locations_by_name` with `{'city': 'LA'}`

[{'id': 2, 'city': 'Los Angeles'}, {'id': 1, 'city': 'New York'}, {'id': 3, 'city': 'Chicago'}, {'id': 4, 'city': 'Houston'}, {'id': 5, 'city': 'Miami'}]
Invoking: `get_current_weather_for_location` with `{'location_id': 2}`

Sunny, Temperature: 75°FThe weather in Los Angeles is sunny with a temperature of 75°F.

> Finished chain.

{'question': 'what is the weather in LA',
 'output': 'The weather in Los Angeles is sunny with a temperature of 75°F.',
 'intermediate_steps': [(ToolAgentAction(tool='find_locations_by_name', tool_input={'city': 'LA'}, log="\nInvoking: `find_locations_by_name` with `{'city': 'LA'}`\n\n\n", message_log=[AIMessageChunk(content='', additional_kwargs={'tool_calls': [{'index': 0, 'id': 'call_hJrCZgP4eDgaj6s4RtCKXTOo', 'function': {'arguments': '{"city":"LA"}', 'name': 'find_locations_by_name'}, 'type': 'function'}]}, response_metadata={'finish_reason': 'tool_calls'}, id='run-23ccffb0-3b17-46a4-b42e-5eaa3220b211', tool_calls=[{'name': 'find_locations_by_name', 'args': {'city': 'LA'}, 'id': 'call_hJrCZgP4eDgaj6s4RtCKXTOo'}], tool_call_chunks=[{'name': 'find_locations_by_name', 'args': '{"city":"LA"}', 'id': 'call_hJrCZgP4eDgaj6s4RtCKXTOo', 'index': 0}])], tool_call_id='call_hJrCZgP4eDgaj6s4RtCKXTOo'),
   [{'id': 2, 'city': 'Los Angeles'},
    {'id': 1, 'city': 'New York'},
    {'id': 3, 'city': 'Chicago'},
    {'id': 4, 'city': 'Houston'},
    {'id': 5, 'city': 'Miami'}]),
  (ToolAgentAction(tool='get_current_weather_for_location', tool_input={'location_id': 2}, log="\nInvoking: `get_current_weather_for_location` with `{'location_id': 2}`\n\n\n", message_log=[AIMessageChunk(content='', additional_kwargs={'tool_calls': [{'index': 0, 'id': 'call_lopYjo00MF9mZtnHtiisTqyp', 'function': {'arguments': '{"location_id":2}', 'name': 'get_current_weather_for_location'}, 'type': 'function'}]}, response_metadata={'finish_reason': 'tool_calls'}, id='run-9bba5827-d98b-464d-8028-25eb4a05d227', tool_calls=[{'name': 'get_current_weather_for_location', 'args': {'location_id': 2}, 'id': 'call_lopYjo00MF9mZtnHtiisTqyp'}], tool_call_chunks=[{'name': 'get_current_weather_for_location', 'args': '{"location_id":2}', 'id': 'call_lopYjo00MF9mZtnHtiisTqyp', 'index': 0}])], tool_call_id='call_lopYjo00MF9mZtnHtiisTqyp'),
   'Sunny, Temperature: 75°F')]}

Benchmarking#

See introduction and benchmark all for information on how to run benchmarks. This notebook is just to here to explain and explore the task.

Relational Data

Contents

Relational Data#

The Environment#

Explore the task#

Benchmarking#