{ "cells": [ { "cell_type": "markdown", "id": "d9cb90d8-e6a1-4c89-9cde-0e6c0a28f5c0", "metadata": {}, "source": [ "# Model Registry\n", "\n", "LangChain Benchmark includes a model registry to make it easier to run benchmarks across different models.\n", "\n", "If you see a model that you want to use and it's missing, please open a PR to add it!" ] }, { "cell_type": "code", "execution_count": 1, "id": "31831289-51fb-4ee5-98f3-0476cf11b187", "metadata": { "tags": [] }, "outputs": [], "source": [ "from langchain_benchmarks import model_registry" ] }, { "cell_type": "code", "execution_count": 2, "id": "aaed190d-fa4b-4445-9bfb-0e784e2a083b", "metadata": { "tags": [] }, "outputs": [ { "data": { "text/html": [ "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
Name Type Provider Description
gpt-3.5-turbo-1106 chat openai The latest GPT-3.5 Turbo model with improved instruction following, JSON mode, reproducible outputs, parallel function calling, and more. Returns a maximum of 4,096 output tokens.
gpt-3.5-turbo chat openai Currently points to gpt-3.5-turbo-0613.
gpt-3.5-turbo-16k chat openai Currently points to gpt-3.5-turbo-0613.
gpt-3.5-turbo-instructllm openai Similar capabilities as text-davinci-003 but compatible with legacy Completions endpoint and not Chat Completions.
gpt-3.5-turbo-0613 chat openai Legacy Snapshot of gpt-3.5-turbo from June 13th 2023. Will be deprecated on June 13, 2024.
gpt-3.5-turbo-16k-0613chat openai Legacy Snapshot of gpt-3.5-16k-turbo from June 13th 2023. Will be deprecated on June 13, 2024.
gpt-3.5-turbo-0301 chat openai Legacy Snapshot of gpt-3.5-turbo from March 1st 2023. Will be deprecated on June 13th 2024.
text-davinci-003 llm openai Legacy Can do language tasks with better quality and consistency than the curie, babbage, or ada models. Will be deprecated on Jan 4th 2024.
text-davinci-002 llm openai Legacy Similar capabilities to text-davinci-003 but trained with supervised fine-tuning instead of reinforcement learning. Will be deprecated on Jan 4th 2024.
code-davinci-002 llm openai Legacy Optimized for code-completion tasks. Will be deprecated on Jan 4th 2024.
gpt-4-1106-preview chat openai GPT-4 TurboNew - The latest GPT-4 model with improved instruction following, JSON mode, reproducible outputs, parallel function calling, and more. Returns a maximum of 4,096 output tokens. This preview model is not yet suited for production traffic.
gpt-4-0613 chat openai Snapshot of gpt-4 from June 13th 2023 with improved function calling support.
gpt-4-32k-0613 chat openai Snapshot of gpt-4-32k from June 13th 2023 with improved function calling support.
gpt-4-0314 chat openai Snapshot of gpt-4 from March 14th 2023 with function calling support. This model version will be deprecated on June 13th 2024.
gpt-4-32k-0314 chat openai Snapshot of gpt-4-32k from March 14th 2023 with function calling support. This model version will be deprecated on June 13th 2024.
llama-v2-7b-chat-fw chat fireworks 7b parameter LlamaChat model
llama-v2-13b-chat-fw chat fireworks 13b parameter LlamaChat model
llama-v2-70b-chat-fw chat fireworks 70b parameter LlamaChat model
mixtral-8x7b-fw-chat chat fireworks 8x7b parameter mixture of experts Mistral model, adapted for Chats
mixtral-8x7b-fw-llm llm fireworks 8x7b parameter mixture of experts Mistral model
claude-2 chat anthropic Superior performance on tasks that require complex reasoning
claude-2.1 chat anthropic Same performance as Claude 2, plus significant reduction in model hallucination rates
claude-instant-1.2 chat anthropic low-latency, high throughput.
claude-instant-1 chat anthropic low-latency, high throughput.
" ], "text/plain": [ "ModelRegistry(registered_models=[RegisteredModel(name='gpt-3.5-turbo-1106', provider='openai', description='The latest GPT-3.5 Turbo model with improved instruction following, JSON mode, reproducible outputs, parallel function calling, and more. Returns a maximum of 4,096 output tokens.', params={'model': 'gpt-3.5-turbo-1106'}, type='chat', path=None, url=None), RegisteredModel(name='gpt-3.5-turbo', provider='openai', description='Currently points to gpt-3.5-turbo-0613.', params={'model': 'gpt-3.5-turbo'}, type='chat', path=None, url=None), RegisteredModel(name='gpt-3.5-turbo-16k', provider='openai', description='Currently points to gpt-3.5-turbo-0613.', params={'model': 'gpt-3.5-turbo-16k'}, type='chat', path=None, url=None), RegisteredModel(name='gpt-3.5-turbo-instruct', provider='openai', description='Similar capabilities as text-davinci-003 but compatible with legacy Completions endpoint and not Chat Completions.', params={'model': 'gpt-3.5-turbo-instruct'}, type='llm', path=None, url=None), RegisteredModel(name='gpt-3.5-turbo-0613', provider='openai', description='Legacy Snapshot of gpt-3.5-turbo from June 13th 2023. Will be deprecated on June 13, 2024.', params={'model': 'gpt-3.5-turbo-0613'}, type='chat', path=None, url=None), RegisteredModel(name='gpt-3.5-turbo-16k-0613', provider='openai', description='Legacy Snapshot of gpt-3.5-16k-turbo from June 13th 2023. Will be deprecated on June 13, 2024.', params={'model': 'gpt-3.5-turbo-16k-0613'}, type='chat', path=None, url=None), RegisteredModel(name='gpt-3.5-turbo-0301', provider='openai', description='Legacy Snapshot of gpt-3.5-turbo from March 1st 2023. Will be deprecated on June 13th 2024.', params={'model': 'gpt-3.5-turbo-0301'}, type='chat', path=None, url=None), RegisteredModel(name='text-davinci-003', provider='openai', description='Legacy Can do language tasks with better quality and consistency than the curie, babbage, or ada models. Will be deprecated on Jan 4th 2024.', params={'model': 'text-davinci-003'}, type='llm', path=None, url=None), RegisteredModel(name='text-davinci-002', provider='openai', description='Legacy Similar capabilities to text-davinci-003 but trained with supervised fine-tuning instead of reinforcement learning. Will be deprecated on Jan 4th 2024.', params={'model': 'text-davinci-002'}, type='llm', path=None, url=None), RegisteredModel(name='code-davinci-002', provider='openai', description='Legacy Optimized for code-completion tasks. Will be deprecated on Jan 4th 2024.', params={'model': 'code-davinci-002'}, type='llm', path=None, url=None), RegisteredModel(name='gpt-4-1106-preview', provider='openai', description='GPT-4 TurboNew - The latest GPT-4 model with improved instruction following, JSON mode, reproducible outputs, parallel function calling, and more. Returns a maximum of 4,096 output tokens. This preview model is not yet suited for production traffic.', params={'model': 'gpt-4-1106-preview'}, type='chat', path=None, url=None), RegisteredModel(name='gpt-4-0613', provider='openai', description='Snapshot of gpt-4 from June 13th 2023 with improved function calling support.', params={'model': 'gpt-4-0613'}, type='chat', path=None, url=None), RegisteredModel(name='gpt-4-32k-0613', provider='openai', description='Snapshot of gpt-4-32k from June 13th 2023 with improved function calling support.', params={'model': 'gpt-4-32k-0613'}, type='chat', path=None, url=None), RegisteredModel(name='gpt-4-0314', provider='openai', description='Snapshot of gpt-4 from March 14th 2023 with function calling support. This model version will be deprecated on June 13th 2024.', params={'model': 'gpt-4-0314'}, type='chat', path=None, url=None), RegisteredModel(name='gpt-4-32k-0314', provider='openai', description='Snapshot of gpt-4-32k from March 14th 2023 with function calling support. This model version will be deprecated on June 13th 2024.', params={'model': 'gpt-4-32k-0314'}, type='chat', path=None, url=None), RegisteredModel(name='llama-v2-7b-chat-fw', provider='fireworks', description='7b parameter LlamaChat model', params={'model': 'accounts/fireworks/models/llama-v2-7b-chat'}, type='chat', path=None, url=None), RegisteredModel(name='llama-v2-13b-chat-fw', provider='fireworks', description='13b parameter LlamaChat model', params={'model': 'accounts/fireworks/models/llama-v2-13b-chat'}, type='chat', path=None, url=None), RegisteredModel(name='llama-v2-70b-chat-fw', provider='fireworks', description='70b parameter LlamaChat model', params={'model': 'accounts/fireworks/models/llama-v2-70b-chat'}, type='chat', path=None, url=None), RegisteredModel(name='mixtral-8x7b-fw-chat', provider='fireworks', description='8x7b parameter mixture of experts Mistral model, adapted for Chats', params={'model': 'accounts/fireworks/models/mixtral-8x7b-fw-chat'}, type='chat', path=None, url=None), RegisteredModel(name='mixtral-8x7b-fw-llm', provider='fireworks', description='8x7b parameter mixture of experts Mistral model', params={'model': 'accounts/fireworks/models/mixtral-8x7b'}, type='llm', path=None, url=None), RegisteredModel(name='claude-2', provider='anthropic', description='Superior performance on tasks that require complex reasoning', params={'model': 'claude-2'}, type='chat', path=None, url=None), RegisteredModel(name='claude-2.1', provider='anthropic', description='Same performance as Claude 2, plus significant reduction in model hallucination rates', params={'model': 'claude-2.1'}, type='chat', path=None, url=None), RegisteredModel(name='claude-instant-1.2', provider='anthropic', description='low-latency, high throughput.', params={'model': 'claude-instant-1.2'}, type='chat', path=None, url=None), RegisteredModel(name='claude-instant-1', provider='anthropic', description='low-latency, high throughput.', params={'model': 'claude-instant-1'}, type='chat', path=None, url=None)])" ] }, "execution_count": 2, "metadata": {}, "output_type": "execute_result" } ], "source": [ "model_registry" ] }, { "cell_type": "markdown", "id": "2974b4f9-575c-4907-97eb-7334ef5f1d8e", "metadata": {}, "source": [ "## Indexing\n", "\n", "Registry supports indexing by position. This ordering may change as more models get added." ] }, { "cell_type": "code", "execution_count": 3, "id": "64bfc631-1f1e-4cf4-8636-b8be7b46fef8", "metadata": { "tags": [] }, "outputs": [ { "data": { "text/html": [ "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
name gpt-3.5-turbo-1106
type chat
provider openai
descriptionThe latest GPT-3.5 Turbo model with improved instruction following, JSON mode, reproducible outputs, parallel function calling, and more. Returns a maximum of 4,096 output tokens.
model_path langchain.chat_models.openai.ChatOpenAI
url ModelPage
" ], "text/plain": [ "RegisteredModel(name='gpt-3.5-turbo-1106', provider='openai', description='The latest GPT-3.5 Turbo model with improved instruction following, JSON mode, reproducible outputs, parallel function calling, and more. Returns a maximum of 4,096 output tokens.', params={'model': 'gpt-3.5-turbo-1106'}, type='chat', path=None, url=None)" ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "registered_model = model_registry[0]\n", "registered_model" ] }, { "cell_type": "markdown", "id": "9150df8d-11a5-4e83-bc1c-b34119f75783", "metadata": {}, "source": [ "Can also index by model name" ] }, { "cell_type": "code", "execution_count": 4, "id": "267e746b-f13e-4484-bcbb-ed5dfbacae67", "metadata": { "tags": [] }, "outputs": [ { "data": { "text/html": [ "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
name gpt-3.5-turbo
type chat
provider openai
descriptionCurrently points to gpt-3.5-turbo-0613.
model_path langchain.chat_models.openai.ChatOpenAI
url ModelPage
" ], "text/plain": [ "RegisteredModel(name='gpt-3.5-turbo', provider='openai', description='Currently points to gpt-3.5-turbo-0613.', params={'model': 'gpt-3.5-turbo'}, type='chat', path=None, url=None)" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "model_registry[\"gpt-3.5-turbo\"]" ] }, { "cell_type": "markdown", "id": "26404672-0832-47be-bc7e-7f74116f6909", "metadata": {}, "source": [ "## Use the model\n", "\n", "To use the models, make sure that you have credentials set up. Most models take either an API key as part of the initializer or will use any ENV variables that might be present." ] }, { "cell_type": "code", "execution_count": 5, "id": "3604d49e-afbe-48ad-ac10-1e538b1ad376", "metadata": { "tags": [] }, "outputs": [], "source": [ "model = model_registry[\"gpt-3.5-turbo\"].get_model(model_params={\"temperature\": 0})" ] }, { "cell_type": "code", "execution_count": 6, "id": "bdece532-9843-427a-a10b-4545ed4ec151", "metadata": { "tags": [] }, "outputs": [ { "data": { "text/plain": [ "AIMessage(content=\"Hello! I am an AI language model developed by OpenAI, and I don't have a personal name. You can simply refer to me as OpenAI Assistant. How can I assist you today?\")" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "model.invoke(\"hello! what is your name?\")" ] }, { "cell_type": "code", "execution_count": 7, "id": "2b60c53b-f25b-4e37-9768-d0ac3859581c", "metadata": { "tags": [] }, "outputs": [ { "data": { "text/plain": [ "AIMessage(content=' Hello! My name is Claude.')" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "model = model_registry[\"claude-2.1\"].get_model()\n", "\n", "model.invoke(\"hello! what is your name?\")" ] }, { "cell_type": "markdown", "id": "abda1f5c-b6c1-4984-8fe6-34621448fd5e", "metadata": {}, "source": [ "## Iteration" ] }, { "cell_type": "code", "execution_count": 10, "id": "306c13c3-39f0-4290-9ce8-7a7e1b79f492", "metadata": { "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "-\n", "gpt-3.5-turbo-1106\n", "I am a language model AI created by OpenAI and do not have a personal name. You can call me OpenAI. \n", "-\n", "gpt-3.5-turbo\n", "I am an AI language model created by OpenAI, so I don't have a personal name. You can simply refer t\n", "-\n", "gpt-3.5-turbo-16k\n", "I am an AI language model developed by OpenAI, and I do not have a personal name. You can simply ref\n", "-\n", "gpt-3.5-turbo-instruct\n", "\n", "\n", "I am an AI digital assistant and do not have a name. You can call me OpenAI. What can I assist you\n", "-\n", "gpt-3.5-turbo-0613\n", "I am an AI language model developed by OpenAI and I don't have a personal name. You can call me Open\n", "-\n", "gpt-3.5-turbo-16k-0613\n", "I am an artificial intelligence created by OpenAI, so I don't have a personal name. You can refer to\n", "-\n", "gpt-3.5-turbo-0301\n", "As an AI language model, I do not have a personal name. You can call me OpenAI or simply AI.\n", "-\n", "text-davinci-003\n", "\n", "\n", "My name is Rebecca.\n", "-\n", "text-davinci-002\n", "\n", "\n", "My name is Sarah.\n", "-\n", "code-davinci-002\n", "Failed: InvalidRequestError(message='The model `code-davinci-002` does not exist or you do not have access to it.', param=None, code='model_not_found', http_status=404, request_id=None)\n", "-\n", "gpt-4-1106-preview\n", "I am an AI developed by OpenAI, so I don't have a personal name. However, you can refer to me as Cha\n", "-\n", "gpt-4-0613\n", "I am an artificial intelligence and do not have a personal name. I am often referred to as OpenAI.\n", "-\n", "gpt-4-32k-0613\n", "Failed: InvalidRequestError(message='The model `gpt-4-32k-0613` does not exist or you do not have access to it. Learn more: https://help.openai.com/en/articles/7102672-how-can-i-access-gpt-4.', param=None, co\n", "-\n", "gpt-4-0314\n", "I am an AI language model, so I don't have a personal name. You can call me OpenAI Assistant if you'\n", "-\n", "gpt-4-32k-0314\n", "Failed: InvalidRequestError(message='The model `gpt-4-32k-0314` does not exist or you do not have access to it. Learn more: https://help.openai.com/en/articles/7102672-how-can-i-access-gpt-4.', param=None, co\n", "-\n", "llama-v2-7b-chat-fw\n", "Hello! My name is Assistant, and I'm here to help you with any questions or concerns you may have. I\n", "-\n", "llama-v2-13b-chat-fw\n", "Hello! My name is LLaMA, I'm a helpful and respectful assistant developed by Meta AI. I'm here to as\n", "-\n", "llama-v2-70b-chat-fw\n", "Hello! My name is Assistant, and I'm here to help you with any questions or concerns you may have. I\n", "-\n", "mixtral-8x7b-fw-chat\n", "My name is Mistral 7B with 8 Experts MoE model. I am a large language model created by Mistral.ai an\n", "-\n", "mixtral-8x7b-fw-llm\n", "\n", "\n", "Where are you from?\n", "\n", "I was born in Los Angeles, but grew up in Phoenix, Arizona. I moved back to L\n", "-\n", "claude-2\n", " My name is Claude.\n", "-\n", "claude-2.1\n", " My name is Claude.\n", "-\n", "claude-instant-1.2\n", " My name is Claude.\n", "-\n", "claude-instant-1\n", " My name is Claude.\n" ] } ], "source": [ "for registered_model in model_registry:\n", " print(\"-\")\n", " print(registered_model.name)\n", " try:\n", " model = registered_model.get_model()\n", " result = model.invoke(\"What is your name?\")\n", " if isinstance(result, str):\n", " print(result[:100])\n", " else: # chat message\n", " print(result.content[:100])\n", " except Exception as e:\n", " print(\n", " f\"Failed: {repr(e)[:200]}\"\n", " ) # Fails if account does not have access to particular model or due to network limits etc" ] }, { "cell_type": "markdown", "id": "34801c48-83ed-4ada-b85b-aa3b8cfce31b", "metadata": {}, "source": [ "## Slicing\n", "\n", "Slicing notation" ] }, { "cell_type": "code", "execution_count": 11, "id": "db40d4da-dc70-4e6d-b7e8-61de1e15ed2e", "metadata": { "tags": [] }, "outputs": [ { "data": { "text/html": [ "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
Name Type Provider Description
gpt-3.5-turbo-1106chat openai The latest GPT-3.5 Turbo model with improved instruction following, JSON mode, reproducible outputs, parallel function calling, and more. Returns a maximum of 4,096 output tokens.
gpt-3.5-turbo chat openai Currently points to gpt-3.5-turbo-0613.
gpt-3.5-turbo-16k chat openai Currently points to gpt-3.5-turbo-0613.
" ], "text/plain": [ "ModelRegistry(registered_models=[RegisteredModel(name='gpt-3.5-turbo-1106', provider='openai', description='The latest GPT-3.5 Turbo model with improved instruction following, JSON mode, reproducible outputs, parallel function calling, and more. Returns a maximum of 4,096 output tokens.', params={'model': 'gpt-3.5-turbo-1106'}, type='chat', path=None, url=None), RegisteredModel(name='gpt-3.5-turbo', provider='openai', description='Currently points to gpt-3.5-turbo-0613.', params={'model': 'gpt-3.5-turbo'}, type='chat', path=None, url=None), RegisteredModel(name='gpt-3.5-turbo-16k', provider='openai', description='Currently points to gpt-3.5-turbo-0613.', params={'model': 'gpt-3.5-turbo-16k'}, type='chat', path=None, url=None)])" ] }, "execution_count": 11, "metadata": {}, "output_type": "execute_result" } ], "source": [ "model_registry[:3]" ] }, { "cell_type": "markdown", "id": "8d0260af-920f-4512-9273-6f7662369ec5", "metadata": {}, "source": [ "## Filtering\n", "\n", "\n", "Filtering" ] }, { "cell_type": "code", "execution_count": 12, "id": "9874846a-52f3-4921-b1ed-0858521bb9a9", "metadata": { "tags": [] }, "outputs": [ { "data": { "text/html": [ "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
Name Type Provider Description
llama-v2-7b-chat-fw chat fireworks 7b parameter LlamaChat model
llama-v2-13b-chat-fwchat fireworks 13b parameter LlamaChat model
llama-v2-70b-chat-fwchat fireworks 70b parameter LlamaChat model
mixtral-8x7b-fw-chatchat fireworks 8x7b parameter mixture of experts Mistral model, adapted for Chats
mixtral-8x7b-fw-llm llm fireworks 8x7b parameter mixture of experts Mistral model
" ], "text/plain": [ "ModelRegistry(registered_models=[RegisteredModel(name='llama-v2-7b-chat-fw', provider='fireworks', description='7b parameter LlamaChat model', params={'model': 'accounts/fireworks/models/llama-v2-7b-chat'}, type='chat', path=None, url=None), RegisteredModel(name='llama-v2-13b-chat-fw', provider='fireworks', description='13b parameter LlamaChat model', params={'model': 'accounts/fireworks/models/llama-v2-13b-chat'}, type='chat', path=None, url=None), RegisteredModel(name='llama-v2-70b-chat-fw', provider='fireworks', description='70b parameter LlamaChat model', params={'model': 'accounts/fireworks/models/llama-v2-70b-chat'}, type='chat', path=None, url=None), RegisteredModel(name='mixtral-8x7b-fw-chat', provider='fireworks', description='8x7b parameter mixture of experts Mistral model, adapted for Chats', params={'model': 'accounts/fireworks/models/mixtral-8x7b-fw-chat'}, type='chat', path=None, url=None), RegisteredModel(name='mixtral-8x7b-fw-llm', provider='fireworks', description='8x7b parameter mixture of experts Mistral model', params={'model': 'accounts/fireworks/models/mixtral-8x7b'}, type='llm', path=None, url=None)])" ] }, "execution_count": 12, "metadata": {}, "output_type": "execute_result" } ], "source": [ "model_registry.filter(provider=\"fireworks\")" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.11.4" } }, "nbformat": 4, "nbformat_minor": 5 }