Model Registry#
LangChain Benchmark includes a model registry to make it easier to run benchmarks across different models.
If you see a model that you want to use and it’s missing, please open a PR to add it!
from langchain_benchmarks import model_registry
model_registry
| Name | Type | Provider | Description |
|---|---|---|---|
| gpt-3.5-turbo-1106 | chat | openai | The latest GPT-3.5 Turbo model with improved instruction following, JSON mode, reproducible outputs, parallel function calling, and more. Returns a maximum of 4,096 output tokens. |
| gpt-3.5-turbo | chat | openai | Currently points to gpt-3.5-turbo-0613. |
| gpt-3.5-turbo-16k | chat | openai | Currently points to gpt-3.5-turbo-0613. |
| gpt-3.5-turbo-instruct | llm | openai | Similar capabilities as text-davinci-003 but compatible with legacy Completions endpoint and not Chat Completions. |
| gpt-3.5-turbo-0613 | chat | openai | Legacy Snapshot of gpt-3.5-turbo from June 13th 2023. Will be deprecated on June 13, 2024. |
| gpt-3.5-turbo-16k-0613 | chat | openai | Legacy Snapshot of gpt-3.5-16k-turbo from June 13th 2023. Will be deprecated on June 13, 2024. |
| gpt-3.5-turbo-0301 | chat | openai | Legacy Snapshot of gpt-3.5-turbo from March 1st 2023. Will be deprecated on June 13th 2024. |
| text-davinci-003 | llm | openai | Legacy Can do language tasks with better quality and consistency than the curie, babbage, or ada models. Will be deprecated on Jan 4th 2024. |
| text-davinci-002 | llm | openai | Legacy Similar capabilities to text-davinci-003 but trained with supervised fine-tuning instead of reinforcement learning. Will be deprecated on Jan 4th 2024. |
| code-davinci-002 | llm | openai | Legacy Optimized for code-completion tasks. Will be deprecated on Jan 4th 2024. |
| gpt-4-1106-preview | chat | openai | GPT-4 TurboNew - The latest GPT-4 model with improved instruction following, JSON mode, reproducible outputs, parallel function calling, and more. Returns a maximum of 4,096 output tokens. This preview model is not yet suited for production traffic. |
| gpt-4-0613 | chat | openai | Snapshot of gpt-4 from June 13th 2023 with improved function calling support. |
| gpt-4-32k-0613 | chat | openai | Snapshot of gpt-4-32k from June 13th 2023 with improved function calling support. |
| gpt-4-0314 | chat | openai | Snapshot of gpt-4 from March 14th 2023 with function calling support. This model version will be deprecated on June 13th 2024. |
| gpt-4-32k-0314 | chat | openai | Snapshot of gpt-4-32k from March 14th 2023 with function calling support. This model version will be deprecated on June 13th 2024. |
| llama-v2-7b-chat-fw | chat | fireworks | 7b parameter LlamaChat model |
| llama-v2-13b-chat-fw | chat | fireworks | 13b parameter LlamaChat model |
| llama-v2-70b-chat-fw | chat | fireworks | 70b parameter LlamaChat model |
| mixtral-8x7b-fw-chat | chat | fireworks | 8x7b parameter mixture of experts Mistral model, adapted for Chats |
| mixtral-8x7b-fw-llm | llm | fireworks | 8x7b parameter mixture of experts Mistral model |
| claude-2 | chat | anthropic | Superior performance on tasks that require complex reasoning |
| claude-2.1 | chat | anthropic | Same performance as Claude 2, plus significant reduction in model hallucination rates |
| claude-instant-1.2 | chat | anthropic | low-latency, high throughput. |
| claude-instant-1 | chat | anthropic | low-latency, high throughput. |
Indexing#
Registry supports indexing by position. This ordering may change as more models get added.
registered_model = model_registry[0]
registered_model
| name | gpt-3.5-turbo-1106 |
| type | chat |
| provider | openai |
| description | The latest GPT-3.5 Turbo model with improved instruction following, JSON mode, reproducible outputs, parallel function calling, and more. Returns a maximum of 4,096 output tokens. |
| model_path | langchain.chat_models.openai.ChatOpenAI |
| url | ModelPage |
Can also index by model name
model_registry["gpt-3.5-turbo"]
| name | gpt-3.5-turbo |
| type | chat |
| provider | openai |
| description | Currently points to gpt-3.5-turbo-0613. |
| model_path | langchain.chat_models.openai.ChatOpenAI |
| url | ModelPage |
Use the model#
To use the models, make sure that you have credentials set up. Most models take either an API key as part of the initializer or will use any ENV variables that might be present.
model = model_registry["gpt-3.5-turbo"].get_model(model_params={"temperature": 0})
model.invoke("hello! what is your name?")
AIMessage(content="Hello! I am an AI language model developed by OpenAI, and I don't have a personal name. You can simply refer to me as OpenAI Assistant. How can I assist you today?")
model = model_registry["claude-2.1"].get_model()
model.invoke("hello! what is your name?")
AIMessage(content=' Hello! My name is Claude.')
Iteration#
for registered_model in model_registry:
print("-")
print(registered_model.name)
try:
model = registered_model.get_model()
result = model.invoke("What is your name?")
if isinstance(result, str):
print(result[:100])
else: # chat message
print(result.content[:100])
except Exception as e:
print(
f"Failed: {repr(e)[:200]}"
) # Fails if account does not have access to particular model or due to network limits etc
-
gpt-3.5-turbo-1106
I am a language model AI created by OpenAI and do not have a personal name. You can call me OpenAI.
-
gpt-3.5-turbo
I am an AI language model created by OpenAI, so I don't have a personal name. You can simply refer t
-
gpt-3.5-turbo-16k
I am an AI language model developed by OpenAI, and I do not have a personal name. You can simply ref
-
gpt-3.5-turbo-instruct
I am an AI digital assistant and do not have a name. You can call me OpenAI. What can I assist you
-
gpt-3.5-turbo-0613
I am an AI language model developed by OpenAI and I don't have a personal name. You can call me Open
-
gpt-3.5-turbo-16k-0613
I am an artificial intelligence created by OpenAI, so I don't have a personal name. You can refer to
-
gpt-3.5-turbo-0301
As an AI language model, I do not have a personal name. You can call me OpenAI or simply AI.
-
text-davinci-003
My name is Rebecca.
-
text-davinci-002
My name is Sarah.
-
code-davinci-002
Failed: InvalidRequestError(message='The model `code-davinci-002` does not exist or you do not have access to it.', param=None, code='model_not_found', http_status=404, request_id=None)
-
gpt-4-1106-preview
I am an AI developed by OpenAI, so I don't have a personal name. However, you can refer to me as Cha
-
gpt-4-0613
I am an artificial intelligence and do not have a personal name. I am often referred to as OpenAI.
-
gpt-4-32k-0613
Failed: InvalidRequestError(message='The model `gpt-4-32k-0613` does not exist or you do not have access to it. Learn more: https://help.openai.com/en/articles/7102672-how-can-i-access-gpt-4.', param=None, co
-
gpt-4-0314
I am an AI language model, so I don't have a personal name. You can call me OpenAI Assistant if you'
-
gpt-4-32k-0314
Failed: InvalidRequestError(message='The model `gpt-4-32k-0314` does not exist or you do not have access to it. Learn more: https://help.openai.com/en/articles/7102672-how-can-i-access-gpt-4.', param=None, co
-
llama-v2-7b-chat-fw
Hello! My name is Assistant, and I'm here to help you with any questions or concerns you may have. I
-
llama-v2-13b-chat-fw
Hello! My name is LLaMA, I'm a helpful and respectful assistant developed by Meta AI. I'm here to as
-
llama-v2-70b-chat-fw
Hello! My name is Assistant, and I'm here to help you with any questions or concerns you may have. I
-
mixtral-8x7b-fw-chat
My name is Mistral 7B with 8 Experts MoE model. I am a large language model created by Mistral.ai an
-
mixtral-8x7b-fw-llm
Where are you from?
I was born in Los Angeles, but grew up in Phoenix, Arizona. I moved back to L
-
claude-2
My name is Claude.
-
claude-2.1
My name is Claude.
-
claude-instant-1.2
My name is Claude.
-
claude-instant-1
My name is Claude.
Slicing#
Slicing notation
model_registry[:3]
| Name | Type | Provider | Description |
|---|---|---|---|
| gpt-3.5-turbo-1106 | chat | openai | The latest GPT-3.5 Turbo model with improved instruction following, JSON mode, reproducible outputs, parallel function calling, and more. Returns a maximum of 4,096 output tokens. |
| gpt-3.5-turbo | chat | openai | Currently points to gpt-3.5-turbo-0613. |
| gpt-3.5-turbo-16k | chat | openai | Currently points to gpt-3.5-turbo-0613. |
Filtering#
Filtering
model_registry.filter(provider="fireworks")
| Name | Type | Provider | Description |
|---|---|---|---|
| llama-v2-7b-chat-fw | chat | fireworks | 7b parameter LlamaChat model |
| llama-v2-13b-chat-fw | chat | fireworks | 13b parameter LlamaChat model |
| llama-v2-70b-chat-fw | chat | fireworks | 70b parameter LlamaChat model |
| mixtral-8x7b-fw-chat | chat | fireworks | 8x7b parameter mixture of experts Mistral model, adapted for Chats |
| mixtral-8x7b-fw-llm | llm | fireworks | 8x7b parameter mixture of experts Mistral model |