LangChain Integration

Introduction

LangChain provides a framework to easily build LLM-powered apps. Developers using LangChain can now utilize OctoAI LLMs and Embedding endpoints to access efficient, fast, and reliable compute.

Using OctoAI’s LLMs and LangChain

To use OctoAI LLMs with LangChain, first obtain an OctoAI API Token. Then paste your API token in the code example below:

1 import os
2 
3 os.environ["OCTOAI_API_TOKEN"] = "OCTOAI_API_TOKEN"
4 os.environ["ENDPOINT_URL"] = "https://text.octoai.run/v1/chat/completions"

Next, run the following Python script:

1 from langchain.chains import LLMChain
2 from langchain_community.llms.octoai_endpoint import OctoAIEndpoint
3 from langchain_core.prompts import PromptTemplate
4 
5 template = """Below is an instruction that describes a task. Write a response that appropriately completes the request.\n Instruction:\n{question}\n Response: """
6 prompt = PromptTemplate.from_template(template)
7 
8 llm = OctoAIEndpoint(
9     model_kwargs={
10         "model": "meta-llama-3-8b-instruct",
11         "max_tokens": 128,
12         "presence_penalty": 0,
13         "temperature": 0.1,
14         "top_p": 0.9,
15         "messages": [
16             {
17                 "role": "system",
18                 "content": "You are a helpful assistant. Keep your responses limited to one short paragraph if possible.",
19             },
20         ],
21     },
22 )
23 
24 question = "Who was leonardo davinci?"
25 
26 llm_chain = LLMChain(prompt=prompt, llm=llm)
27 
28 print(llm_chain.run(question))

It should produce the following output:

  Sure thing! Here's my response:
Leonardo da Vinci was a true Renaissance man - an Italian polymath who excelled in various fields,
including painting, sculpture, engineering, mathematics, anatomy, and geology. He is widely considered
one of the greatest painters of all time, and his inventive and innovative works continue to inspire and
influence artists and thinkers to this day. Some of his most famous works include the Mona Lisa,
The Last Supper, and Vitruvian Man.

Learn with our demo apps

Get started today by following along with one of our demo apps:

1	import os
2
3	os.environ["OCTOAI_API_TOKEN"] = "OCTOAI_API_TOKEN"
4	os.environ["ENDPOINT_URL"] = "https://text.octoai.run/v1/chat/completions"

1	from langchain.chains import LLMChain
2	from langchain_community.llms.octoai_endpoint import OctoAIEndpoint
3	from langchain_core.prompts import PromptTemplate
4
5	template = """Below is an instruction that describes a task. Write a response that appropriately completes the request.\n Instruction:\n{question}\n Response: """
6	prompt = PromptTemplate.from_template(template)
7
8	llm = OctoAIEndpoint(
9	model_kwargs={
10	"model": "meta-llama-3-8b-instruct",
11	"max_tokens": 128,
12	"presence_penalty": 0,
13	"temperature": 0.1,
14	"top_p": 0.9,
15	"messages": [
16	{
17	"role": "system",
18	"content": "You are a helpful assistant. Keep your responses limited to one short paragraph if possible.",
19	},
20	],
21	},
22	)
23
24	question = "Who was leonardo davinci?"
25
26	llm_chain = LLMChain(prompt=prompt, llm=llm)
27
28	print(llm_chain.run(question))