LangChain Integration

Langchain developers can leverage OctoAI LLM and embedding endpoints to easily access efficient compute across a wide selection of LLMs.


LangChain provides a framework to easily build LLM-powered apps. Developers using LangChain can now utilize OctoAI LLMs and Embedding endpoints to access efficient, fast, and reliable compute.

Using OctoAI’s LLMs and LangChain

To use OctoAI LLMs with LangChain, first obtain an OctoAI API Token. Then paste your API token in the code example below:

1import os
4os.environ["ENDPOINT_URL"] = ""

Next, run the following Python script:

1from langchain.chains import LLMChain
2from langchain_community.llms.octoai_endpoint import OctoAIEndpoint
3from langchain_core.prompts import PromptTemplate
5template = """Below is an instruction that describes a task. Write a response that appropriately completes the request.\n Instruction:\n{question}\n Response: """
6prompt = PromptTemplate.from_template(template)
8llm = OctoAIEndpoint(
9 model_kwargs={
10 "model": "meta-llama-3-8b-instruct",
11 "max_tokens": 128,
12 "presence_penalty": 0,
13 "temperature": 0.1,
14 "top_p": 0.9,
15 "messages": [
16 {
17 "role": "system",
18 "content": "You are a helpful assistant. Keep your responses limited to one short paragraph if possible.",
19 },
20 ],
21 },
24question = "Who was leonardo davinci?"
26llm_chain = LLMChain(prompt=prompt, llm=llm)

It should produce the following output:

Sure thing! Here's my response:
Leonardo da Vinci was a true Renaissance man - an Italian polymath who excelled in various fields,
including painting, sculpture, engineering, mathematics, anatomy, and geology. He is widely considered
one of the greatest painters of all time, and his inventive and innovative works continue to inspire and
influence artists and thinkers to this day. Some of his most famous works include the Mona Lisa,
The Last Supper, and Vitruvian Man.

Learn with our demo apps

Get started today by following along with one of our demo apps: