Migrate from OpenAI in 3 lines of code

If you've been using GPT-3.5 or GPT-4 with the OpenAI API standard, switching to OctoAI is easy!

OctoAI LLMs are available to use through our compatible OpenAI API. Additionally, if you have been building or prototyping using OpenAI’s Python SDK you can keep your code as-is and use OctoAI’s LLM models.

In this example, we will show you how to change just three lines of code to make your Python application use OctoAI’s Open Source models through OpenAI’s Python SDK.

What you will build

Migrate OpenAI’s Python SDK example script to use OctoAI’s LLM endpoints.

These are the three modifications necessary to achieve our goal:

  1. Redefine OPENAI_API_KEY your API key environment variable to use your OctoAI key.
  2. Redefine OPENAI_BASE_URL to point to https://text.octoai.run/v1
  3. Change the model name to an Open Source model, for example: llama-2-13b-chat


We will be using Python and OpenAI’s Python SDK.


  • Set up a Python virtual environment. Read Creating Virtual Environments here.
$python3 -m venv .venv
>source .venv/bin/activate
  • Install the pip requirements in your local python virtual environment
$python3 -m pip install openai

Environment setup

To run this example, there are simple steps to take:

  • Get an OctoAI API token by following these instructions.
  • Expose the token in a new OCTOAI_TOKEN environment variable:
$export OCTOAI_TOKEN=<your-token>
  • Switch the OpenAI token and base URL environment variable
>export OPENAI_BASE_URL="https://text.octoai.run/v1"

If you prefer, you can also directly paste your token into the client initialization.

Example code

Once you’ve completed the steps above, the code below will call OctoAI LLMs:

1from openai import OpenAI
3client = OpenAI()
5completion = client.chat.completions.create(
6 # model="gpt-3.5-turbo",
7 model="meta-llama-3-8b-instruct",
8 messages=[
9 {"role": "system", "content": "You are a helpful assistant."},
10 {"role": "user", "content": "Hello!"},
11 ],

Note that you need to supply one of OctoAI’s supported LLMs as an argument, as in the example above. For a complete list of our supported LLMs, check out our REST API page.

Example output

The code above produces the following object:

1ChatCompletionMessage(content=" Hello! How can I assist you today? Do you have any questions or tasks you'd like help with? Please let me know and I'll do my best to assist you.", role='assistant' function_call=None, tool_calls=None)