Getting started with our Text Gen Solution

The OctoAI Text Gen Solution offers market-leading price and performance for a growing list open source LLMs including Llama3 and Mistral (see Supported models section below). We offer a WebUI playground, API endpoints, and Python/TypeScript SDK solution for interacting with these models. All of our endpoints are callable via chat completions format currently popular in the industry (see API documentation).

Self-Service Models

Presently, OctoAI supports the following models & checkpoints for self-service models:

Meta-Llama-3.1 Meta’s latest LLM family. Llama-3.1 offers improved understanding and instruction following as compared to Llama-3.0, expands context length, supports 8 languages (English, French, German, Hindi, Italian, Portuguese, Spanish, and Thai), and includes support for function calling. Model is offered in 8B-, 70B-, and 405B-parameter variants. The 405B variant is the most powerful open source model yet released.

Mistral-8x7b-Instruct: Mistral AI’s December 2023 release, Mistral-8x7b-Instruct, is a “mixture of experts” model utilizing conditional computing for efficient token generation, reducing computational demands while improving response quality (GPT-4 is widely believed to be an MoE model). Mistral-8x7b-Instruct brings these efficiencies to the open-source LLM realm, and it is licensed for commercial use. It supports up to 32,768 tokens. Read more.

Nous-Hermes-2-Mixtral-8x7b-DPO The flagship Nous Research model trained over the Mixtral 8x7B MoE LLM. The model was trained on over 1,000,000 entries of data, as well as other high quality data from open datasets across the AI landscape, achieving state of the art performance on a variety of tasks. It supports up to 32,768 tokens.

For pricing of all of these endpoints, please refer to our pricing page.

Web UI playground

You can start familiarizing yourself with our Text Gen features using the web UI, but note that we have even more features available via the API.

First, click on the top navigation bar and click Text Tools. Here you will see the different model families that we offer for self-service users:

Click the Demo or API selections to enter our playground, where you can:

  • Easily switch between all of our models, parameter counts, and quantization settings
  • Test each model using our chat interface
  • Adjust common settings such as temperature
  • See the pricing and context limits for any selected model.

Selecting the “API” toggle will show you code samples in Python, TypeScript, and CURL format for calling the endpoint that you’ve selected, as well as key input & output parameters:

Billing

For pricing of all of these endpoints, please refer to our pricing page.

Once you provide billing information and generate an API key, any usage of these endpoints will be viewable under Accounts -> Billing & Usage -> Text Generation Usage. Note that these endpoints are very price competitive, so you’ll generally needs to rack up tens of thousands of tokens before you can see the charges!

API Docs

When you’re ready to start calling the endpoint programmatically, check out our REST API, Python SDK, and TypeScript SDK docs.