Text Gen TypeScript SDK
The OctoAI Text Gen TypeScript SDK supports both the Chat Completions API and the Completions API.
At a Glance
This guide will walk you through how to use the TypeScript SDK to call our Text Gen API. The TypeScript SDK supports streaming and non-streaming inferences for both the Chat Completions API and legacy Completions API. There are also additional parameters such as frequencyPenalty
, maxTokens
, presencePenalty
, etc. that can be used for finer control.
Requirements
- Please create an OctoAI API token if you don’t have one already.
- Please also verify you’ve completed TypeScript SDK Installation & Setup.
- If you use the
OCTOAI_TOKEN
envvar for your token, you can instantiate the client withoctoai = new OctoAIClient()
, otherwise you will need to pass an API token using:octoai = new OctoAIClient({ apiKey: process.env.OCTOAI_TOKEN })
- If you use the
Chat Completions API
Non-Streaming Example
To make a chat completions call, you will need to provide the model you wish to call and a list of chat messages.
Streaming Example
The above example can work great in some scenarios, but if you’re dealing with larger requests or are building a highly-interactive user experience, using the streaming interface may be a better choice. The available options between non-streaming and streaming inferences are identical, but there are two main code changes needed:
- You will need to use the
createChatCompletionStream()
method instead ofcreateChatCompletion()
. - Instead of grabbing the final text message from the response, you will need to loop over the individual chunks and concatenate the tokens.
Completions API
The TypeScript SDK also supports the legacy Completions API with the same customization options as the Chat Completions API. The key difference between the two is that you provide a prompt
string instead of a list of chat message objects. Much like the Chat Completions API, you can choose between non-streaming and streaming inference.