A list of messages comprising the conversation so far.
modelstringRequired
The identifier of the model to use. Can be a shared tenancy or custom model identifier.
streamtrueRequired
If set, tokens will be streamed incrementally to users. One or more tokens will be sent as server-sent events (SSE) as they become available. For more information on read above.
frequency_penaltydoubleOptionalDefaults to 0
Penalizes new tokens based on their frequency in the generated text so far.
ignore_eosbooleanOptional
Whether to ignore the EOS token and continue generating tokens after the EOS token is generated.
log_promptbooleanOptional
OctoAI stores request prompt if True.
logit_biasmap from strings to optional doublesOptional
Modify the likelihood of specified tokens appearing in the completion. Accepts a JSON object that maps tokens (specified by their token ID in the tokenizer) to an associated bias value from -100 to 100. Mathematically, the bias is added to the logits generated by the model prior to sampling. The exact effect will vary per model, but values between -1 and 1 should decrease or increase likelihood of selection; values like -100 or 100 should result in a ban or exclusive selection of the relevant token. As an example, you can pass {'50256': -100} to prevent the <|endoftext|> token from being generated.
loglikelihoodbooleanOptional
Return log probabilities for all prompt tokens excluding the first one from prefill step if True.
logprobsbooleanOptional
Whether to return log probabilities of the output tokens or not. If true, returns the log probabilities of each output token returned in the content of message.
max_tokensintegerOptionalDefaults to 512
Maximum number of tokens to generate per output sequence.
min_pdoubleOptionalDefaults to 0
Sets a dynamic threshold of the top tokens to consider based on the probability value of the most likely next token. Following https://arxiv.org/abs/2407.01082.
nintegerOptionalDefaults to 1
Number of output sequences to return.
peftstringOptional
Parameter-efficient fine-tuning ID.
presence_penaltydoubleOptionalDefaults to 0
Penalizes new tokens based on whether they appear in the generated text so far.
repetition_penaltydoubleOptionalDefaults to 1
Controls the likelihood of the model generating repeated texts.
response_formatobjectOptional
Allows specification of a response format and associated schema that will constrain the LLM output to that structure. For example, using the json_object type allows you to provide a desired json schema for the output to follow.
stopstring or list of stringsOptional
Generation stop condition.
stream_optionsobjectOptional
If set, usageStats will be streamed on the last content-containing chunk.
temperaturedoubleOptionalDefaults to 1
Controls the randomness of the sampling.
tool_choice"auto" or "required" or objectOptional
Controls which (if any) tool is called by the model. none means the model will not call any tool and instead generates a message. auto means the model can pick between generating a message or calling one or more tools. required means the model must call one or more tools. Specifying a particular tool via {"type": "function", "function": {"name": "my_function"}} forces the model to call that tool. none is the default when no tools are present. auto is the default if tools are present.
toolslist of objectsOptional
A list of tools the model may call. Currently, only functions are supported as a tool. Use this to provide a list of functions the model may generate JSON inputs for. A max of 128 functions are supported.
top_logprobsintegerOptionalDefaults to 0
An integer between 0 and 20 specifying the number of most likely tokens to return at each token position, each with an associated log probability. logprobs must be set to true if this parameter is used.
top_pdoubleOptionalDefaults to 1
Controls the cumulative probability of the top tokens to consider.
userstringOptional
A unique identifier representing your end-user.
Response
This endpoint returns a stream of object
choiceslist of objects
A list of chat completion choices.Can be more than one if n is greater than 1.
createdinteger
The Unix timestamp (in seconds) of when the chat completion was created.
idstring
A unique identifier for the entire chat completion request. Each chunk in the stream has the same ID.