TypeScript SDK Fine-tuning

How to create a fine-tuned LoRA using OctoAI's TypeScript SDK

This guide will walk you through creating a fine-tuned LoRA using our TypeScript SDK to upload image file assets, creating a tune, and then using that LoRA to run an inference using our Image Gen service once it’s ready.

Please see Fine-tuning Stable Diffusion for more specifics about each parameter in the fine-tuning API, using curl, or the Python SDK. Our Asset Library API reference documentation goes more into the specifics of using different asset methods as well.

Requirements

  • Please create an OctoAI API token if you don’t have one already.
  • Please also verify you’ve completed TypeScript SDK Installation & Setup.
  • If you use the OCTOAI_TOKEN environment variable for your token, you can instantiate the client with const octoai = new OctoAIClient() or pass the token as a parameter to the constructor like const octoai = new OctoAIClient({ apiKey: process.env.OCTOAI_TOKEN }).
  • An account and API token is required for all the following steps.

High-level steps to creating a fine-tuned LoRA

In order to run a LoRA fine-tuning job, you need to complete a few steps:

  1. Create image file assets using the Asset Library, then wait for those assets’ status to be ready
  2. Either create a checkpoint asset you would like to use or get one from OctoAI’s public checkpoints.
  3. Create a tune job, then wait for the status to be succeeded.
  4. Run an inference with the new LoRA.

Directions with all the code put together are included at the bottom of the document, but at each step we will cover additional information.

1) Creating image file assets

Asset Library in the TypeScript SDK covers more specifics about the methods, so this example will be focused on a code snippet for uploading multiple files from a folder at once.

In this example, we will use multiple photos of a toy poodle named Mitchi.

TypeScript
1import fs from "node:fs";
2import { OctoAIClient } from "@octoai/sdk";
3
4const octoai = new OctoAIClient({
5  apiKey: process.env.OCTOAI_TOKEN,
6});
7
8const fileNames = fs.readdirSync("./images");
9
10console.log("Uploading files...");
11
12const assets = await Promise.all(
13    fileNames.map((fileName) => {
14        const imageName = fileName.split(".")[0];
15
16        return octoai.assetLibrary.upload(`./images/${fileName}`, {
17            name: imageName,
18            description: imageName,
19            assetType: "file",
20            data: {
21                assetType: "file",
22                fileFormat: "jpg",
23            },
24        });
25    })
26);
27
28console.log("Waiting for assets to be ready...");
29
30for (const { asset } of assets) {
31    await octoai.assetLibrary.waitForReady(asset.id);
32    console.log(`Asset "${asset.name}" is ready`);
33}

After this completes, all assets will hopefully be in the ready state, or you should time out. Mitchi is now on OctoAI!

astropus.png

2) Get a checkpoint asset to use for tuning our LoRA

Next, you’ll need a checkpoint to use to tune your asset. In this example, we will just use the default checkpoint using Stable Diffusion XL, but you can also use other public OctoAI checkpoints or create your own using the Asset Library.

TypeScript
1const checkpoint = await octoai.assetLibrary.get("octoai:default-sdxl");

3) Creating a tune

We can create a tune by passing in the id of the checkpoint we’d like to use and the ids of the file assets that we created in Step 1. If you want more accurate results, you can add captions to each image to give more thorough descriptions. If no custom captions are provided, the trigger word will be used as a default.

TypeScript
1console.log("Creating tune...");
2
3let tune = await octoai.fineTuning.create({
4    name: "mitchi",
5    description: "mitchi",
6    details: {
7        tuneType: "lora_tune",
8        baseCheckpoint: {
9            checkpointId: checkpoint.asset.id,
10        },
11        // You can add a `caption` to each file for more accurate results
12        files: assets.map(({ asset }) => ({ fileId: asset.id })),
13        steps: 10,
14        triggerWords: ["sksmitchi"],
15    },
16});
17
18console.log("Waiting for tune to be ready...");
19
20while (tune.status !== "failed" && tune.status !== "succeeded") {
21    await new Promise((resolve) => setTimeout(resolve, 1000));
22    tune = await octoai.fineTuning.get(tune.id);
23}

4) Run an inference with the tuned LoRA

Next, you can run an inference with the tuned LoRA

TypeScript
1console.log("Generating image...");
2
3const { images } = await octoai.imageGen.generateSdxl({
4    prompt: "A photo of an sksmitchi as a puppy",
5    negativePrompt: "Blurry photo, distortion, low-res, poor quality, extra limbs, extra tails",
6    loras: {
7        mitchi: 0.8,
8    },
9    numImages: 1,
10});
11
12images.forEach((image, index) => {
13    if (image.imageB64) {
14        const buffer = Buffer.from(image.imageB64, "base64");
15        fs.writeFileSync(`result${index}.jpg`, buffer);
16    }
17});

The end result will be a saved poodle to your local folder.

Stable Diffusion Tuned LoRA generated toy poodle puppy

result0.jpg

Putting it all together: From Asset Creation to Running an Inference with Tuned LoRA

TypeScript
1import { writeFileSync } from "node:fs";
2import { OctoAIClient } from "@octoai/sdk";
3
4const octoai = new OctoAIClient({
5    apiKey: process.env.OCTOAI_TOKEN,
6});
7
8const fileNames = fs.readdirSync("./images");
9
10console.log("Uploading files...");
11
12const assets = await Promise.all(
13    fileNames.map((fileName) => {
14        const imageName = fileName.split(".")[0];
15
16        return octoai.assetLibrary.upload(`./images/${fileName}`, {
17            name: imageName,
18            description: imageName,
19            assetType: "file",
20            data: {
21                assetType: "file",
22                fileFormat: "jpg",
23            },
24        });
25    })
26);
27
28console.log("Waiting for assets to be ready...");
29
30for (const { asset } of assets) {
31    await octoai.assetLibrary.waitForReady(asset.id);
32    console.log(`Asset "${asset.name}" is ready`);
33}
34
35const checkpoint = await octoai.assetLibrary.get("octoai:default-sd15");
36
37console.log("Creating tune...");
38
39let tune = await octoai.fineTuning.create({
40    name: "mitchi",
41    description: "mitchi",
42    details: {
43        tuneType: "lora_tune",
44        baseCheckpoint: {
45            checkpointId: checkpoint.asset.id,
46        },
47        files: assets.map(({ asset }) => ({ fileId: asset.id })),
48        steps: 10,
49        triggerWords: ["sksmitchi"],
50    },
51});
52
53console.log("Waiting for tune to be ready...");
54
55while (tune.status !== "failed" && tune.status !== "succeeded") {
56    await new Promise((resolve) => setTimeout(resolve, 1000));
57    tune = await octoai.fineTuning.get(tune.id);
58}
59
60console.log("Generating image...");
61
62const { images } = await octoai.imageGen.generateSdxl({
63    prompt: "A photo of an sksmitchi as a puppy",
64    negativePrompt: "Blurry photo, distortion, low-res, poor quality, extra limbs, extra tails",
65    loras: {
66        mitchi: 0.8,
67    },
68    numImages: 1,
69});
70
71images.forEach((image, index) => {
72    if (image.imageB64) {
73        const buffer = Buffer.from(image.imageB64, "base64");
74        fs.writeFileSync(`result${index}.jpg`, buffer);
75    }
76});