Simple Chat
Learn how to build a simple Python chat app using Llama 2 70B Chat on OctoAI and LangChain. It’s a slimmed down question and answer app. If you are already a Python developer, or looking to start building new apps with LLMs, then this is a great starting place.
Build and modify our Simple Chat app
We built the Simple Chat app purposefully more barebones to help folks get started using LLMs in app development. This simple app was made so you could expand upon it with your different use cases. There are plenty of ways to improve upon this app: connecting to a vector DB to augment the app responses, using a fine-tuned LLM for more specific needs, and adding memory so the app can hold a conversation.
Prerequisites to get started
Before trying to setup or run this app you will need to do the following:
Install Python and dependencies
Create an OctoAI API token
Import the OctoAI endpoint in LangChain via the following:
If you have used other LLM providers, notice this should allow for a quick find and replace for other LLM endpoints you might have already used.
To use the OctoAI Llama 2 70B add the following lines:
The above simply:
Sets the model to llama2-70b-chat
Defines the LLM behavior
Shows streaming mode is off
Sets max tokens to 256 (for concise output)
Lastly, you need to set up the language model chain, with the prompt string being stored from the user’s input:
Start building
Setup Python virtual environment:
Install the pip requirements in your local Python virtual environment:
Environment setup
Grab your previously generated OctoAI API token, and paste in the .env
file.
Run the app
You need to execute the main script in your terminal.
Run
chat.py
to chat with the LLM endpoint:
Share what you build
We are excited to see what you build – feel free to show us in our Discord, and see what other community members are creating.
Your choice of models on our SaaS or in your environment
Run any model or checkpoint on our efficient, reliable, and customizable API endpoints. Sign up and start building in minutes.