Why we made an Open Source Chatbot with Metal

Like many in the AI and LLM space, we’ve seen an explosion of one kind of application in particular: the chatbot. This is not surprising, as the launch of LLMs like ChatGPT are responsible for kickstarting the massive amount of building taking place today.

Chatbots are a great way to start learning how to use LLMs. These applications help developers understand the building blocks of the stack, incorporating the elements of data (embeddings, vector storage), querying (information retrieval) and observability (logging and analytics) into a single application.

But if you’re just getting started with the stack, having to manually code a chatbot every time can be tedious or downright inefficient. We know this from experience, as our team is hands-on with every customer – often helping users build their first application with Metal to learn the stack alongside our team. Translation: we made lots of chatbots. So we asked how we can help developers move faster.

Introducing Metal Chat OSS

Metal’s open source AI Chatbot is designed to be pulled off the shelf and directly integrated into Metal’s vector storage and information retrieval capabilities. You can think of it as a “starter kit”, helping developers ship their first LLM application in a fraction of the time.

What’s in Metal Chat today?

For starters, there would be no chatbot without chat capabilities! You can use ChatGPT-3.5 or 4, depending on your use case.

We also provide tooling to customize the chat experience.

Temperature allows you to control how creative your chatbot will be in generating responses. Lower temperature’s will keep the LLM’s answers as close to the indexed data as possible, making the answers more deterministic and focused.

Max tokens allows you to configure the maximum amount of tokens returned in a response, limiting the number of characters the LLM will output. While not a 1-1 comparison, you can think of a single token as ~4 characters in English.

Chunk Count limits the number of relevant embedding chunks that will be returned given a query. While Metal Chat can currently return up to 100 chunks, please note that this may produce an error as the total size of chunks may exceed GPT’s context window limit.

Export makes it easy to create transcripts of your chatbot’s responses to user queries. The export feature will create a .txt file of a conversation, which can be easily shared and reviewed.

Clear History allows you to clear the current conversation from the chatbot.

Prompt Libary allows you to update the system message for the chatbot. These are essentially instructions for how the chatbot should respond to queries.

The view sources feature will show you which chunks of embeddings were used to generate a response from the LLM. These are sorted by relevancy, with the most relevant chunks at the top of the list. When you select the chunk by embedding ID, you can view the entirety of the chunk’s text in Metal’s platform.

What’s does the future hold for Metal Chat?

The future is what we make it! Developers are more than welcome to contribute to this project and the goal will remain the same – helping developers build and ship LLM chat apps faster.

For our part, we will maintain the project and add new functionality as it’s released. Some of the features we have planned involve more powerful ways to index and store data, more powerful retrieval mechanisms that can work with structured data, and more.

If you would like to see a feature in this project, feel free to open an issue or drop us a line here.