Optimizing Data Performance with Index Archiving

As your application grows in usage, the amount of data needed to support it typically grows in parallel. To help manage this, Metal provides index archiving to efficiently store data at scale.

This post will explain how index archiving works, how it can reduce costs, and how you can use it to better understand your users behavior. Let’s begin!

What is Index Archiving?

Index archiving allows you to manage how much data you store in Metal by temporarily shutting down indexes that are not in use. This allows you to optimize the usage of your apps, helping you save costs and make the most of your resources.

An index is a collection of data that has been processed by Metal to enable fast and accurate retrieval for LLMs. Depending on your subscription with us, there's a cap on the number of indexes and embeddings you can maintain. Naturally, this threshold is influenced by your data volume. When you reach the limit, you need to either delete some data or upgrade your plan.

But what if you don't want to delete or upgrade? That's where index archiving comes in handy. Think of it like a house: you'd turn off the lights in unused rooms to save electricity. Similarly, with indexes you're not using, you can 'turn them off' by archiving. When you need them again, just like flipping on a light switch, you can quickly unarchive and access them.

The best part is that unarchiving is fast and easy. You don't need to generate the embeddings again, which can take a lot of time and resources. Metal preserves the embeddings and metadata of your archived indexes, so you can restore them in a fraction of the time. Indexes will also be unarchived when a user makes a query against it, ensuring their needs are met while storing your data efficiently.

Benefits of Index Archiving

Managing your indexes in Metal provides a few advantages:

• Save costs: You only pay for the storage of live indexes. Archived indexes do not count towards your storage limit.

• Optimize usage: You can archive indexes that are not relevant to your current projects or goals, and unarchive them when they become useful again.

• Gain insights: You can monitor how often your indexes are being used, and decide whether they are worth keeping or deleting.

Take one of Metal’s customers, for example. Onboard AI provides an easy way to help developers understand code repositories through a chat interface.

“At Onboard AI, we store thousands of repositories to help our users navigate and understand any codebase through chat. With this much data, index archiving is key to managing our storage costs as we scale” - Vaishant Kameswaran, Onboard AI

For Onboard AI, index archiving provides peace of mind that their costs will not increase unsustainably as they grow.

Understanding Index Statuses

Before jumping into how to archive an Index, let's get familiar with the different statutes an Index can have:

• Live: The index is ready to use. You can search and index new data.

• Archived: The index has been stored after 30 days of inactivity. No search or indexing has occurred during this time. You can unarchive it at any time.

• Unarchived: The index has been reactivated and is ready to be reindexed and used again.

• Reindexing: The index is currently being updated and reindexed after being unarchived.

• Deactivating: The index is in the process of being archived.

How to Archive and Unarchive Indexes

There are two ways to archive and unarchive indexes in Metal: using the UI or using the API.

Using the UI

To archive an index using the UI, follow these steps:

1. Go to the Metal dashboard and select the index you want to archive.
2. Click on the Settings tab and scroll down to the Archive section.
3. Click on the Archive button and confirm your action.

Using the API

To archive an index using the API, use the update_index method with the status parameter set to DEACTIVATING.

Example:

metal.update_index('index_id', {'status': 'DEACTIVATING'})

To unarchive an index, set the status parameter to 'UNARCHIVED'.

metal.update_index('index_id', {'status': 'UNARCHIVED'})

Wrapping UP

By leveraging the power of index archiving in Metal, you can optimize your data performance, reduce costs, and gain more control over your data management. Take advantage of this feature to unlock the full potential of your app and ensure efficient utilization of your resources.

Feel free to reach out if you have any questions or need further assistance with index archiving.