The development process of AI/ML models is inherently iterative. While rapid iterations are crucial, things can slip out of hand, and the next thing you know, you are repeatedly introducing and fixing the same issues.
You can use Openlayer to avoid running in circles during development. It all starts with versioning.
In this guide, we focus on the versioning aspects of Openlayer. We dive deep into what exactly is being versioned and how to onboard these artifacts to the platform.
Every iteration of development produces artifacts. Generally, these artifacts are the training and validation datasets, and the resulting model. The bundle with all the artifacts produced during an iteration is called a commit bundle. The figure below illustrates a few commit bundles for a sample project:
When we talk about versioning on the Openlayer platform, we are referring to versioning the commit bundles. Versioning the commit bundles is the best way to ensure full reproducibility. With this approach, it is possible to navigate history and have a full, self-contained view of everything involved in model development. The idea is to push the commit bundle to your Openlayer project after every round of development.
With the definition out of the way, you might be wondering how to push commit bundles to the platform. That’s what we explore in the next section.
Uploads are done using the
openlayer Python client. The step-by-step of the commit bundle upload process — including code snippets — is laid out in the How to upload datasets and models for development guide.
In broad strokes, the process starts by adding the datasets and models to a project’s staging area with the
add_* methods (for example,
add_dataset), then adding a commit message with
commit, and finally pushing (uploading) the bundle to Openlayer with
Here, we explore the nuances of adding datasets and models.
You only need two things to add a dataset to the commit bundle that will get pushed to Openlayer:
- The dataset, which can be a pandas dataframe or a CSV file, and
- The dataset configuration, which can be a Python dictionary or a YAML file. The dataset config contains dataset metadata, and you can refer to the How to write dataset config guides to check out the requirements for each task type.
Then, adding a dataset is as simple as using the code snippets from the How to upload datasets and models for development guide.
After adding the dataset, if you call the
project.status() method, you should see either
validation datasets in the staging area.
Important detail for uploading models as part of the same commit bundle.
In this case, you must ensure the dataset has a column with the model output for every row. Furthermore, this column needs to be signaled in the dataset config. For example, for LLMs, the column with the model outputs needs to be referenced in the
outputColumnName field of the dataset config.
The Write dataset configs guides make the fields clear for each task type.
When it comes to adding models, there are a few paths you can follow depending on your intended use within the platform. To facilitate your decision, consider using the flowchart below:
Let’s now explore each possibility, from the simplest to the most complex.
A shell model is comprised of only metadata. You signal that you are uploading a shell model by calling the
add_model method with just the model configuration as a parameter.
The Openlayer performance tests for shell models happen based on the model output, uploaded as part of the datasets.
For most use cases, shell models are the best choice. Note how the shell model option supports any workflow you use, no matter how complex. Here, we only care about the model outputs and not about how the outputs were generated. The model metadata is only required so that, later, you can backtrack the model used.
The direct-to-API approach only makes sense if you are working with LLMs. Furthermore, this path is only recommended if you want to interact with the model on the platform and run inference on data that is not part of the dataset you upload. Otherwise, you can use the shell model approach without issues.
The direct-to-API approach is suitable if you only need to call an LLM provider API (such as OpenAI, Anthropic, Cohere) with a prompt injected with data to get your model outputs. No chains or other intermediate API calls involved.
The easiest way to follow the direct-to-API approach is to use Openlayer’s UI. You can also call the
add_model method with a model configuration. However, in this case, you must ensure the config contains all the information needed to make the API requests, such as the
prompt. Here is a notebook example that uses this approach.
For traditional ML tasks (such as tabular classification, tabular regression, and text classification), the full model enables explainability. Explainability techniques, such as LIME and SHAP, require running synthetic data samples through the model — that’s why using a shell model is not enough in these situations.
To add a full model for traditional ML, we must prepare a model package. A model package is nothing more than a folder with all the necessary information to run inference with the model.
The package should include the following:
requirements.txtfile listing the dependencies for the model.
- Serialized model files, such as model weights, encoders, etc., in a format specific to the framework used for training (e.g.
.pbfor TensorFlow, and so on.)
prediction_interface.pyfile that acts as a wrapper for the model and implements the prediction function.
Other than the model package, a model config is needed, with information about the model for the Openlayer platform.
For full model upload examples — including samples of each of the model package components — refer to the examples gallery.
Full model uploads for LLMs is coming soon!
Navigating between versions
After a commit bundle gets pushed to the platform, it gets evaluated against the set of tests created. The next topic guide covers everything there is to know about Openlayer tests.
Furthermore, you get to easily navigate between versions to understand the project at a certain point in time and visualize the tests passing/failing. You can also download the artifacts from a specific version.