In this part, we:
- Log into the Openlayer platform.
- Explain what are projects.
- Explain the tests that can be created inside a project.
Log into Openlayer
Please navigate and log into the Openlayer platform before starting this part of the walkthrough.
After running the notebook from the previous part and logging into the Openlayer platform, the first thing you see is the “Churn Prediction” project and the artifacts uploaded (i.e., the model, and the datasets).
What are Openlayer projects?
A project is the house of model development on the Openlayer platform.
Each project is versioned, which means that you can keep track of changes to models and datasets as you push them. This allows you to maintain a history of your project and enables collaboration with other team members.
Now, it’s time to start evaluating and improving our model. To avoid relying on trial and error as the main driver for model development, we need to start by identifying the issues affecting our model. Enter the world of Openlayer tests.
What are Openlayer tests?
Openlayer tests define guardrails set up to ensure our data and models meet specific standards.
For now, let’s focus on how the test-creation process can help us identify issues negatively affecting our model.
Scroll down to the “Create your first test” section.
Notice how Openlayer automatically suggests multiple tests that run on the artifacts uploaded. If you click the “Create tests” button, all of these will be created, giving you a head start on your model development journey.
Here, we will explore the tests manually, to understand their nuances. Click the “Explore all tests” button. Feel free to spend some time browsing through the different test-types, using the left sidebar.
Note how there are issues that likely have ripple effects on the model’s performance. To mention a few, notice there are duplicate rows in the training set, quasi-constant features, feature drift, and more.
For now, the supported test types are:
- Data integrity tests: assesses the data quality of the training and validation sets.
- Data consistency tests: related to the consistency between the training and validation data.
- Performance tests: guardrails on the model’s performance on the whole validation set or specific subpopulations of it.
- Fairness tests: assess the model using different fairness metrics.
- Robustness tests: assert robustness to different data perturbation recipes.
As a final comment, notice how there is a hierarchy between the test types. Our recommended approach involves beginning with the easily achievable data integrity tests, before moving on to each successive test type and ultimately concluding with robustness.
Let’s move to the next part of the tutorial, where we will start creating tests that help us build a high-quality model!