Model assessment and model traceability are two crucial steps when deploying a machine learning model. Model assessment ensures that the model is running accurately and efficiently, while model traceability deals with the history of the machine learning model. These two components are critical for deployment, but how can we make sure we are successful in these areas? Today we are going to dive into model assessment and model traceability, and discuss the importance of these processes.
Let’s begin with model assessment. There are two main pieces to assessing a machine learning model. The first piece is model performance. This is basically a matter of whether or not the math is doing its job. The data science team ensures this in a couple of different ways. One way is by using ROC (Receiver Operating Characteristics) curves. An ROC curve is a probability curve that can be used to measure AUC (Area Under the Curve), which represents the degree of separability the model has. The higher the AUC, the better the model is performing. ROC curves define if the model is doing a good job.
Another way to ensure the math is doing its job is with a . Confusion matrices are only suitable for classifiers, not regressors. In other words, a confusion matrix can only be used for models that predict class labels, not numbers. Confusion matrices can be useful to see how often the model is accurate, as well as how often the model is wrong. They can also be used to look at true positive and false positive rates for the model. These tests allow data scientists to make sure the model is predicting for what it should be, and gives them high confidence when inputting a new line of data.
The second piece of model assessment is model telemetry, which speaks more towards the true operational characteristics of a model. This includes latency and throughput, how much memory the model is using, if it is limited by data throughput or other constraints, and if so, can it be improved with computational resources. Answering these questions can help assure the IT and application teams that the model is running efficiently.
Next, we are going to dive into what it means to trace a model. There are many words used to describe model traceability, such as history, lineage, and metadata. Model traceability refers not only the history of the model, such as who created the model and when, but also includes how the model was trained, in what context was the model used, and if there are any aspects of errors, acceptance, and performance that need to be captured and understood.
Model traceability is about being able to trace exactly where a model has been and what it’s been doing, both backwards in time and across the various environments it may pass through. This could mean creation, pre-production, or production environments. You can think of it as analogous to a “history” or “audit trail” for the model. This feature is mainly useful for understanding what changes have occurred for a model, which is specifically valuable for long-running or frequently updated models. Model traceability is also useful as for legal and compliance reasons and related questions of security and ownership. For example, traceability allows you to know what specific actions certain people did to the machine learning model.
Proper model assessment and traceability capabilities lead to a more organized and efficient model deployment process. For enterprise grade model operations, systems must provide insights to each of these avenues, integrating data and insights across the Model Development Life Cycle. It’s imperative to build systems for model assessment and traceability that are agnostic to modeling environments, and can capture data from the myriad of systems used in the deployment of models. FastScore offers microservices such as FastScore Engine and Manage that incorporate model assessment and traceability into the deployment process natively, making the model easy to keep track of throughout production. On top of this, FastScore adds efficiency to the entire model operations workflow, allowing machine learning models to be deployed and optimized 10x faster than typical enterprise processes.
