How to Monitor Your Machine Learning Models


machine learning monitoringIn our previous blog we discussed how to deploy machine learning models into production successfully and efficiently. Getting models into production can be difficult, but that isn’t the only challenge you will face with machine learning models throughout their lifetime. Once the model has made it into production, it must be monitored in order to ensure that everything is working properly. There are many different roles involved in the getting the model into production. Similarly, the monitoring of each machine learning model requires the attention from many different perspectives to ensure that each aspect of the model is running accurately and efficiently.  Let’s take a closer look into the different perspectives we must consider when monitoring machine learning models, and why each is so important:

  1. Monitoring the machine learning model from a data science perspective.

When data scientists are monitoring their machine learning models, they are primarily checking for one thing: drift. Drift means that the data is no longer relevant or useful to the problem at hand. Because data is always changing, drift occurs naturally. Data scientists must monitor the machine learning models to ensure that the model inputs look similar to those used in training. If they do not, the data may be drifting, which signifies that the data is out of date or no longer relevant.

  1. Monitoring the machine learning model from the operational perspective.

On the operational side, it’s important to keep an eye out for the amount of resource consumption that is occurring, including the CPU, memory, disk, and network I/O. These are signals as to how efficiently the model is running. Other key performance indicators on the operational side are latency and throughput. Latency is the delay before a transfer of data begins following an instruction for its transfer, while throughput is the amount of data successfully moved from one place to another in a given time period. These are important aspects to monitor to ensure that everything is.

  1. Monitoring the machine learning model from a cost perspective.

Many of Open Data Group customers need to monitor their analytic model performance  in terms of records/second. Although this gives some insight into the efficiency of the model, companies should also be focused on the benefit they gain from the model vs. the cost. This is why we recommend monitoring the records/second/cost of your models. With this information you can keep an eye out for how much this model is costing you, and whether the value generated form the model is worth the cost.

  1. Monitoring the machine learning model from a service perspective.

Many core business processes require a service level agreements (SLAs). For example, software companies may commit to a 4 hour response time for critical bug fixes. When monitoring your analytic, and the entire analytic workflow, it’s important to establish, monitor, and meet the SLAs that are agreed for business success.   SLAs for analytics might include the maximum time it takes to create a model, deploy a model, and / or iterate on a model that’s in production.

Machine learning models can be extremely valuable. One key to maintain the value comes from properly monitoring the deployed model. At Open Data Group, we understand the how important monitoring models is in the deployment process. We work with our customers to leverage FastScore, and ensure their machine learning models are deployed and monitored for the best business outcomes.


All ModelOp Blog Posts 

ModelOp Golden Ale Takes a Holiday – Part 2

ModelOp Golden Ale Takes a Holiday – Part 2

2 Minute Read By Greg Lorence Before we go much further, I feel obligated to state what is likely already obvious: I’m not all about that #InstaLife. All accompanying photography was snapped with little regard for composition, typically while stretching out from 4-6...

Q&A with Ben Mackenzie, AI Architect

Q&A with Ben Mackenzie, AI Architect

2 Minute Read By Ben Mackenzie & Linda Maggi How AI Architects are the Key to Operationalize and Scale Your AI Initiatives Each week we meet more and more clients who are realizing the importance of operationalizing the AI model lifecycle and who are dismissing...

Behind the scene of ModelOp by our Brewmasters- Part1

Behind the scene of ModelOp by our Brewmasters- Part1

2 Minute Read By Greg Lorence As a long-time homebrewer, when our President, Scott asked me, “wouldn’t it be cool if you and Jim brewed a beer to commemorate our rebrand later this year?” my reaction, after the immediate “heck yeah! Beer is awesome”, was honestly...

Open Data Group Officially Becomes ModelOp

Open Data Group Officially Becomes ModelOp

2 Minute Read By ModelOp Today, Open Data Group rebrands as ModelOp. Read more on Globe Newswire It is an exciting day for us, if only because people will stop asking “Why are you called Open Data Group?” after they understand what we do. More importantly the name...

Gartner & WIA Conferences Exit Poll

Gartner & WIA Conferences Exit Poll

2 Minute Read By Garrett Long As we continue into our “Year of Model Operations”, I thought it would be useful to highlight some of the key things I observed, learned and shared over the last few weeks at both the Gartner Data and Analytics Summit March 18-21, 2019 in...

Machine Learning Model Interpretation

To either a model-driven company or a company catching up with the rapid adoption of AI in the industry, machine learning model interpretation has become a key factor that helps to make decisions towards promoting models into business. This is not an easy task --...

Matching for Non Random Studies

Experimental designs such as A/B testing are a cornerstone of statistical practice. By randomly assigning treatments to subjects, we can test the effect of a test versus a control (as in a clinical trial for a proposed new drug) or can determine which of several web...

Distances And Data Science

We're all aware of what 'distance' means in real-life scenarios, and how our notion of what 'distance' means can change with context. If we're talking about the distance from the ODG office to one of our favorite lunch spots, we probably mean the distance we walk when...