How ModelOp Center Automates Model Life Cycles and Why it Matters (part 2/2)

by ModelOp Product Team –  3 minutes read

Scenarios for MLC Processes

Following up our former post on Model Life Cycles, in this blog post we will discuss different scenarios for MLC Processes.

The MLC Manager provides flexibility with how you manage and automate the various life cycles of models across the enterprise. Each model in the enterprise can take a wide variety of paths to production, have different patterns for monitoring, and have various continuous improvement or retirement steps.

MLC Processes are triggered by external events occurring within the business. Some typical examples of this include:

• a model is marked ready for productionization

• a time based event

• new data arrives in a location

• a notification is received

• a manual intervention by a user

There are additional types of signals that can start an MLC Process including custom items from your business. In fact, the ModelOp Command Center has several UI features that leverage MLC Processes under the hood, including submitting the model for productionization from the Model Details screen, executing a Batch Job from the Runtimes screen, or approving or rejecting tasks. As you can already tell, there are many ways to automatically do a variety of different things with MLC Processes. Following are some examples.

Model Productionization

MLC Processes can automate the productionization of a model, regardless of whether the path to production is simple or complex. For example, you can use an MLC Process to deploy a newly registered model into your QA runtime, run the model through a series of tests, trigger an automated security scan, and seek appropriate approvals before it is deployed into Production. MLC Processes can be created in a flexible manner to meet the needs of your team. They can be configured to automatically locate an runtime that is compatible with the current model, or a specific group of runtimes can be targeted by tags. See Deploy a Model into a ModelOp Runtime for more details on how this is accomplished. The example in On Model Changed includes these deployment pieces.

Model Refresh & Retraining

After the initial deployment, it’s important to have a way to rapidly retrain or trigger the refresh of a model to ensure it is performing optimally. Retraining can be automated within an MLC process to run on a schedule or when new labeled data is available. Using the same MLC process, the new candidate model can be compared against the current deployed model using a Champion/Challenger Model Comparison. Finally, the MLC Process can automate the steps required for Change Management including re-testing and approvals. The example On Model Changed demonstrates how you can build these operations into an MLC Process.

Throughout the MLC Process, you can include User Tasks to direct specific team members or roles to review and approve changes to the models. Users can interact with these tasks within ModelOps Center, or the User Tasks can be sent to existing ticketing systems like ServiceNow. Approvals and tasks can contain model metadata to provide context for the approval.

You can monitor models using MLC Processes by automatically running Batch Jobs on a model. You can run Batch Jobs on a schedule or based on new labeled or ground truth data becoming available. For example, you can run a Test Job with the Metrics Function to calculate the statistical efficacy or determine if it has started to produce ethically partial predictions, and then use decision criteria to determine which action to take. A common pattern is to generate an alert into ModelOp Center for the ModelOp Support Team to triage.

For more details see:

• Monitor a Deployed Model

• Model Efficacy Metrics and Monitoring

• Model Governance: Bias & Interpretability

• Model Batch Jobs and Tests

Sample MLC Processes

This section describes some specific examples of MLC Processes in detail.

On Model Changed MLC Process

On Model Changed is an MLC Process that incorporates several of the patterns described earlier in a single MLC Process for managing the creation of a model, or changes to an existing model.

1. Model Submitted – a deployable model object has been created, meaning the model is considered complete, and all associated assets are included with the model. Clicking Submit in the Model Details triggers the start of this MLC process.

2. Training – based on a metadata flag, a Training Job can be initiated to train the model. A polling task automatically checks for the Training Job to finish before proceeding.

3. Testing – a Test Job executes, and then the results are persisted in ModelOp Center.

4. Approval Based on Type – based on the metadata, you can route the model to the correct group to approve the changes. The details of the model, including all of the core information about the model, the changes to the model and the test results, are passed on to the reviewer.

5. Model Deployment – the MLC Process finds a ModelOp Runtime that is tagged with the appropriate tag, and attempts to deploy the model onto that runtime. If there are any errors during deployment, a task is created to review the error.

6. Error handling – when models are rejected, errors occur running the tests, or the model fails to deploy, the process creates human tasks to review the reasons for failure so they can take the appropriate actions.

Daily Backtesting MLC Process

The Daily Backtesting MLC Process is an example of a simple monitor that gets exact version of a model that is currently deployed, and tests against a new set of labeled data from the night before.

1. Start Event – a timed start event initiates the monitor every morning at a specific time.

2. Run Test – next, a Test Job evaluates the model with the new data.

3. Parse Test Results – parses the raw test results into a standardized format and captures the data within Model Manager for future review.

4. Exceeded Threshold – if the the business threshold is not met, an alert is sent to the support team to investigate.

Create & Deploy a New MLC Process Using Camunda Modeler

1. Download and install Camunda Modeler on your local machine. Go to https://camunda.com/download/modeler/

2. Create a new BPMN.

3. When the MLC Process is ready, select the Deploy icon to put it in the MLC Manager. Note: while the URL will be highly dependent on your environment’s exact setup, it likely uses a path such as http://<moc-base-url>:8090/mlc-service/rest

   

4. Verify that your new MLC Process is registered with MLC Manager. Go to the Command Center and click the Models icon in the left column.