How to Implement ModelOp

This comprehensive guide walks you through implementing ModelOp Center from initial planning through deployment and configuration. Based on official ModelOp documentation, this guide provides a structured approach to ensure a smooth installation.

The main sections of this guide are as follows:

1. Phase 1: Pre-Implementation Planning
2. Phase 2: Environment Setup
3. Phase 3: Resource Sizing
4. Phase 4: Pre-Install Validation
5. Phase 5: Configure Helm Charts
6. Phase 6: Install ModelOp Center
7. Phase 7: Post-Installation Configuration
8. Phase 8: Ongoing Management
9. Best Practices

Implementation Overview

ModelOp Center is a microservices-architected application deployed to Kubernetes. The implementation process requires careful preparation, stakeholder alignment, and systematic execution across multiple phases.

Estimated Timeline: 2-4 weeks depending on environment complexity and stakeholder availability

Key Roles Required:

• Kubernetes Administrator
• Database Administrator
• Security/Identity Management Specialist
• DevOps Engineer
• ModelOp Technical Lead

Phase 1: Pre-Implementation Planning

Before beginning installation, organize and prepare critical infrastructure components and credentials.

Technical Prerequisites Checklist

Complete this checklist before proceeding with installation:

Source Control & Repository:

• Select Git repository platform (GitHub, GitLab, Bitbucket, Azure DevOps)
• Create repository access credentials
• Configure network access from Kubernetes cluster

Database Infrastructure:

• MongoDB - Determine external vs. PVC-backed standalone
  • If external: Obtain MongoDB URI connection string
  • If standalone: Plan persistent volume requirements (20-80GB recommended)
• PostgreSQL or MSSQL - Determine external vs. PVC-backed standalone
  • If external: Obtain connection string
  • If standalone: Plan persistent volume requirements (4-8GB recommended)

Object Storage:

• Select S3-compatible storage (AWS S3, Azure Blob, Google Cloud Storage, or MinIO)
• If external cloud storage: Obtain access credentials
• If MinIO standalone: Plan persistent volume requirements (16-32GB recommended)

Kubernetes Environment:

• Confirm Kubernetes platform (OpenShift, AKS, EKS, GKE)
• Identify Kubernetes SME for installation support
• Verify cluster access and credentials
• Confirm namespace allocation

Container Registry:

• Select image registry (Artifactory, GAR, ECR, Quay, ACR, DockerHub)
• Complete image intake process
• Obtain registry credentials
• Import ModelOp images or configure DockerHub access

Security & Authentication:

• Document secrets management approach (vaults, Kubernetes secrets, pipelines)
• Identify authentication methods (OAuth 2.0 providers, Kerberos, etc.)
• Plan credential storage strategy
• Document security requirements

Integrations:

• Identify ticketing system (Jira, ServiceNow) - obtain credentials
• Identify pipeline tools (Jenkins, GitLab CI, Azure DevOps) - obtain credentials
• Document required data sources for monitoring and models
• List all external technology integrations

Networking:

• Define required egress rules
• Define required ingress rules
• Verify connectivity between cluster and all external technologies
• Document firewall requirements

📖 Reference: ModelOp Center Installation Overview

Phase 2: Environment Setup

Install Required Tools

Ensure your workstation or jumpbox has the necessary utilities installed.

1. Install kubectl

The Kubernetes command-line tool for cluster interaction.

bash

# Verify installation
kubectl version --client

Installation Guide: Kubernetes Install Tools

2. Install Helm 3+

Helm manages Kubernetes deployments through charts and releases.

bash

# Verify installation
helm version

# Expected output (version 3.0+):# version.BuildInfo{Version:"v3.9.0", ...}

Installation Guide: Helm Installation

3. Configure Kubernetes Context

Establish connectivity to your target cluster.

bash

# Verify cluster access
kubectl get namespaces

# If this fails, configure your context# Follow guide: https://kubernetes.io/docs/tasks/access-application-cluster/configure-access-multiple-clusters/

Windows Users

Run all commands using WSL (Windows Subsystem for Linux): WSL Documentation

Acquire Helm Charts

Contact your ModelOp Account Executive or Sales Engineer to obtain the Helm charts (.tgz file) delivered via secure bucket link.

Extract the charts:

bash

tar -xvf moc-[version].tgz

This creates a moc directory containing all chart files.

Understanding the Chart Structure

moc/
├── Chart.yaml              # Release metadata and versioning
├── values.yaml             # Default configuration values (DO NOT MODIFY)
├── starter/
│   └── simple.yaml         # Starting template for customization
├── scripts/
│   └── generatePullSecrets.sh  # Registry credential setup
├── sccs/                   # Spring Cloud Config Server base files
└── templates/              # Kubernetes manifest templates

Important: Copy starter/simple.yaml outside the chart directory and customize ONLY that file. Never modify values.yaml directly.

📖 Reference: Helm Primer

Phase 3: Resource Sizing

Properly size your Kubernetes cluster based on your deployment needs.

Core Services Requirements

Runtime Services (per runtime instance)

Recommended starting point: 3 runtime instances

⚠️ Note: Without runtimes, monitoring, scoring, retraining, and model tests cannot be performed. Scale runtimes based on the number and complexity of models.

Optional Infrastructure Services

Environment Sizing Recommendations

Development/Sandbox:

• Use minimum specifications
• 1-2 runtime instances
• Total: ~13 CPU, ~28GB RAM, ~18GB disk

Production:

• Use recommended specifications
• 3+ runtime instances
• Total: ~26+ CPU, ~62+ GB RAM, ~65+ GB disk

📖 Reference: ModelOp Center Sizing Considerations

Phase 4: Pre-Install Validation

The Pre-Install Utility (PIU) validates configurations before the main installation, catching issues early.

Deploy the Pre-Install Utility

1. Configure your override file (simple.yaml):

yaml

moc:
 namespace: my-moc-namespace
 version: "3.1"
 
 services:
   pre-install:
     enabled: true
     serviceType: LoadBalancer  # Or use port-forward with ClusterIP
   sccs:
     enabled: true
   # Comment out 'core' during pre-install phase
   # core:
   #   enabled: true

2. Install the PIU:

bash

helm install PIU-RELEASE moc -f simple.yaml

3. Access the PIU interface:

If using LoadBalancer:

bash

# Get the external IP
kubectl get services | grep pre-install

# Access at: http://<EXTERNAL-IP>:2888

If using port-forward:

bash

# Forward to local machine
kubectl port-forward <POD-NAME> 2888:2888

# Access at: http://localhost:2888

Validation Checks

The PIU validates:

✅ Database Connectivity

• MongoDB URI accessibility and credentials
• PostgreSQL/MSSQL connection strings
• Read/write permissions (optional, configurable)

✅ Storage Access

• S3-compatible bucket connectivity
• Bucket permissions

✅ External Integrations

• Git repository access
• Jira connectivity (if configured)
• Jenkins connectivity (if configured)
• Other data sources

✅ Security Configuration

• OAuth 2.0 provider settings
• Certificate validity
• Credential accessibility

✅ Spring Cloud Config Server

• SCCS connectivity
• Configuration retrieval
• Property validation

Using the PIU Interface

Web Interface: Navigate to http://localhost:2888

• Properties Page: Verify all configuration properties are correctly loaded
• Health Page: Check connectivity to all external services
• OAuth Page: Validate OAuth 2.0 configuration

Command Line Interface:

bash

# Check all properties
curl http://localhost:2888/api/properties

# Check all health indicators
curl http://localhost:2888/api/health

# Test specific URL connectivity
curl "http://localhost:2888/api/health/url?url=https://your-service.com"

# Validate OAuth configuration
curl http://localhost:2888/api/oauth

Resolving Issues

If validation fails:

1. Update Spring Cloud Config Server properties - Modify SCCS configuration files
2. Override via environment variables - Use PIU's Environment Variables page
3. Verify network connectivity - Ensure firewall rules allow traffic
4. Check credentials - Confirm all usernames/passwords are correct

Do not proceed to main installation until all critical health checks pass.

📖 Reference: ModelOp Center Pre-Install Utility

Phase 5: Configure Helm Charts

Customize your deployment by creating an override file based on starter/simple.yaml.

Create Your Override File

bash

# Copy the starter template outside the chart directory
cp moc/starter/simple.yaml my-moc-config.yaml

Essential Configuration

Minimum required settings:

yaml

moc:
 # Deployment metadata
 namespace: production-moc
 version: "3.1"
 
 # Image registry (if using private registry)
 imageRegistry: your-registry.example.com/modelop
 
 # Services to deploy
 services:
   core:
     enabled: true
   sccs:
     enabled: true
   pre-install:
     enabled: false  # Disable after validation
 
 # Runtime configuration
 sparkRuntimes:
   alpha:
     replicas: 3

Database Configuration

Configure connections to your databases:

yaml

moc:
 mongodb:
   # If using external MongoDB
   external: true
   uri: "mongodb://username:password@host:27017/database?authSource=admin"
   
 postgres:
   # If using external PostgreSQL
   external: true
   host: postgres.example.com
   port: 5432
   database: mlc_manager
   username: moc_user
   # Password should reference a Kubernetes secret
   passwordSecret:
     name: postgres-credentials
     key: password

Storage Configuration

Configure S3-compatible storage:

yaml

moc:
 s3:
   endpoint: https://s3.amazonaws.com
   region: us-east-1
   bucket: modelop-artifacts
   # Credentials should reference Kubernetes secrets
   accessKeySecret:
     name: s3-credentials
     key: access-key
   secretKeySecret:
     name: s3-credentials
     key: secret-key

Security Configuration

For secured deployments with OAuth 2.0:

yaml

moc:
 security:
   mode: rwx  # Enable role-based access control
   oauth2:
     issuerUri: https://your-idp.example.com
     clientId: modelop-client
     clientSecretRef:
       name: oauth-secrets
       key: client-secret

📖 Reference: Configurations Required to Support RWX

Advanced Configurations

Spark Runtime with Hadoop:

yaml

moc:
 sparkRuntimes:
   alpha:
     env:
       HADOOP_CONF_DIR: /modelop/hadoop/conf
       SPARK_HOME: /modelop/bin/spark-2.4.4-bin-hadoop2.6
     configMaps:
       hadoop-config:
         mountPath: /modelop/hadoop/conf
         
 configMaps:
   hadoop-config: hadoop-conf-files/*

📖 Reference: Configuring the Spark Runtime via Helm

Phase 6: Install ModelOp Center

With configuration complete and validation successful, deploy ModelOp Center.

Generate Registry Credentials

Create Kubernetes secrets for pulling images:

bash

cd moc/scripts
./generatePullSecrets.sh --apply

# You'll be prompted for:# - Username (provided by ModelOp or your registry admin)# - Password/Token

This creates a modelop-regcreds secret in your namespace.

Execute Installation

bash

# From the directory containing the 'moc' folder
helm install moc-production moc -f my-moc-config.yaml -n production-moc

# Parameters:# - moc-production: Release name (choose any meaningful name)# - moc: Path to chart directory# - my-moc-config.yaml: Your override file# - production-moc: Target namespace

Expected output:

NAME: moc-production
LAST DEPLOYED: [timestamp]
NAMESPACE: production-moc
STATUS: deployed
REVISION: 1
NOTES:
ModelOp Center Version 3.1
Install complete!

Verify Installation

1. Check pod status:

bash

kubectl get pods -n production-moc

# All pods should reach 'Running' status# This may take 5-10 minutes

2. Monitor deployment:

bash

# Watch pods come online
kubectl get pods -n production-moc -w

# Check for errors
kubectl logs <pod-name> -n production-moc

3. Access the Gateway:

bash

# Get the Gateway service details
kubectl get services -n production-moc | grep gateway

# Example output:# gateway   LoadBalancer   10.100.8.109   external-lb.amazonaws.com   8090:30285/TCP   1m

4. Access ModelOp Center:

Open your browser to:

http://<GATEWAY-IP>:8090

You should see the ModelOp Center Dashboard.

5. Verify service registration:

Click the "Status" link in the bottom-left corner of the Dashboard to confirm all services are registered and healthy.

Phase 7: Post-Installation Configuration

Configure Integrations

After successful deployment, configure external integrations through the ModelOp Center UI or via configuration updates.

Common integrations:

• Git repositories for model storage
• Jira for governance and ticketing
• Jenkins for CI/CD pipelines
• Data sources for model monitoring

Backup Strategy

Implement regular backup procedures for critical data stores.

MongoDB Backup Example:

bash

# Local backup
mongodump --host=mongodb.example.net --port=27017 \
 --username=user --password="pass" --out /backup/mongo

# Backup to S3
mongodump --host=mongodb.example.net --port=27017 \
 --username=user --password="pass" --archive \
 | aws s3 cp - s3://backups/mongo/backup-$(date +%Y%m%d).archive

PostgreSQL Backup Example:

bash

# Local backup
pg_dump dbname -h hostname -p 5432 -U username > /backup/postgres.sql

# Backup to S3
pg_dump dbname -h hostname -p 5432 -U username \
 | aws s3 cp - s3://backups/postgres/backup-$(date +%Y%m%d).sql

📖 Reference: How To Backup/Restore ModelOp Center's Databases

Security Hardening

1. Create Kubernetes secrets for credentials:

bash

# Example: Create S3 credentials secret
kubectl create secret generic s3-credentials \
 --from-literal=access-key=YOUR_ACCESS_KEY \
 --from-literal=secret-key=YOUR_SECRET_KEY \
 -n production-moc

2. Enable OAuth 2.0:

Update your configuration to enable secured mode and configure your identity provider.

3. Configure network policies:

Implement Kubernetes network policies to restrict traffic between services.

Phase 8: Ongoing Management

Updating ModelOp Center

When configuration changes are needed:

1. Update your override file:

bash

# Edit my-moc-config.yaml with new values

2. Apply changes:

bash

helm upgrade moc-production moc -f my-moc-config.yaml -n production-moc

Helm performs a 3-way merge and updates only changed resources.

Scaling Runtimes

As your model workload grows:

yaml

moc:
 sparkRuntimes:
   alpha:
     replicas: 5  # Increase from 3

Apply the change:

bash

helm upgrade moc-production moc -f my-moc-config.yaml -n production-moc

Version Upgrades

1. Obtain new Helm charts from ModelOp

2. Extract to new directory:

bash

tar -xvf moc-[new-version].tgz -C moc-new/

3. Review changes:

bash

# Compare values.yaml for new configuration options
diff moc/values.yaml moc-new/values.yaml

4. Update your override file with any new required settings

5. Perform upgrade:

bash

helm upgrade moc-production moc-new -f my-moc-config.yaml -n production-moc

6. Monitor the upgrade:

bash

kubectl get pods -n production-moc -w

Troubleshooting

Pods not starting:

bash

# Describe the pod for events
kubectl describe pod <pod-name> -n production-moc

# Check logs
kubectl logs <pod-name> -n production-moc

# Check previous logs if pod restarted
kubectl logs <pod-name> -n production-moc --previous

Service connectivity issues:

bash

# Test from within the cluster
kubectl run -it --rm debug --image=busybox --restart=Never -- sh
# Then: wget -O- http://service-name:port

Configuration not updating:

If you update a ConfigMap, restart the affected pods:

bash

kubectl rollout restart deployment/<deployment-name> -n production-moc

Best Practices

Configuration Management

✅ DO:

• Keep a single source of truth for your override file (version control)
• Use Kubernetes secrets for sensitive data
• Document all configuration changes
• Test changes in a development environment first

❌ DON'T:

• Modify values.yaml directly in the chart
• Store credentials in your override file
• Maintain multiple copies of override files
• Skip validation before deploying to production

Helm Management

✅ DO:

• Use meaningful release names
• Track release history: helm list -n production-moc
• Keep Helm charts outside your override file directory
• Use helm diff plugins to preview changes

❌ DON'T:

• Manually edit deployed Kubernetes resources
• Delete resources that Helm manages
• Mix Helm and kubectl for managing the same resources

Security

✅ DO:

• Rotate credentials regularly
• Use network policies to limit traffic
• Enable audit logging
• Implement backup and disaster recovery procedures
• Use OAuth 2.0 for production deployments

❌ DON'T:

• Commit secrets to version control
• Use default passwords
• Skip security validation in PIU
• Allow unrestricted network access

Quick Reference

Essential Commands

bash

# Check Helm releases
helm list -n <namespace>

# View release history
helm history <release-name> -n <namespace>

# Rollback to previous version
helm rollback <release-name> -n <namespace>

# Get pod status
kubectl get pods -n <namespace>

# View pod logs
kubectl logs <pod-name> -n <namespace>

# Port forward for local access
kubectl port-forward <pod-name> <local-port>:<pod-port> -n <namespace>

# Execute command in pod
kubectl exec -it <pod-name> -n <namespace> -- bash

Key Documentation Links

• Installation Overview
• Installing with Helm
• Helm Primer
• Pre-Install Utility
• Sizing Considerations
• Backup/Restore Procedures
• Spark Runtime Configuration
• RWX Security Configuration

Support

For additional assistance:

• Contact your ModelOp Account Executive
• Reach out to your assigned Sales Engineer
• Consult the complete documentation at the links provided throughout this guide

This guide synthesizes information from official ModelOp documentation. For the most current and detailed information, always refer to the linked official documentation.

‍