MLflow v2 migration often feels like a tightrope walk, but you can upgrade your tracking server and client libraries without invalidating your existing runs.

Let’s see MLflow v2 in action. Imagine we have a simple Python script that logs parameters, metrics, and an artifact:

import mlflow
import os

# Set the tracking URI to your MLflow server
# For local testing, you can use a local file path:
# mlflow.set_tracking_uri("file:///tmp/mlruns")
# In a real scenario, this would be your server URL:
# mlflow.set_tracking_uri("http://your-mlflow-server:5000")

# Ensure the tracking URI is set, otherwise it defaults to ./mlruns
if mlflow.get_tracking_uri() is None:
    mlflow.set_tracking_uri("file:///tmp/mlruns_v2_demo") # Use a fresh directory for this demo

experiment_name = "v2_migration_demo"
mlflow.set_experiment(experiment_name)

with mlflow.start_run() as run:
    run_id = run.info.run_id
    print(f"Starting run: {run_id}")

    # Log parameters
    mlflow.log_param("learning_rate", 0.01)
    mlflow.log_param("optimizer", "adam")

    # Log a metric
    mlflow.log_metric("accuracy", 0.95)

    # Log an artifact
    artifact_path = "model_config.txt"
    with open(artifact_path, "w") as f:
        f.write("model_type: RandomForest\n")
        f.write("n_estimators: 100\n")
    mlflow.log_artifact(artifact_path)

    print(f"Run completed: {run_id}")
    print(f"MLflow UI available at: {mlflow.get_tracking_uri()}")

When you run this script, MLflow creates a new experiment named "v2_migration_demo" (or uses it if it exists) and logs the details of this training run. You can then view this information in the MLflow UI by running mlflow ui in your terminal from the directory where your mlruns folder is located (or pointing the UI to your server URL).

The core problem MLflow v2 migration addresses is how to evolve the MLflow backend store (where run data, params, metrics, etc., are stored) and the client libraries (the Python package you use to interact with MLflow) without breaking compatibility with older runs. MLflow achieves this by having a well-defined schema for its backend store and versioning it. When you upgrade your MLflow server, it checks the current schema version. If it’s older than what the new server version expects, it runs an automatic migration process to update the schema. The client libraries are designed to be backward-compatible, meaning newer client versions can typically write to older backend schema versions, and older client versions can still read data written by newer ones, within reasonable limits.

The key levers you control are:

  • Tracking URI: This tells your MLflow client where to send the data. It can be a local filesystem path (file:///path/to/mlruns), an SQLAlchemy database URI (sqlite:///mlflow.db, postgresql://user:password@host/dbname), or a Databricks-specific URI. The server component of MLflow reads from and writes to this URI.
  • MLflow Server Version: This is the software running on your server that manages the backend store. Upgrading this is the primary action for v2 migration.
  • MLflow Client Library Version: This is the mlflow Python package installed in your environment. You’ll want to upgrade this to leverage new features and ensure compatibility.
  • Backend Store Schema Version: This is an internal version number managed by MLflow within the backend store itself, indicating the structure of the data.

Consider a scenario where you have an MLflow server running with version 1.x, and your backend store reflects schema version 1. You then upgrade your MLflow server to version 2.x. When the v2.x server starts, it detects the schema version is 1. If v2.x requires schema version 2, the server will automatically execute a migration script to update the schema in your backend store to version 2. Your existing runs, which were logged using the v1.x client and v1.x schema, will remain accessible because the v2.x schema is designed to be backward compatible.

The crucial part for a smooth migration is understanding that MLflow’s backend schema is versioned. When you upgrade your MLflow server (e.g., from mlflow-server==1.15.0 to mlflow-server==2.0.0), the server will check the mlflow_experiment.runs table (or equivalent in your database) for a schema_version column. If this version is less than what the new server expects, it will attempt an automatic schema upgrade. For database backends, this usually involves running SQL ALTER TABLE statements. For file-based backends, it might involve updating metadata files. The client libraries are designed to write data in a format that is forward-compatible with schema upgrades.

When you upgrade your MLflow tracking server, the automatic schema migration handles the backend. However, to fully benefit from v2 features and ensure seamless operation, you should also upgrade your MLflow client libraries in all environments where you interact with MLflow. This means updating your mlflow Python package to the latest version, typically with pip install --upgrade mlflow. The client library upgrade ensures that your code is generating data and interacting with the server using the latest protocols and best practices, minimizing potential compatibility glitches.

The next logical step after a successful v2 migration is to explore the new artifact repository integrations and advanced experiment tracking features introduced in v2.

Want structured learning?

Take the full Mlflow course →