MLflow, Weights & Biases (W&B), and Neptune are all powerful tools for experiment tracking in MLOps, but they approach the problem from subtly different angles, with W&B and Neptune offering a more opinionated, integrated experience compared to MLflow’s more modular, framework-agnostic design.
Let’s see them in action. Imagine you’re training a simple PyTorch model.
Here’s how you might log parameters, metrics, and an artifact with W&B:
import wandb
import torch
import torch.nn as nn
import torch.optim as optim
# 1. Initialize a W&B run
run = wandb.init(project="my-pytorch-project", config={"learning_rate": 0.01, "epochs": 10})
# Access config
config = wandb.config
# Define a simple model
class SimpleNet(nn.Module):
def __init__(self):
super(SimpleNet, self).__init__()
self.fc = nn.Linear(10, 2)
def forward(self, x):
return self.fc(x)
model = SimpleNet()
# Dummy data and optimizer
criterion = nn.CrossEntropyLoss()
optimizer = optim.SGD(model.parameters(), lr=config.learning_rate)
dummy_input = torch.randn(1, 10)
dummy_target = torch.tensor([1])
# Training loop (simplified)
for epoch in range(config.epochs):
optimizer.zero_grad()
outputs = model(dummy_input)
loss = criterion(outputs, dummy_target)
loss.backward()
optimizer.step()
# 2. Log metrics
wandb.log({"loss": loss.item(), "epoch": epoch})
# 3. Log an artifact (e.g., the trained model)
# Create a dummy file to save
with open("model_weights.pth", "wb") as f:
torch.save(model.state_dict(), f)
# Log the file as an artifact
artifact = wandb.Artifact('trained-model', type='model')
artifact.add_file("model_weights.pth")
run.log_artifact(artifact)
# Finish the run
run.finish()
Now, let’s do the same with Neptune:
from neptune.new import init_run
import torch
import torch.nn as nn
import torch.optim as optim
# 1. Initialize a Neptune run
run = init_run(project="my-organization/my-project", api_token="ANONYMOUS", source_files=["*.py"])
# Log parameters
run["parameters"] = {"learning_rate": 0.01, "epochs": 10}
# Access parameters
learning_rate = run["parameters"]["learning_rate"].fetch()
epochs = run["parameters"]["epochs"].fetch()
# Define a simple model (same as above)
class SimpleNet(nn.Module):
def __init__(self):
super(SimpleNet, self).__init__()
self.fc = nn.Linear(10, 2)
def forward(self, x):
return self.fc(x)
model = SimpleNet()
# Dummy data and optimizer
criterion = nn.CrossEntropyLoss()
optimizer = optim.SGD(model.parameters(), lr=learning_rate)
dummy_input = torch.randn(1, 10)
dummy_target = torch.tensor([1])
# Training loop (simplified)
for epoch in range(epochs):
optimizer.zero_grad()
outputs = model(dummy_input)
loss = criterion(outputs, dummy_target)
loss.backward()
optimizer.step()
# 2. Log metrics
run["metrics/loss"].log(loss.item())
run["metrics/epoch"].log(epoch)
# 3. Log an artifact (e.g., the trained model)
# Create a dummy file to save
with open("model_weights.pth", "wb") as f:
torch.save(model.state_dict(), f)
# Log the file as an artifact
run["model_weights"].upload("model_weights.pth")
# Finish the run (optional, Neptune often auto-saves)
# run.stop()
And here’s a minimal MLflow example:
import mlflow
import torch
import torch.nn as nn
import torch.optim as optim
# 1. Start an MLflow run
mlflow.start_run()
# Log parameters
params = {"learning_rate": 0.01, "epochs": 10}
mlflow.log_params(params)
# Define a simple model (same as above)
class SimpleNet(nn.Module):
def __init__(self):
super(SimpleNet, self).__init__()
self.fc = nn.Linear(10, 2)
def forward(self, x):
return self.fc(x)
model = SimpleNet()
# Dummy data and optimizer
criterion = nn.CrossEntropyLoss()
optimizer = optim.SGD(model.parameters(), lr=params["learning_rate"])
dummy_input = torch.randn(1, 10)
dummy_target = torch.tensor([1])
# Training loop (simplified)
for epoch in range(params["epochs"]):
optimizer.zero_grad()
outputs = model(dummy_input)
loss = criterion(outputs, dummy_target)
loss.backward()
optimizer.step()
# 2. Log metrics
mlflow.log_metric("loss", loss.item())
mlflow.log_metric("epoch", epoch)
# 3. Log an artifact (e.g., the trained model)
# Create a dummy file to save
with open("model_weights.pth", "wb") as f:
torch.save(model.state_dict(), f)
# Log the file as an artifact
mlflow.log_artifact("model_weights.pth")
# End the run
mlflow.end_run()
The core problem these tools solve is the chaos of uncontrolled experimentation. Without them, tracking which hyperparameters, code versions, datasets, and results belong to which experiment is a manual, error-prone nightmare. They provide a centralized, searchable repository for all your experimental artifacts and metadata.
MLflow is built around three core components:
- Tracking: Logs parameters, code versions, metrics, and artifacts. This is what we’ve shown above. It can log to a local file, a database, or a remote server.
- Projects: Packages code in a reusable, reproducible format. This allows you to easily run experiments defined by others or share your own.
- Models: Provides a standard format for packaging machine learning models from various frameworks, enabling easy deployment.
W&B and Neptune are more integrated platforms. They offer a hosted UI out-of-the-box (though self-hosting is often an option) and are generally more opinionated about how you structure your logging. They excel at providing rich visualizations, collaboration features, and hyperparameter sweeps directly within their dashboards.
A key difference in their API design is how they handle logging. W&B uses wandb.log() which is a flexible dictionary-based interface. Neptune uses a hierarchical dictionary-like structure (run["metrics/loss"].log(...)), which can feel more organized for complex projects. MLflow’s mlflow.log_metric() and mlflow.log_param() are more direct, function-call-based logging.
The most surprising thing is how much of the "magic" in W&B and Neptune is simply a well-designed, reactive UI built on top of a robust backend. They abstract away much of the complexity of storing, indexing, and visualizing, making it incredibly easy to get started and gain insights. MLflow, while powerful, often requires more explicit setup for a polished UI experience (e.g., running mlflow ui separately or configuring a remote tracking server).
The concept of "artifacts" is crucial. Beyond just metrics and parameters, these tools allow you to log entire files, datasets, models, plots, and even videos. This provides a complete snapshot of an experiment’s state, ensuring reproducibility. For instance, logging the exact model weights used when a specific metric was achieved is far more reliable than just logging the metric itself.
The next problem you’ll likely encounter is managing and comparing large numbers of experiments, which leads into hyperparameter optimization and experiment visualization tools.