The most surprising thing about Kubernetes multi-tenancy is that it’s not a feature, but a philosophy you engineer into the system.
Imagine you’ve got a Kubernetes cluster, and you want to let different teams use it without them stepping on each other’s toes. Team A is building a new microservice, Team B is running a legacy batch job, and Team C is experimenting with AI models. You can’t just dump all their pods into the same namespace and hope for the best.
Here’s how we can set up a concrete example. Let’s say we have two teams, "frontend" and "backend," and we want to give them their own dedicated environments within the same cluster.
First, we create namespaces, which are the fundamental isolation boundary in Kubernetes.
kubectl create namespace frontend-prod
kubectl create namespace backend-prod
Now, any resources created within frontend-prod are logically separated from backend-prod. This is the first layer of isolation. But what about resource limits? Without them, a runaway pod in frontend-prod could starve the backend-prod team of CPU and memory.
We use ResourceQuota objects to enforce these limits. For the frontend-prod namespace, let’s set some limits:
apiVersion: v1
kind: ResourceQuota
metadata:
name: frontend-quota
namespace: frontend-prod
spec:
hard:
requests.cpu: "2"
requests.memory: 4Gi
limits.cpu: "4"
limits.memory: 8Gi
pods: "20"
services: "10"
This ResourceQuota ensures that all pods in frontend-prod collectively cannot request more than 2 CPU cores and 4Gi of memory, and cannot burst beyond 4 CPU cores and 8Gi of memory. It also limits the total number of pods and services they can create.
For the backend-prod team, we might have different needs:
apiVersion: v1
kind: ResourceQuota
metadata:
name: backend-quota
namespace: backend-prod
spec:
hard:
requests.cpu: "4"
requests.memory: 8Gi
limits.cpu: "8"
limits.memory: 16Gi
pods: "30"
services: "15"
This gives the backend team more resources, reflecting potentially more demanding workloads.
But namespaces and resource quotas only handle resource isolation. What about network isolation? By default, pods in different namespaces can still talk to each other. This is where NetworkPolicy comes in.
Let’s create a policy for the frontend-prod namespace that only allows ingress traffic from a specific source (e.g., an Ingress controller or another trusted namespace) and egress traffic to external services or specific backend services.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: deny-all-ingress
namespace: frontend-prod
spec:
podSelector: {} # Selects all pods in the namespace
policyTypes:
- Ingress
- Egress
ingress:
- from:
- podSelector: {} # Allow ingress from any pod in the same namespace (e.g., internal communication)
namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: ingress-nginx # Example: allow from ingress controller
egress:
- to:
- ipBlock:
cidr: 0.0.0.0/0 # Allow egress to any IP
except:
- 10.0.0.0/8 # Except private IP ranges (adjust as needed)
ports:
- protocol: TCP
port: 80
- protocol: TCP
port: 443
- to: # Allow egress to specific backend services
- podSelector: {}
namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: backend-prod # Example: allow communication with backend-prod namespace
This policy is a bit verbose but demonstrates a few key things: it denies all ingress by default (unless explicitly allowed), and it allows egress traffic. A more common starting point might be to allow ingress only from the ingress controller and egress only to the backend namespace and external IPs.
The real power of multi-tenancy comes from combining these elements: namespaces for logical separation, ResourceQuota for resource guarantees, and NetworkPolicy for network segmentation. You can also layer on LimitRange for default pod resource requests/limits, and even use PodSecurityPolicies (or the newer PodSecurityAdmission) to enforce security contexts.
A crucial, often overlooked, aspect of multi-tenancy is how you manage access. Simply having namespaces doesn’t stop Team A from creating resources in Team B’s namespace if they have cluster-wide edit permissions. This is where Role-Based Access Control (RBAC) is paramount.
You define Roles or ClusterRoles that specify permissions (like create, get, list, delete on pods, deployments, etc.) and then bind these roles to users or groups within specific namespaces using RoleBindings or ClusterRoleBindings.
For instance, to give the frontend team’s users full control only within their frontend-prod namespace:
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: frontend-admin
namespace: frontend-prod
rules:
- apiGroups: ["", "apps", "batch", "extensions"]
resources: ["pods", "services", "deployments", "statefulsets", "jobs", "cronjobs", "ingresses", "configmaps", "secrets"]
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: frontend-admin-binding
namespace: frontend-prod
subjects:
- kind: Group
name: "frontend-team" # This group would be defined in your OIDC/LDAP provider
apiGroup: rbac.authorization.k8s.io
roleRef:
kind: Role
name: frontend-admin
apiGroup: rbac.authorization.k8s.io
This ensures that users in the frontend-team group can only manage resources within frontend-prod.
The most common pitfall is assuming that namespaces alone provide sufficient isolation. They are a logical construct, not a hard security boundary. Without careful configuration of RBAC, NetworkPolicies, and ResourceQuotas, you’re leaving your cluster vulnerable to accidental or malicious interference between tenants.
The next challenge you’ll face is managing cluster-wide resources like storage classes or node selectors, and how to fairly allocate those across different tenants.