GKE tenants are not inherently isolated from each other; they share the same Kubernetes cluster and API server by default.
Let’s see how this looks in practice. Imagine you have a GKE cluster with two namespaces, dev-team-a and dev-team-b. Both namespaces are managed by different teams, but without any specific isolation, a user with broad permissions in the cluster could potentially see or even modify resources in both namespaces.
Here’s a simplified view of what a kubectl get pods --all-namespaces might show:
NAMESPACE NAME READY STATUS RESTARTS AGE
dev-team-a app-a-deployment-xxxxx-yyyyy 1/1 Running 0 2d
dev-team-a app-a-other-zzzzz-aaaaa 1/1 Running 0 2d
dev-team-b app-b-service-bbbbb-ccccc 1/1 Running 0 1d
dev-team-b app-b-database-ddddd-eeeee 1/1 Running 0 1d
kube-system coredns-xxxxx-yyyyy 1/1 Running 0 30d
...
Without proper RBAC and network policies, there’s no strong boundary between dev-team-a and dev-team-b.
To achieve tenant isolation, we primarily use two Kubernetes features: Role-Based Access Control (RBAC) and Network Policies.
RBAC defines who can do what to which resources. We’ll create Roles (permissions within a namespace) or ClusterRoles (permissions cluster-wide) and then bind them to users or groups using RoleBindings or ClusterRoleBindings. For tenant isolation, we’ll focus on Roles and RoleBindings to restrict access within specific namespaces.
Network Policies, on the other hand, control network traffic between pods. By default, all pods in a Kubernetes cluster can communicate with each other. Network Policies allow us to define rules that dictate which pods can connect to which other pods, and on which ports.
Let’s build a robust isolation model.
1. Namespace Creation: First, create dedicated namespaces for each tenant. This is the foundational organizational unit.
kubectl create namespace tenant-alpha
kubectl create namespace tenant-beta
2. RBAC for Namespace Isolation:
For tenant-alpha, we want to grant a specific user, alpha-admin@example.com, full control only within the tenant-alpha namespace.
First, define the Role:
# tenant-alpha-role.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: tenant-alpha-admin-role
namespace: tenant-alpha
rules:
- apiGroups: ["", "apps", "extensions", "batch", "networking.k8s.io"] # Common API groups
resources: ["*"] # All resources within these groups
verbs: ["*"] # All actions
Apply it:
kubectl apply -f tenant-alpha-role.yaml
Then, create a RoleBinding to link the user to this role:
# tenant-alpha-rolebinding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: tenant-alpha-admin-binding
namespace: tenant-alpha
subjects:
- kind: User
name: alpha-admin@example.com # Your identity provider's user identifier
apiGroup: rbac.authorization.k8s.io
roleRef:
kind: Role
name: tenant-alpha-admin-role
apiGroup: rbac.authorization.k8s.io
Apply it:
kubectl apply -f tenant-alpha-rolebinding.yaml
Now, alpha-admin@example.com can manage resources in tenant-alpha but cannot see or modify anything in tenant-beta or kube-system. To verify, try kubectl get pods --namespace tenant-beta as alpha-admin@example.com; it should return an empty list or an error indicating no access.
Repeat this for tenant-beta and beta-admin@example.com.
3. Network Policy for Pod-to-Pod Isolation:
Even with RBAC, pods in tenant-alpha can still talk to pods in tenant-beta by default. We need Network Policies. We’ll use Calico, which is often the default CNI in GKE, or another CNI that supports NetworkPolicy.
Let’s assume we want to:
- Default Deny: No pod can communicate with any other pod unless explicitly allowed.
- Tenant Intra-communication: Pods within
tenant-alphacan communicate with each other. - Tenant Inter-communication Blocked: Pods in
tenant-alphacannot communicate with pods intenant-beta, and vice-versa. - Ingress Allowed: Pods need to accept traffic from an Ingress controller.
First, apply a default deny policy to the tenant-alpha namespace:
# tenant-alpha-default-deny.yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny-all
namespace: tenant-alpha
spec:
podSelector: {} # Selects all pods in the namespace
policyTypes:
- Ingress
- Egress
Apply it:
kubectl apply -f tenant-alpha-default-deny.yaml
Now, no pod in tenant-alpha can receive or send traffic.
Next, allow intra-namespace communication within tenant-alpha:
# tenant-alpha-allow-same-namespace.yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-same-namespace
namespace: tenant-alpha
spec:
podSelector: {} # Apply to all pods in tenant-alpha
policyTypes:
- Ingress
ingress:
- from:
- podSelector: {} # Allow ingress from any pod in the same namespace
Apply it:
kubectl apply -f tenant-alpha-allow-same-namespace.yaml
Now, pods within tenant-alpha can talk to each other. To allow ingress from your Ingress controller (e.g., GKE Ingress, Nginx Ingress), you’d add a specific rule targeting the pods that need to receive external traffic, often identified by labels. For GKE Ingress, this might involve selecting pods with specific labels that the Ingress controller targets.
A common pattern is to allow ingress from pods with a specific label, like app: my-web-server, and from the kube-system namespace if your Ingress controller is deployed there.
# tenant-alpha-allow-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-ingress-from-ingress-controller
namespace: tenant-alpha
spec:
podSelector:
matchLabels:
app: my-web-server # Or whatever label your app pods have
policyTypes:
- Ingress
ingress:
- from:
- podSelector: {} # Allow from any pod in tenant-alpha (already handled above, but good for clarity if you want to be explicit)
- namespaceSelector: # Allow from specific namespaces
matchLabels:
kubernetes.io/metadata.name: ingress-nginx # Example for Nginx Ingress
podSelector: {} # Allow from any pod in that namespace
You would then apply similar default-deny and specific-allow policies for tenant-beta.
The Counterintuitive Part:
Most people assume that simply putting resources into different namespaces provides isolation. However, namespaces are primarily organizational constructs. Without RBAC, users can still access resources across namespaces if their permissions are broad enough. Without Network Policies, pods can freely communicate across namespace boundaries, which is often undesirable for security and stability in multi-tenant environments. The true isolation comes from the combination of strict RBAC rules to control who can access what, and Network Policies to control how pods can communicate.
The next step is often to implement more granular resource quotas and limit ranges per namespace to prevent noisy neighbors and ensure fair resource allocation.