Multi-tenancy
Why?
usually more cost-effective
easier access to shared resources, for example controllers
reduced management overhead
no need to wait for cluster and related infrastructure creation for new tenants
Why not?
resource starvation
security aspects
hard to get right
Use cases
These use cases are listed in David Oppenheimer's talk
Enterprise
users all from same organization
users are semi-trusted
personas
cluster admin
namespace admin
user
vanilla container isolation might be sufficient depending on the workload
inter-pod communication might be limited
to only within namespace
to other namespaces depending on the application topology
one application tier per namespace
Kubernetes as a Service (KaaS) / Platform as a Service (PaaS)
untrusted users running untrusted code
users can create namespaces and CRUD non-policy objects with their namespace(s)
needs stronger control plane, node, and network isolation
resource quotas per how much customer pays
Software as a Service (SaaS): multi-tenant app
customer does not access Kubernetes apiserver, but the application
in single instance for all customers model, Kubernetes multi-tenant models are not concerned as multi-tenancy is handled at application level
in multi-intance model, each customer has their own application instance
in this model proxy can interact with Kubernetes apiserver for creating new application instances
namespace per application instance
code is semi-trusted
if plugins are allowed (for example Wordpress) that code is untrusted
certain namespace might host shared infrastructure
Isolation mechanisms
namespace isolation
best practice is to have namespace per tenant. Or several namespaces per tenant
label your namespaces so network policies can target them
with labels, network policies can target multiple namespaces
most of the isolation features expect this
Pod Security Policies
limit pod/container security contexts
for example, pod can not be run as root
Security contexts
RBAC
Network Policies
limit network access between tenants
need to have enforcing network overlay (CNI)
Calico
Weave
etc
Secrets
can be encrypted at rest with
EncryptionConfigurationresourceapiserverneeds to be configured with--encryption-provider-configoption
secrets are encrypted at write
Scheduling
Resource quotas
limit size of resource requests and limits per namespace
Resource requests
Resource limits
memory: kills
cpu: throttling
Limit ranges
enforce setting resource limits
QoS classes
Guaranteed
Burstable
BestEffort
derived from request/limit settings
Dedicated nodes (sole-tenant nodes)
taints and tolerations
Affinity / anti-affinity
for example, which pods can co-exist in a node
-> pod isolation
Priority and preemption (alpha)
low and high priority pods
if high-priority pod can not be scheduled, low-priority pod will be killed
OpenPolicyAgent
higher-level enforcing of certain policy sets
Misc
Stateful workloads are more complex to manage in multi-tenant setups
Node security must be kept in mind, because the isolation features of Kubernetes won't help if the node is compromised
Related
Software architecture
multi-tenant
each of the customer in the single application instance and single database
single-point-of-failure
multi-instance
each customer has their own application instance and database
stability - no single-point-of-failure
each of the instances can be scaled seperately
data safety in case of for example security breach
Resources
Last updated