# Multi-tenancy ## Why? * usually more cost-effective * easier access to shared resources, for example controllers * reduced management overhead * no need to wait for cluster and related infrastructure creation for new tenants ## Why not? * resource starvation * security aspects * hard to get right ## Use cases * These use cases are listed in David Oppenheimer's talk * Enterprise * users all from same organization * users are semi-trusted * personas * cluster admin * namespace admin * user * vanilla container isolation might be sufficient depending on the workload * inter-pod communication might be limited * to only within namespace * to other namespaces depending on the application topology * one application tier per namespace * Kubernetes as a Service (KaaS) / Platform as a Service (PaaS) * untrusted users running untrusted code * users can create namespaces and CRUD non-policy objects with their namespace(s) * needs stronger control plane, node, and network isolation * resource quotas per how much customer pays * Software as a Service (SaaS): multi-tenant app * customer does not access Kubernetes apiserver, but the application * in single instance for all customers model, Kubernetes multi-tenant models are not concerned as multi-tenancy is handled at application level * in multi-intance model, each customer has their own application instance * in this model proxy can interact with Kubernetes apiserver for creating new application instances * namespace per application instance * code is semi-trusted * if plugins are allowed (for example Wordpress) that code is untrusted * certain namespace might host shared infrastructure ## Isolation mechanisms * namespace isolation * best practice is to have namespace per tenant. Or several namespaces per tenant * label your namespaces so network policies can target them * with labels, network policies can target multiple namespaces * most of the isolation features expect this * Pod Security Policies * limit pod/container security contexts * for example, pod can not be run as root * Security contexts * RBAC * Network Policies * limit network access between tenants * need to have enforcing network overlay (CNI) * Calico * Weave * etc * Secrets * can be encrypted at rest with `EncryptionConfiguration` resource * `apiserver` needs to be configured with `--encryption-provider-config` option * secrets are encrypted at write * Scheduling * Resource quotas * limit size of resource requests and limits per namespace * Resource requests * Resource limits * memory: kills * cpu: throttling * Limit ranges * enforce setting resource limits * QoS classes * Guaranteed * Burstable * BestEffort * derived from request/limit settings * Dedicated nodes (sole-tenant nodes) * taints and tolerations * Affinity / anti-affinity * for example, which pods can co-exist in a node * -> pod isolation * Priority and preemption (alpha) * low and high priority pods * if high-priority pod can not be scheduled, low-priority pod will be killed * OpenPolicyAgent * higher-level enforcing of certain policy sets ## Misc * Stateful workloads are more complex to manage in multi-tenant setups * Node security must be kept in mind, because the isolation features of Kubernetes won't help if the node is compromised ## Related * Software architecture * multi-tenant * each of the customer in the single application instance and single database * single-point-of-failure * multi-instance * each customer has their own application instance and database * stability - no single-point-of-failure * each of the instances can be scaled seperately * data safety in case of for example security breach ## Resources * Multi-Tenancy in Kubernetes: Best Practices Today, and Future Directions - David Oppenheimer - * Managing a Multi-Tenanted Kubernetes Cluster in Production by Josh Bowen, Apigee - * Getting started with Kubernetes for your SaaS - * A Pathway to Multi-Tenancy in Kubernetes - * Kubernetes Multi-Tenancy Best Practices - * Cluster multi-tenancy - * Multi-tenant design considerations for Amazon EKS clusters - * AWS re:Invent 2019: Architecting multi-tenant PaaS offerings with Amazon EKS (GPSTEC337) - --- # Agent Instructions: Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter: ``` GET https://notes.tatusl.dev/kubernetes/multi-tenancy.md?ask= ``` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.