Multi-tenancy

Why?

  • usually more cost-effective

  • easier access to shared resources, for example controllers

  • reduced management overhead

  • no need to wait for cluster and related infrastructure creation for new tenants

Why not?

  • resource starvation

  • security aspects

  • hard to get right

Use cases

  • These use cases are listed in David Oppenheimer's talk

  • Enterprise

    • users all from same organization

    • users are semi-trusted

    • personas

      • cluster admin

      • namespace admin

      • user

    • vanilla container isolation might be sufficient depending on the workload

    • inter-pod communication might be limited

      • to only within namespace

      • to other namespaces depending on the application topology

        • one application tier per namespace

  • Kubernetes as a Service (KaaS) / Platform as a Service (PaaS)

    • untrusted users running untrusted code

    • users can create namespaces and CRUD non-policy objects with their namespace(s)

    • needs stronger control plane, node, and network isolation

    • resource quotas per how much customer pays

  • Software as a Service (SaaS): multi-tenant app

    • customer does not access Kubernetes apiserver, but the application

    • in single instance for all customers model, Kubernetes multi-tenant models are not concerned as multi-tenancy is handled at application level

    • in multi-intance model, each customer has their own application instance

      • in this model proxy can interact with Kubernetes apiserver for creating new application instances

      • namespace per application instance

    • code is semi-trusted

      • if plugins are allowed (for example Wordpress) that code is untrusted

    • certain namespace might host shared infrastructure

Isolation mechanisms

  • namespace isolation

    • best practice is to have namespace per tenant. Or several namespaces per tenant

    • label your namespaces so network policies can target them

    • with labels, network policies can target multiple namespaces

    • most of the isolation features expect this

  • Pod Security Policies

    • limit pod/container security contexts

    • for example, pod can not be run as root

  • Security contexts

  • RBAC

  • Network Policies

    • limit network access between tenants

    • need to have enforcing network overlay (CNI)

      • Calico

      • Weave

      • etc

  • Secrets

    • can be encrypted at rest with EncryptionConfiguration resource

      • apiserver needs to be configured with --encryption-provider-config option

    • secrets are encrypted at write

  • Scheduling

    • Resource quotas

      • limit size of resource requests and limits per namespace

    • Resource requests

    • Resource limits

      • memory: kills

      • cpu: throttling

    • Limit ranges

      • enforce setting resource limits

    • QoS classes

      • Guaranteed

      • Burstable

      • BestEffort

      • derived from request/limit settings

    • Dedicated nodes (sole-tenant nodes)

      • taints and tolerations

    • Affinity / anti-affinity

      • for example, which pods can co-exist in a node

      • -> pod isolation

    • Priority and preemption (alpha)

      • low and high priority pods

      • if high-priority pod can not be scheduled, low-priority pod will be killed

  • OpenPolicyAgent

    • higher-level enforcing of certain policy sets

Misc

  • Stateful workloads are more complex to manage in multi-tenant setups

  • Node security must be kept in mind, because the isolation features of Kubernetes won't help if the node is compromised

Related

  • Software architecture

    • multi-tenant

      • each of the customer in the single application instance and single database

      • single-point-of-failure

    • multi-instance

      • each customer has their own application instance and database

      • stability - no single-point-of-failure

      • each of the instances can be scaled seperately

      • data safety in case of for example security breach

Resources

PreviousDebuggingNextNetwork Policies

Last updated