# Notes My personal notes about things, mostly about tech. Hosted as GitBook at # aws # CloudWatch ## Logs Insights ### Log query examples Get all log entries with certain Kubernetes pod name and which log message contain the string "error" ``` fields @timestamp, @message | filter kubernetes.container_name='$CONTAINER_NAME' | filter @message like "error" | sort @timestamp desc ``` # EKS ## Service account IAM roles ### Obtaining CA thumbprint of OIDC provider AWS IAM identity provider needs to be configured with CA Thumbprint. More specifically, this is a SHA1 fingerprint (in lowercase and without colons) of the root CA certificate. Thumbprint can be obtained with openssl or other tools, but there couple of shortcuts: * Oneliner from GH issue (): ``` echo | openssl s_client -servername oidc.eks.${REGION}.amazonaws.com -showcerts -connect oidc.eks.${REGION}.amazonaws.com:443 2>&- | tail -r | sed -n '/-----END CERTIFICATE-----/,/-----BEGIN CERTIFICATE-----/p; /-----BEGIN CERTIFICATE-----/q' | tail -r | openssl x509 -fingerprint -noout | sed 's/://g' | awk -F= '{print tolower($2)}' ``` * `kubergrunt` tool (): `kubergrunt eks oidc-thumbprint --issuer-url $ISSUER_URL` ## Resources Amazon EKS Best Practices Guide for Security - # IAM ## Minimum privilege sets ### ECR #### Push image ``` "ecr:GetAuthorizationToken", "ecr:PutImage", "ecr:InitiateLayerUpload", "ecr:UploadLayerPart", "ecr:CompleteLayerUpload", "ecr:BatchCheckLayerAvailability" ``` # Key Management Service (KMS) ## Cross-account key access Setting up cross-account KMS access feels tricky sometimes. However, AWS have documentation page describing the process and minimal set of IAM actions needed. # security # Attacks against AWS infrastructure ## Attack types ### ECS Task Definition exploit In their blog post, Rhino Security Labs describe how to weaponize ECS task definitions to steal access keys and other information: # vpc # AWS Transit Gateway > A transit gateway is a network transit hub that you can use to interconnect your virtual private clouds (VPC) and on-premises networks. * Transit Gateway (TGW) makes it possible to hub-and-spoke network design with multiple VPCs, VPNs, and on-premise networks. * TGW acts as hub and other networks connect it. TGW controls routing between networks * This makes network management easier when comparing to connecting multiple VPCs with only VPC peering * TGWs can be peered with other TGWs with Transit Gateway Peering ## Attaching VPC to Transit Gateway * Share Transit Gateway resource to target account (which has the VPC to be attached) using AWS Resource Manager principal association * Accept the shared resource from target account * From target account, attach the VPC to TGW using Transit Gateway VPC attachment * Accept the attachment from account which has the VPC ## Resources: * * # azure # Azure AD ## Creating Azure AD users with Terraform Azure AD users can be managed with Terraform. The following is almost complete example for the key parts: ``` data "azuread_domains" "aad_domains" {} locals { aad_domain = data.azuread_domains.aad_domains.domains[0].domain_name # assumes that only one domain exists users = [ "Foo Bar", "Bar Baz" ] } resource "random_password" "rnd_pw" { for_each = toset(local.users) length = 16 special = true number = true keepers = { name = each.key } } /* * Sets initial password to one generated with Terraform. * This password is stored to state. Password change is forced * for new users and changes to passwords are ignored by Terraform, * so Terraform will not override the new password */ resource "azuread_user" "users" { for_each = toset(local.users) user_principal_name = "$SOME_REPEATABLE_PATTERN@${local.aad_domain}" display_name = "${each.key}" password = random_password.rnd_pw[each.key].result force_password_change = true lifecycle { ignore_changes = [password] } } ``` ### Fetch initial credentials from Terraform state Here is a command to fetch users and their initial passwords from Terraform state with jq: `terraform show -json | jq ' .values.root_module.resources | map(select( .address |contains("azuread_user") )) | map({user_principal_name: .values.user_principal_name, password: .values.password})'` # Azure CDN ## HTTP to HTTPS redirect CDN endpoint rules engine can be used to redirect HTTP request to HTTPS port. In Terraform, this can be achieved with the following `delivery_rule` block for `azure_endpoint_resource`: ``` delivery_rule { name = "EnforceHTTPS" order = "1" request_scheme_condition { operator = "Equal" match_values = ["HTTP"] } url_redirect_action { redirect_type = "Found" protocol = "Https" } } ``` Azure docs shows how to do this from Portal ## Resources * Migrating a Static Site to Azure with Terraform - * Set up the Standard rules engine for Azure CDN - # DNS in Azure ## Private DNS ## Resources * Azure Private Link and DNS - * Azure Private Link and DNS - * Azure Private Link and DNS – Part 2 - * Azure Private Endpoint DNS configuration - # Hub-spoke network topology * The hub-spoke is a network topology in Azure where one virtual network acts as a hub, and other virtual networks as spokes. * In usual setting, the virtual network acting as a hub hosts central services and connections, like VPN connection to on-premises network. The spokes are connected to the hub with virtual network peering, mening that spokes can use services and connections provided by the hub. * In addition, the hub can control connectivity and security, for example using Azure Firewall * With default configuration, spokes are isolated from each other ## Resources * Hub-spoke network topology in Azure - * Tutorial: Create a hub and spoke hybrid network topology in Azure using Terraform # Identity and access management ## Azure AD * authentication system in Azure * also authentication system for other Microsoft cloud services like Office 365 etc * subscriptions are created to Azure AD tenant * there can be multiple subscriptions under single Azure AD tenant * Azure AD Connect can synchronize users between on-premise AD and Azure AD ### Managed Identities * Internally, managed identities are service principals of a special type, locked to only be used with Azure resources * when the managed identity is deleted, the corresponding service principal is removed. * Azure takes care of rolling the credentials used by Managed identity, so there is no need for manual credential rotation * There are two types of managed identities * system-assigned managed identity * enabled directly on an Azure service instance. Tied to a lifecycle of that service. * user-assigned managed identity * created as a standalone Azure resource. This identity can be assigned to one or more Azure service instances. * has lifecycle of its own ## RBAC Allows to authorize to user specific actions to specific Azure resources * Roles * Role definition is a collection of permissions * Three built-in roles * Owner - can perform all actions on all resource types * Contributor - similar than Owner, but does not allow managing RBAC itself * Reader - can perform all read actions on all resource types * Azure also provides resource specific built-in roles * Custom roles can be created * Access to resources can be granted by assigning role to security principal, which are * User * Group * Service Principal * Assignment structure * Security Principal * Role * Scope * Subscription * Resource Group * Resource * When assignment is done on subscription or resource group level, the resources below in this hierarchy inherit this assignment ## Resources * Azure Essentials: Identity and Access Management - * Understand Azure role definitions - # Azure Landing zones > The term landing zone refers to an environment that's been provisioned and prepared to host workloads in a cloud environment like Azure. * Azure Landing Zones are part of the Microsoft Cloud Adoption Framework (CAF). Landing Zones sets best practices for automatically provision well-architected and secure Azure subscriptions to be consumed by the users in the organization. * Landing Zones consists of multiple design areas, like identity management, network topology and connectivity, and resource organization * They can be multi-layered and they can be implemented on different scale, for example like setting up just the basics or going full enterprise-scale ## Landing zones with Terraform [Azure Cloud Adoption Framework landing zones for Terraform](https://github.com/Azure/caf-terraform-landingzones) is a project which presents an idea how to build Landing Zones with Terraform. There are also some ready-made modules, but as for now these did not look very mature yet. ## Resources * What is an Azure landing zone? - * Azure Cloud Adoption Framework landing zones for Terraform - # Storage ## Blob storage ### Access control Azure portal uses "Access key" authentication method by default. This means storage account key is used for authentication. Authentication method can be switched to Azure AD user account from blob container overview page. ### Syncing objects from AWS S3 bucket to blob container Objects can be synced from AWS S3 bucket to Azure blob container using AzCopy. For example: ```bash azcopy cp "https://$BUCKET_NAME.s3.$AWS_REGION.amazonaws.com" "https://$STORAGE_ACCOUNT_NAME.blob.core.windows.net/$CONTAINER_NAME" --recursive ``` It seems that AzCopy can only read AWS credentials from `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` environment variables. So, those need to be exported. If session credentials are used, `AWS_SESSION_TOKEN` needs to be also exported. On the Azure side, the data layer has in a sense its own access policies. Even though, an user has Owner role in subscription, blob data level roles might need to be added. # certifications # aws-sa-pro ## AWS Certified Solutions Architect Professional ### IAM #### Security Token Service (STS) * If permission policy of a role is changed, the change is immediately reflected to the privileges of temporary credentials obtained with AssumeRole * If temporary credentials are leaked, they can revoked by doing the following * Update role permission policy with AWSRevokeOlderSessions inline DENY for any sessions older than now * In the IAM role UI, there is a helper for that in "Revoke sessions" tab. This will apply the inline policy to role * This will revoke all temporary credentials which were obtained before the current, but legitimate users can get new credentials with AssumeRole call * Fetching EC2 instance role temporary access keys from metadata endpoint * Get name of role: `curl http://169.254.169.254/latest/meta-data/iam/security-credentials` * Get the credentials: `curl //169.254.169.254/latest/meta-data/iam/security-credentials/$ROLE_NAME` #### Permission boundaries * Only identity permissions are affected by permission boundaries * permission boundary can be applied to IAM Users or IAM Roles * They don't grant access by themselves, but only define maximum permissions identity can receive * Boundaries can be used to solve delegation problems with IAM permissions #### Policy evaluation logic * Within same AWS account * Explicity Deny * Organization Service Control Policies (SCPs) * Resource Policies - can allow access and evaluation stops * IAM Identity Boundaries * Sessions Policies * Identity Policies * Cross-acccount access - different AWS accounts * needs explicit allow from source account identity policy * allow from destination account to source account #### S3 cross-account access * If bucket ACLs are used, canonical user IDs need to be used to reference other AWS accounts * Cross account S3 access can be done using three methods * ACLs - context is the whole bucket and the object * Bucket policies - policies can be applied to bucket or objects * Assuming role in destination account, which has access to S3 and/or objects ### Resource Access Manager (RAM) * RAM can be used to share resources between AWS accounts * Shared service needs to have RAM support * As AZ names are rotated between different accounts, AWS introduced AZ ids, which are consistent across accounts * Resources shared with accounts in same AWS Organization are automatically accepted, if sharing within organization feature is enabled * This enables for example to create shared services VPC, which can be shared to other accounts * Resource-wise, participant accounts only have read-only access * each account owns the resources, which it creates to VPC * as the resources are in the same VPC, there is network-level connectivity * Some resources can be shared with any AWS accounts, some can only with AWS acxounts in the same organization * Name tag is not shared ### Service Quotas * Service quota request template can be used to request quota changes to all AWS accounts in the same AWS organization * Cloudwatch Alarms can be created for reaching certain percent of the quota * Changes to quotas can be done from * Console Service Quota service * Using support ticket * Using AWS API, for example with AWS CLI ### Advanced Identities and Federation * Use SAML 2.0 (Security Assertion Markup Language) federation if * enterprise identity provider which supports SAML 2.0 is used * there is existing identity management team and federation source is not migrated * single source of truth with more than 5000 users * SAML 2.0 federation uses IAM Roles and AWS Temporary Credentials (max. 12h duration) * AWS SSO is preferred for identity federation if there are no exclusions * SSO can also manage access for external applications * SSO permissions sets bind SSO Users or Groups to certain permissions in one or more AWS accounts * AWS Cognito * provides authentication, authorization, and user management for web/mobile apps * two main concepts * User Pool - allows to sign-in and get JSON Web Token (JWT) * user directory, sign-up, sign-in, and etc. * Identity Pool - allows you to offer access to Temporary AWS credentials * unauthenticated identities - guest users * swap external identities (FB, Google, Apple, SAML 2.0, User Pool) for short term AWS credentials to access AWS resources ### AWS Workspaces * AWS Workspaces use Directory Service for authentication and user management * simple AD * managed full AD * external AD with AD connector * Workspace use VPC networking to access internet or on-prem networks, so there are extra costs * Can access on-prem resources using VPN or direct connect * Not HA, because they use only on AZ ### Directory Service #### Microsoft AD * Native Active Directory * HA, min. 2 AZs * Support one-way and two-way external and forest trusts with on-prem AD * Supports more than 5000 users * Automatic patching and maintenance * Supports AD native schema extensions * Supports MFA with Radius #### AD connector * redirects requests to existing directory services * no directory data stored in AWS * requires working network connection to existing AD * requires two subnets in VPC, different AZs ### VPC #### DHCP in VPC * By default resources use Amazon provided DNS server in VPC * Route53 resolver in VPC * EC2 instances get private hostnames * EC2 instances get public hostnames if public ip is configured * If wanted, user can provide own DNS server * DHCP is configured with option sets * immutable, so can't edited after creation * option set can be associated with zero or more VPCs * Each VPC max one DHCP option set * When VPC DHCP option set is changed, it requires DHCP renew * Default gateway in option set can't be changed * it's always VPC Router (subnet +1) * DNS server can be changed, by default it's Route53 Resolver (subnet +2) #### VPC Router * HA across all AZs in the region * Routes traffic between subnets * Routes from external networks into the VPC and from VPC into external networks * Has interface in every subnet (subnet+1) * Controlled usin route tables * Every VPC in created with default route table * it's default for every subnet in VPC * Custom route table can associated for subnet * Subnet are associated with one route table * Most specific route wins in route table * Local routes always take precedence #### Network Access Control Lists (NACL) * Every subnet has associated NACL * NACLs filter traffic crossing the subnet boundary * Connections within subnet is not impacted by NACLs * There are inbound and outbound rules * Rules match dst/src ip range and port. Rules can allow or deny * Rules are processed in order, lowest number first. Once match occurs, processing stops * * is an implicit deny, if nothing else matches * NACLs are stateless, so both inbound and outbound traffic needs to be allowed * VPC is created with default NACL * Custom NACLs are created for a specific VPC and are initially associated with no subnets * At first custom NACLs has only deny all rules * NACLs can be used to deny specific IPs or ranges, where as security groups can't do that * Generally security groups are used to allow traffic, NACLs to deny * Can't reference logical resources and can't be assigned to logical resources * One NACL can be associated with many subnets ======= ### VPC #### Security Groups * No explicit deny * Allow referencing logical resources, like other security Groups * self-referencing can be done * Attached to ENI (Elastic Network Interface) #### BGP (Border Gateway Protocol) * AS - Autonomous System - routers controller by one entity, network in BGP * ASN Range is 0-65535, controlled by IANA * private range is 64512-65534 * Operates over tcp port 137 * Peering between ASs is manually configured * Path-vector protocol is exchanges the best path to destination between peers - ASPATH * iBGP - internal BGP - routing within an AS * eBGP - external BGP - routing between ASs * in path table, i is the origin * By default, BGP does not take link speed to account, only the path length * this can overridden with AS Path Prepending * this is done by adding artificial ASs to AS Path #### AWS Global Accelerator * Anycast IPs allow ip address to be in multiple location at the same time * Routing moves traffic to the closest location * Global Accelerator Edge Location has pair of anycast IPs * traffic can be the routed to the closest location * When traffic is directed to Global Accelerator ip addresses, rest of the transit is using AWS links and networks * Works with TCP/UDP - difference from CloudFront * Does not cache anything #### Site-to-site VPN * Logical connection between a VPC and on-prem network * encrypted using IPSec * Runs over public internet * HA - when correctly architected and implemented * Quick to provision * Virtual Private Gateway (VGW) is associated to VPC and can be target in route tables * has endpoints in each AZ * Customer Gateway (CGW) logical device to act as VPN connection endpoint * VPN connection - linked to one VGW and one CGW * Static VPC means that VPN connection is statically configured with ip ranges for both sides * Customer on-prem router is single-point-of-failure - partial high-availability * There can be two on-prem customer gateways, so the whole architecture is HA * Dynamic VPN uses Border Gateway Protocol (BGP) * Routers exchange routing information * can communicate status of links between customer gateways and AWS side * if route propagation is enabled, routes are automatically added * AWS VPN speed limitation is 1,25Gbps and there is encryption overhead * Latency can be inconsistent, because traffic is routed over public internet * Cost - AWS hourly cost, GB out cost, data cap (on-premises) * Can be used as backup for Direct Connect * Can be used with Direct Connect #### Transit Gateway * Network transit hub, connecting VPCs to an on-premises network using Direct Connect or Site-to-Site VPN connections * Network Gateway object - HA and scalable * Other AWS network resources uses attachments to connect to Transit Gateway * VPC * Site-to-site VPN * Direct Connect Gateway * Is capable of transitive routing * Can be peered with another TGWs - cross-account or cross-region * TGWs can be shared to other accounts using Resource Access Manager (RAM) * One DX gateway can be connected to three TGWs * By default, Transit Gateway has one default route table, which has routes to attached network resources * all attachments use this route table for routing decisions * all attachments propagate routes to it * the two above-mentioned combined means all attachments can route to all attachments * Routes are not propagated over peering attachments (two peered TGWs) - static routes need to be used * Use unique ASNs for future route propagation features * Public DNS to private IP resolution is not supported over peers * Up to 50 peering attachments per TGW * different regions and accounts * Data is encrypted * Attachments can be only associated with one route table * Route tables can be associated with many attachments * Attachments can propagate to multiple route tables - even those they are not associated with * Route table association is used when data is exiting an attachment * Route table propagation controls which route tables are populated by routes known by the attachment * The above-mentioned features can be used to create route isolation #### Advanced VPC Routing * IPv4 and IPv6 are handed separately within a route table * Route table default limit is 50 static routes and 100 dynamic (propagated) routes * Main route table is implicitly attached to all subnets * When custom route table is associated to subnet, main route table association is removed * However, if custom route table association is removed, main route table applies again * It's not possible to have subnet without route table association * Propagated route tables can be enabled per route table * More specific route always gets prioritized (for example /28 is selected over /32) * If prefix length is equal for same route, static route is selected over propagated one * For the same routes with same prefix lenght learned dynamically, the precedence order is the following: * Direct Connect * VPN Static * VPN BGP * AS\_PATH (Path between two ASNs, distance between systems) * Ingress routing * Gateway route tables can be used to control actions on inbound traffic for gateway (for example IGW) * for example forward it to security appliance #### IPSEC VPNs * Group of protocols to setup secure tunnels across insecure networks between two peers (local and remote) * Only "interesting" traffic is sent over the tunnel * Provides authentication and encryption * IPSEC has two phases * IKE phase 1 (slow and heavy) - Internet Key Exchange * authentication using pre-shared key or certificate * uses asymmetric encryption to agree on and create a shared symmetric key * end result is IKE SA created (phase 1 tunnel) * Uses Diffie-Hellman for key exchange * IKE phase 2 (Fast & Agile) * uses the keys agreed on phase 1 * agree encryption method, and keys for bulk data transfer * end result is IPSEC SA (security association) * There are two types of IPSEC VPNs * policy-based * rule sets match traffic per pair of security associations * allows network to have different security settings for different type of traffic * route-based * target matching (prefix) matches single pair of security associations * Tunnels have two ends, left and right. Customer and AWS, meaning local and remote. * IPs of these ends are referred as AWS outside IP and customer outside IP * Inside the tunnels there are AWS inside IP address and customer inside IP address * Routing and raw data travels through inside of the tunnel - encrypted traffic runs outside of the tunnel across public internet ## Accelerated Site-to-Site VPN * By default S2S VPN traffic transit is over public network * When S2S VPN is used in accelerated mode, AWS Global Accelerator network is used * VPN tunnel IPs are global, and connections are routed to the closest global accelerator edge location * Public internet is only used for minimal amount of time to route traffic to nearest edge location * Acceleration can be only enabled when creating a TGW VPN attachment, not using VPNs with VGW * Fixed accelerator cost and a transfer fee * New advanced S2S VPNs features are usually only released for TGW VPN, so the default recommendation is to use it ## AWS Client VPN * Managed OpenVPN product * Client VPN endpoint is associated to one VPC * Also associated with one or more target networks * After above-mentioned associations, Client VPN product starts to charge * Billing is based on * Number of subnet associations (flat rate) * hourly fee of Client VPN connection * One subnet per AZ * Defined route table is pushed to clients * By default, all traffic is routed through Client VPN * Pushed route table takes precedence over local route tables of clients * Split-tunneling enables clients to route only selected traffic through the VPN tunnel * not the default mode * needs to be separated manually * Auth methods * identity based * AWS Directory Service, for example * certificate based * User needs to create certificates and upload them to the AWS ACM * identity-based authentication only needs server certificate * Subnet association needs to be done seperately after the Client VPN Endpoint is created * Accessed networks need to be configured with authorization rules ## Direct Connect * Physical connection to AWS region * 1, 10, or 100 Gbps * Connection goes from business premises to DX Locatino to AWS Region * Orderind Direct Connect is in fact ordering a network port at a DX Location * Costs * Port Hourly costs * Outbound data transfer * Because the connection is physical, there is no resilience or high-availability * Provides low and consistent latency * Best way to achieve high speeds for hybrid networking * Can be used to access AWS private services (in VPCs) and AWS public services without internet connection * Internet connection needs proxy or other networking appliance * Cross Connect is a connection between AWS cage and customer/comms partner cage at the DX location # Certified Kubernetes Administrator Random notes mostly from watching Cloud Native Certified Kubernetes Administrator (CKA) material from LinuxAcademy ## Some commands Check master component statuses: `kubectl get componentstatus` Check available api-resources in the cluster: `kubectl api-resources` ## Cluster bootstrap and creation ### Custom cluster (manual install) * The recommended way is to run components as pods except kubelet which needs to be run on the node itself. That's because kubelet is responsible for runnning everything else as pods. * However, the components can be run on the node itself as binaries. ### Creating HA Kubernetes cluster **Creating cluster with HA control plane protects from kube-system components failure.** * Each component of the master node can be replicated * However, some components may need to stay in standby state, in order to eliminate conflicts between replicated components * Only one scheduler and controller manager can be active at a time, others need to be in standby * Standby mode is achieved by leader-election mechanism * This is done with `endpoint` resource, which annotation tells the current leader. Once becoming the leader, it must update the annotations. This happens by default every two seconds * Replicating `etcd` * Replicated etcd can be in two topologies * stacked topology - each control plane node creates its own local etcd and only communicated with it * external topology - etcd is external to the Kubernetes cluster * etcd uses Raft consensus algorithm which requires majority. In order to have majority, there needs to be odd number of etcd instances ## Backing up a cluster * Mostly it comes down to backing up the etcd * Grab and install the etcd-client () * Create snapshot with: `ETCDCTL_API=3 etcdctl snapshot save snapshot.db --cacert /etc/kubernetes/pki/etcd/server.crt --cert /etc/kubernetes/pki/etcd/ca.cert --key /etc/kubernetes/pki/etcd/ca.key` * Also backup certificate and keys from `/etc/kubernetes/pki/etcd/`, they are needed in restore * Recovery examples ## Pod and node networking ### Pod and node networking * Linux networking namespaces are used for pod to network connectivity. * Pod has a ip address and virtual network interface pair. Another one is given to the pod (eth0) and another one is on node. * Pause container is used to hold a network namespace for a pod * Pod to pod node communication within node is done with Linux ethernet bridge. Bridge initiates communication with ARP request ### Node to node - CNI * Pod to pod communication between different nodes is done with a Container Network Interface (CNI) * simplified, the source ip address is switched from pods ip to nodes ip. Otherwise, the router would drop the packet * this could be also done with Layer 3 routing, but it would be complicated to manage * CNI is a network overlay, creating tunnel between nodes. This is done by encapsulating packets * CNI is not built-in to Kubernetes, but can be installed as add-on * Following are popular CNI plugins * calicoho * flannel * WeaveNet * Romana * Kubelet needs to know that CNI plugin will be used * In `kubeadm` this done with `--pod-network-cidr` parameter **Networking from internet** **Services** * Service provides one virtual interface, distributing traffic to pods * kube-proxy controls `iptables` for traffic routing * Endpoint object is created with service and it keeps cache of pod ip addresses belonging to that service * Different service types * ClusterIP * NodePort * LoadBalancer (extension to NodePort) * LoadBalancer can be set to favor pod on a node with `externalTrafficPolicy=Local` annotation. Otherwise the request could be routed to a pod on a another node, which creates latency ### Ingress * Ingress is like a LoadBalancer, but it can expose multiple services. * It exposes http/https routes outside of the cluster to the pods inside * It also provides Layer 7 routing, based on hostname or path etc. ### DNS * From version 1.13, coredns is the default DNS service for the Kubernetes * service name is still `kube-dns` for backward compatibility * Pods and services are assigned DNS names * Headless service is a service without a cluster ip (`clusterIP: None`). It will respond with set of ips, instead of one * Pods DNS config can be managed with `dnsPolicy` and `dnsConfig`, otherwise the pod will inherit nodes DNS config * each ip will point to individual pod of the service ## Scheduling * Scheduling decisions can be modified with your own scheduling rules * Scheduler goes through the list of checks when scheduling a pod to a node * Affinity rules configure for example if the user wants that certain pod goes to a node with certain label * By default, scheduler tries to spread the pods that belong to same replica set to different nodes * Multiple schedulers can be run at the same time * Pod select their scheduler with annotation * Taints allow nodes to repel certain pods * Tolerations are set to pod to allow scheduling to a node with matching taint * Resource requests (for example CPU and memory) specify the amount of resources a container is quaranteed to get * Resource limits make sure that a container does not go above certain value * Limit can never be lower than request * DaemonSets does not use the scheduler and they run one pod per node * Scheduling events can be checked (namespace scoped) with a command: \`kubectl get events\`\` * Scheduler logs can be checked like any other logs ## Application lifecycle management ### kubectl commands Kubectl commands for manipulating application lifecycle * Create a deployment (with a record for rollbacks) `kubectl create -f deployment.yaml --record` * Check status of deployment (rollout) `kubectl rollout status deployments $DEPLOYMENT` * Deployment creates the underlying replica set * Scale deployment `kubectl scale deployment $DEPLOYMENT --replicas=5` * Expose the deployment, this is same as creating a service `kubectl expose deployment $DEPLOYMENT --port $PORT --target-port $TARGET_PORT --type $SERVICE_TYPE` * Perform rolling update by setting a different Docker image `kubectl set image deployments/$DEPLOYMENT app=$REPO/$IMAGE:tag` * Undo rollout and rollback `kubectl rollout undo deployments $DEPLOYMENT` * Pause an ongoing rollout `kubectl rollout pause deployment $DEPLOYMENT` * Resume the paused rollout `kubectl rollout resume deployment $DEPLOYMENT` ### Readiness probe * Readiness probe states when the pod is able to respond to requests ### Passing configuration options * Environment variables * ConfigMaps and Secrets (for sensitive data) * Can be passed via environment varialbes * Or mounted as volume ## Data management in the cluster * Data can be persisted using volumes, for example by provisioning disk from the cloud infrastructure and attaching that to the pod ### Persistent Volumes (PV) * Persistent Volume resousrce provides higher level abstraction * PV access modes * defines if the volume can be written and read by multiple nodes or just single node * RWO (ReadWriteOnce) - only one pod can mount the volume for writing and reading * ROX (ReadOnlyMany) - multiple nodes can mount the volume for reading * RWX (ReadWriteMany) - multiple nodes can mount the volume for reading and writing * Only one mode can be used at the same time * Mount capability is for node, not the pod * `persistenVolumeReclaimPolicy` defines what happens to the PV if the claim is released. * Retain - keeps the PV * Recycle - Volume can be reused by another PVC * Delete - PV will be deleted * Volumes that are already in use by a pod are protected against data loss. * Even if PVC is deleted, volume can be accessed by the pod - Storage object in use Protection ### Persistent Volume Claims (PVC) * When referencing to a PV from a pod, PVC needs to be used * Abstracts away the storage layer details from the developer * `storage` attribute can't be higher than in PV resource * PVC defines the used access mode * PVCs are namespace specific ### Storage class * Storage class provides an easy way to create PVCs without defining PV * Major cloud provide their block disk services as storage classes * Also hostPath can be used to provision worker node disk. ## Security ### Apiserver authentication * Evaluation is done whether the request is coming from service account or "normal user" * Kubernetes do not have User object, but users can authenticate themselves with * private key * user store * file with a list of user names and passwords * ServiceAccounts (SA) are for "machine access", if the software needs access to Kubernetes API * token (specified as Kubernetes secret) is used for authentication * SA can be added to pod by specifying it in the manifest. * Default service account is used if SA is not specified * `kubectl config view` lists address of the cluster in use and authentication mechanism * It is a good idea to add separate SA for separate pods or replicated pods ### Authorization (RBAC) * Authentication = who can access * Authorization = what given entity can do * In Kubernetes authorization is done by RBAC * Roles and ClusterRoles * what can be done for which resource * RoleBindings and ClustrerRoleBindings * who can do it * Role and RoleBinding is namespace level * ClusterRole and ClusterRoleBinding is cluster-level * Bindings can be done against user, service-accounts or groups ### Network policies * Govern how pod communicate with each other * Ingress and egress rules * CNI needs to support Network polices. For example * Canal * Calico * PodSelectors and NamespaceSelectors can be used to select desired pods for which the policies apply ### TLS certificates * CA is used to generate certificates * Certificates are used to authenticate to apiserver * To create new certificates, first a CSR (certificate signing request) needs to be generated using chosen tool * like `cfssl` and `cfssljson` * After that, `CertificateSigningRequest` object needs to be create to Kubernetes API * CSR needs to be approved: `kubectl certificate approve $CSR_NAME` ### Secure images * Images come from container registry, by default Docker Hub * ImagePullPolicy should be `Always` as other users have access to local Docker images even if they don't have ImagePullSecrets * ImagePullSecrets can be used to define credentials to private Docker registries ### Security Contexts * Can be used to limit pod or container access to certain objects. * For example limit container running as root: `runAsUser: $UID` or `runAsNonRoot: true` * If container needs kernel capabilities of a need, it can be run in privileged mode: `privileged: true` * Kernel features can be lock-down from a pod by setting capabilities in manifest: ``` capabilities: add: - SYS_TIME ``` * Capabilities can dropped with `drop` keyword * Container filesystem can be set to read-only with `readOnlyRootFilesystem` * If security context is specified in pod level, all containers of the pod inherit it * However, it can be also specified for certain container ### Secrets * Can be passed to a pod or container as environment variables or as a file via volumes * Applications can dump their environment variables for example in crash reports, so passing them as volumes are preferred * Secrets shared as volumes used `tmpfs`, so they are written to memory, not to disk ## Logging and monitoring * metrics-server expose metric data via Metrics API on node and pod level * `kubectl top node` * `kubectl top pod` * Liveness and readiness probes check if the pod is running properly and if container is ready to receive client requests * Container logs are in `/var/log/containers` directory * Kubelet logs are in `/var/log` * If container writes to STDOUT/STDERR, logs can be checked with `kubectl logs $POD_NAME -c $CONTAINER_NAME` # containers # Examples ## Docker Bypass network, pid, and IPC namespaces. Mount host filesystem to `/host` and chroot to `host` in container. ``` docker run -ti --privileged --net=host --pid=host --ipc=host --volume /:/host busybox chroot /host ``` ## Resources * The Most Pointless Docker Command Ever - # Linux Container Primitives > Containers are an abstraction over several different Linux technologies ## Linux Kernel ### Namespaces * "What you can see" * isolation mechanism for processes * changes to resources within namespace can be invisible outside the namespace * examples of available namespaces * network * filesystem (mount) * processes * namespaces can be shared with process * OR process can have its own namespaces, like a container * commonly used namespaces in containers * network * `veth` devices (or pairs) can connect different namespaces * Docker uses separate network namespace per container * by default Docker connects container network with `veth` pair to Linux bridge to allow outbound connectivity * in Kubernetes, containers in a pod share the same network namespace * mount * used to give containers their own filesystem * `procfs` virtual filesystem * introspection of Linux kernel data * Linux syscalls are used working with namespaces * clone * unshare * namespace can not be empty * tools to enter namespaces * nsenter * ip-netns * Leveraging namespaces with containers * `nsenter` or `ip netns` to troubleshoot container networking * monitor containers by entering the `pid` namespace * access binaries in your containers with the mount namespace * Further reading: `man 7 {namespaces, pid_namespaces, user_namespaces}` ### Control groups * "What you can use" * Cgroups * Organizes all processes in the system, whether they are in container or not * Account for resource usage and gather utilization data * Limit or prioritize resource utilization * Cgroup system is an abstract framework * subsystem are concrete implementations * memory * cpu time * block I/O * number of discrete processes (pids) * devices * etc * subsystems are independent * cgroups can be interacted with throught virtual filesystem * typically in `ace}/sys/fs/cgroup` * `tasks` file holds all pids in cgroup * get cgroups for pid: `cat /proc/$PID/cgroups` ### Filesystems * images are representation of a filesystem * Docker uses image layers * Layers are typically implemented with union filesystems * efficient use of storage when there are only minor modifications to image * Overlay filesystem is built in to Linux * Docker's default layer storage uses the overlay filesystem * Leveraging with Docker * locate files in your layers * examine which layers files contribute to disk usage * understand the impact of writable files in your containers ## Container runtimes * software tool that configures Linux primitives to create and run containers on a host * Examples include * Docker * containerd * runc * CRI-O * systemd-nspawn * `runc` is OCI (Open Containers Initiative) reference implementation which powers Docker, containerd and CRI-O * OCI runtime spec * containers are "bundles" * Filesystem * JSON document * describes underlying technologies which "make the container" like cgroups, namespaces, Linux capabilities, and more ## Resources * Linux Container Primitives: cgroups, namespaces, and more! - * CGroups documentation - * Overlay filesystem documentation - * Golang UK Conf. 2016 - Liz Rice - What is a container, really? Let's write one in Go from scratch - # databases # Relational databases * Tackling performance problems with more hardware usually just delays problems * Instead of just provisioning bigger database instance, one should try to optimize database schema and queries * normalizing data is usually the first thing to do and check * normal forms * in schema design, one should avoid data duplication * Primary keys can be also short strings instead of integers * with this, some information can be retained * PostgreSQL has auto compress feature (TOAST) ## Using EXPLAIN * shows the query plan * using `EXPLAIN ANALYZE` actually executes the statement and run time statistics are added to display ## Optimizing performance * Sequential scans are usually bad for performance ### Indices * To simplify, indices in tables turn sequential scans to index scans * Most of the regularly used queries should have index * however the come with a cost * especially when writing to a table * What if I have sequential scans even after adding an index * small table (not lots of rows) * on multi field indices, fields can be omitted only from the end * using function in where clause * however function can be used in index itself * Postgres specific reasons why indices won't work * VACUUM hasn't been run, so table statistics are not updated ## Object-relational mappings (ORM) * Usually it's a good thing to write simple code with abstractions * however, abstractions might have a performance cost * Using ORM is usually recommend, but when performance is critical it could be better to write pure SQL ## Sharing responsibilities between application code and database * Let the database do the heavy lifting * when database works, app sleeps * the is overhead going back and forth between application and database * Prefer small number of big queries instead of big number of small queries ## Resources * [PostgreSQL 14 Internals ebook](https://postgrespro.com/community/books/internals) * [Postgres Weekly newsletter](https://postgresweekly.com/) ## References * My own notes and interpretations on my colleagues excellent presentation with title "Your Customer Might Not Need a Bigger Database Instances" (17.01.2025) # gcp # IAM ## Main concepts ### Permissions * Permissions are fine-grained permission model defined by GCP services * structure: .. * example: storage.buckets.create ### Roles * Roles are abstractions to group permissions together * Primitive roles * broad access * spans services * three roles * owner * editor * viewer * Not recommended for production usage * Predefined roles * narrower access * permissions to a single service * for example * service admin * service viewer * Custom roles * create from scratch or from predefined roles * can be used to combine roles * or remove or add permissions to role ### Bindings * Bind roles to users, groups (can be nested) or service-accounts ### Policies * Policies connect the resource, roles and members via bindings ### Resource hierarchy * Groups resources according to organization structure * Bindings are inherited down and apply to all nodes under the layer they are applied * Project provides trust boundary and resource isolation * The higher up, the more powerful (organization level) * The lowest level is attaching to certain resource ### Service accounts * Identity of a service * They are principal (identity) and resource themselves ## Best practices * Grant roles to groups, not users. Provides scalability for the future * Grant least privilege * Avoid powerful operations, like: * Set IAM policy * Act as a service account * Retain audit logs * Forward events for centralized logging (Stackdriver logging event) ## Resources * Better Practices for Cloud IAM (Cloud Next '18)') - # git # Git ## Multi-user with dirvenv To summarize, export following environment variables in `.envrc`: ``` export GIT_AUTHOR_NAME="Foo Bar" export GIT_AUTHOR_EMAIL="foo@bar.com" export GIT_COMMITTER_NAME="Foo Bar" export GIT_COMMITTER_EMAIL="foor@bar.com" ``` See for more details. ## Resources * # golang # Building Go projects ## Dependencies managed with `dep` In projects where dependencies are managed with `dep`, the local project needs to be located in `$GOPATH`. Because of this, using container as a build environment is the easiest solution in most cases. For example to build the `kube-psp-advisor` project: ``` docker run -it --rm -v "$PWD":/go/src/github.com/sysdiglabs/kube-psp-advisor golang:1.11 /bin/bash ``` # Concurrency ## Goroutines * Goroutine is a function that runs independently of the function that started it * Goroutine can be any function that's called after special keyword `go` * Frequently used to run function in the "background" while the main part does something else ## Channels * Pipeline for sending and receiving data * Channels allows one goroutine to send structured data to another goroutine * Channels can be used to block the main goroutine until all other goroutines finish their execution * Like network socket, they can be unidirectional or bidirectional * Can be short-live or long-lived * Declared with the keyword `chan` along with a data type * Default value of a channel is `nil`, so value needs to be assigned * Value can be assigned with the `make` function * Writing and reading from channel is done with arrow syntax, where the arrow points the direction * Sending and reading are blocking operations, meaning * when data is sent to channel using a goroutine, it will be blocked until data is consumed by another goroutine * when receiving data from channel using goroutine, it will be blocked until the data is available in the channel * To avoid deadlocks, sender can close a channel. This means channel can't be communicated over * Buffered channels can be used to create channels which don't block until channel capacity is exceeded * For example: `make(chan int, 100)` ## Waitgroups * Waitgroup allows to block a specific code block to allow a set of of goroutines to complete execution * The following are the main methods of waitgroups * `add` - waitgroup act as counter holding the number of goroutines or functions to wait for. When counter is 0. the waitgroup releases the goroutines * `wait` - blocks the execution of the application until the waitgroup counter becomes 0 * `done` - decreases the waitgroup counter by a value of 1 ## References * * # Project structure ## References * Standard Go Project Layout - # infosec # SSH ## Create keypair and write private key to STDOUT ```bash mkfifo key && ((cat key ; rm key)&) && (echo y | ssh-keygen -N '' -q -f key -t ed25519 -b 4096 > /dev/null ``` Source: # SSL ## Check server certificate with OpenSSL ### With SNI ```bash openssl s_client -showcerts -servername www.example.com -connect www.example.com:443 /dev/null | \ openssl x509 -text ``` # Kubernetes > Kubernetes (K8s) is an open-source system for automating deployment, scaling, and management of containerized applications. # Admission Controllers * Admission controllers checks whether request to apiserver should be allowed or not (after authentication and authorization) * Dynamic admission controllers * can call validating service with webhook * this is beneficial for example in managed Kubernetes environments where API server can't be modified * configured with `ValidatingWebhookConfiguration` * Mutating admission controller can modify workloads, for example inject init containers * Kubernetes can send out `AdmissionReview` object to admission controller, which wraps the object which will be validated * Admission controller need then respond with the validation results to the apiserver ## Resources * TGI Kubernetes 119: Gatekeeper and OPA - # Autoscaling ## In general In general, autoscaling can be done on pod or node level in Kubernetes. In most cases, autoscaling is done on CPU or memory usage, or scheduling constraints. It can also be done on custom metrics and thresholds. ## Kubernetes requests and limits Kubernetes requests and limits associate with autoscaling in Kubernetes as they * requests * what the scheduler should carve out * limits * enforced at runtime (cgroup) * simplified, they are upper bound for resoure usage, making sure that pod does not consumere more resources than what's specified in limits * if request is not defined, value of limit is used in its place ## Kubernetes built-in components ### Horizontal pod autoscaler (HPA) * HPA is a controller and it will scale pods by adding new replicas * By default HPA will scale by CPU * Scaling should be set againt deployment, not replica set * For example: `kubectl autoscale deployment $DEPLOYMENT --cpu-percent=50 --min=1 --max=10` ### Vertical pod autoscaler (VPA) * In general, VPA scales resources available to pods vertically, meaning pod will get for example more memory and CPU * TGIK episode 097 did not touch this extensively ### Cluster autoscaler (CA) * As CA communicates with the API of a cloud provider, there are certain gotchas with the different cloud providers which should be taken into account when setting up CA * For example, currently CA can not handle ASGs which spans multiple AZs. Instead, it expects ASG per AZ * CA will scale nodes up if there are scheduling constraints due to lack of resources ## Resources * TGI Kubernetes 097, , released 08th November 2018 # Debugging ## Run ad-hoc container in cluster In general: ``` kubectl run -it $IMAGE_NAME --image=$IMAGE --rm --restart=Never -- $COMMAND ``` For example: ``` kubectl run -i --tty curl --image=tutum/curl --restart=Never --rm -- sh ``` # Multi-tenancy ## Why? * usually more cost-effective * easier access to shared resources, for example controllers * reduced management overhead * no need to wait for cluster and related infrastructure creation for new tenants ## Why not? * resource starvation * security aspects * hard to get right ## Use cases * These use cases are listed in David Oppenheimer's talk * Enterprise * users all from same organization * users are semi-trusted * personas * cluster admin * namespace admin * user * vanilla container isolation might be sufficient depending on the workload * inter-pod communication might be limited * to only within namespace * to other namespaces depending on the application topology * one application tier per namespace * Kubernetes as a Service (KaaS) / Platform as a Service (PaaS) * untrusted users running untrusted code * users can create namespaces and CRUD non-policy objects with their namespace(s) * needs stronger control plane, node, and network isolation * resource quotas per how much customer pays * Software as a Service (SaaS): multi-tenant app * customer does not access Kubernetes apiserver, but the application * in single instance for all customers model, Kubernetes multi-tenant models are not concerned as multi-tenancy is handled at application level * in multi-intance model, each customer has their own application instance * in this model proxy can interact with Kubernetes apiserver for creating new application instances * namespace per application instance * code is semi-trusted * if plugins are allowed (for example Wordpress) that code is untrusted * certain namespace might host shared infrastructure ## Isolation mechanisms * namespace isolation * best practice is to have namespace per tenant. Or several namespaces per tenant * label your namespaces so network policies can target them * with labels, network policies can target multiple namespaces * most of the isolation features expect this * Pod Security Policies * limit pod/container security contexts * for example, pod can not be run as root * Security contexts * RBAC * Network Policies * limit network access between tenants * need to have enforcing network overlay (CNI) * Calico * Weave * etc * Secrets * can be encrypted at rest with `EncryptionConfiguration` resource * `apiserver` needs to be configured with `--encryption-provider-config` option * secrets are encrypted at write * Scheduling * Resource quotas * limit size of resource requests and limits per namespace * Resource requests * Resource limits * memory: kills * cpu: throttling * Limit ranges * enforce setting resource limits * QoS classes * Guaranteed * Burstable * BestEffort * derived from request/limit settings * Dedicated nodes (sole-tenant nodes) * taints and tolerations * Affinity / anti-affinity * for example, which pods can co-exist in a node * -> pod isolation * Priority and preemption (alpha) * low and high priority pods * if high-priority pod can not be scheduled, low-priority pod will be killed * OpenPolicyAgent * higher-level enforcing of certain policy sets ## Misc * Stateful workloads are more complex to manage in multi-tenant setups * Node security must be kept in mind, because the isolation features of Kubernetes won't help if the node is compromised ## Related * Software architecture * multi-tenant * each of the customer in the single application instance and single database * single-point-of-failure * multi-instance * each customer has their own application instance and database * stability - no single-point-of-failure * each of the instances can be scaled seperately * data safety in case of for example security breach ## Resources * Multi-Tenancy in Kubernetes: Best Practices Today, and Future Directions - David Oppenheimer - * Managing a Multi-Tenanted Kubernetes Cluster in Production by Josh Bowen, Apigee - * Getting started with Kubernetes for your SaaS - * A Pathway to Multi-Tenancy in Kubernetes - * Kubernetes Multi-Tenancy Best Practices - * Cluster multi-tenancy - * Multi-tenant design considerations for Amazon EKS clusters - * AWS re:Invent 2019: Architecting multi-tenant PaaS offerings with Amazon EKS (GPSTEC337) - # Network Policies > A network policy is a specification of how groups of pods are allowed to communicate with each other and other network endpoints. ## Resource * Kubernetes Network Policy recipes - * Network Policies - # Pod Priority **Pod Priority** is a mechanism in Kubernetes which can make certain pods more important than others. Therefore, they are favoured when making scheduling decisions. For example, lower priority pods can be evicted from node to make room for higher priority pods. This can be useful for cluster add-ons like ingress controllers, log shippers, etc. ## References * Pod Priority and Preemption - * Guaranteed Scheduling For Critical Add-On Pods - # Pod Security Policies Mostly notes from watching TGI Kubernetes 078: Pod Security Policies * Pod Security Policies (PSP) are mechanism to prevent security aspects and capabilities of containers. * It is implemented by as an optional, but recommended, admission controller * Depending on the platform and Kubernetes distribution, PSP might need to enabled first * PSPs are cluster-wide resources * Kubernetes objects need to be granted privileges to **use** PSPs by RBAC * For example, create ClusterRole and then RoleBinding to bind it to specific namespaces * In my opinion, PSPs should be used in every production cluster * It is recommended to create multiple PSPs for different types of workloads. For example: * More privileged PSP for kube-system pods. Check TGIK 078 or Kube docs example for this. * Restricted for "normal" workloads * Something in between, if needed * You can use `kubectl describe/get $OBJECT` to check which PSP the pod is using from its annotations * In order to create a default and restrictive policy which always resolves, use ClusterRoleBinding and bind to rule to group `system:authenticated` * Security context configures security aspects for the pods * You can provide exceptions to PSPs on * workload level * binding the role to specific ServiceAccount * namespace level * Helper and auditing tool: * Advices the set of PSP against your workload ## References * Kubernetes docs - * TGI Kubernetes 078: Pod Security Policies - * Configure a Security Context for a Pod or Container - # Secrets * `Secret` API object * stores secrets as base64 encoded in etcd * encryption comes with extra-configuration or integration * plain-text values can be used with "stringData" attribute * Encryption at rest (with `EncryptionConfiguration` API object) * Defines a key which encrypts and decrypts Secrets in etcd * however, the encryption key is in plain-text in apiserver * this is a concern if apiserver and etcd are co-hosted on a same node * keys must rotated * KMS (Envelope encryption) * Data is encrypted with Data encyption key (DEK) * DEK is encrypted with Key Encryption Key (KEK) * Data and and enrypted DEK are stored side-by-side * When data is decrypted, a call to KMS provider is done to decrypt the DEK * so the secret is never transmitted to KMS provider * Most usable with cloud providers with KMS service * External provider * Hashicorp Vault * Integration on * platform level * Vault Injection * creates init container for fetching secrets from Vault * uses mutating webhook to inject Vault init container * or run as sidecar, and fetch if secrets secrets are modified * application level * Sealed secrets * mostly solves the problem of keeping secrets safe *outside of Kubernetes* * plain-text copies of sealed secret controller secrets and secrets itself are stored to `etcd`. So encrypting `etcd` need to be solved somehow * csi-secret-driver * mount secrets/keys/certs to pod using a CSI volume * still in experimental state ## Decode secrets Decodes all values of certain secret. ```bash kubectl get secret -o go-template='{{range $k,$v := .data}}{{printf "%s: " $k}}{{if not $v}}{{$v}}{{else}}{{$v | base64decode}}{{end}}{{"\n"}}{{end}}' ``` ## Resources * TGI Kubernetes 113: Kubernetes Secrets Take 3 - * * * * How To Decode / Decrypt Kubernetes Secret - # StatefulSet ## Resizing EBS disks of StatefulSet without major downtime Kubernetes has supported resizing of volumes without recreating PVCs and PVs from version 1.11 with certain disk types. EBS is one of them. However, if VolumeClaimTemplate is used certain tricks needs to be done. * Find PVCs belonging to StatefulSet, for example `kubectl get pvc --all-namespaces` * Edit PVC API object on-the-fly, for example: `kubectl edit pvc/$PVC_NAME -n $NAMESPACE` * Change value of field `spec.resources.requests.storage` and save changes * Ensure that resize is finished by inspecting PVC with `kubectl describe pvc/$PVC_NAME -n $NAMESPACE`. Events field should state that resize is finished and changes take place after pod restart * Restart pod in question * Repeat the steps for all PVCs of the StatefulSet * Delete StatefulSet without deleting its pods: `kubectl delete sts --cascade=false $STS_NAME -n $NAMESPACE` * Change disk size in VolumeClaimTemplate and apply changes * If needed, trigger StatefulSet rollout with `kubectl rollout restart sts $STS_NAME -n rabbitmq`. This will restart the pods one-by-one In addition, Kubernetes has beta support for resizing an in-use PersistentVolumeClaim from version 1.15: References: * * # additional-services # Debugging ArgoCD RBAC The command `argocd admin settings rbac` can be used to test RBAC access. For example: \` ```shell argocd admin settings rbac can update projects 'default' -n argocd Yes ``` Currently logged in account privileges can be checked with: ```shell argocd account can-i ``` # open-policy-agent ## Open Policy Agent (OPA) > Policy-based control for cloud native environments * Can be used generally with structured data, not only with Kubernetes * Uses Rego DSL for its policies * Rego is for data queries, it's not general purpose language * Can be used to validate manifests, for example `"Container must provide app label for pod selectors"` * Output JSON can passed to external systems, like Gatekeeper * Or just respond with `AdmissionReview` Kubernetes object, so OPA can be plugged to Admission Controller ### Gatekeeper * Provides more "Kubernetes-native" abstraction and functionalities over OPA * Policies are stored as CRDs * Constraint Templates * Enables you to provide a policy and pass variables during the valuation * Gatekeeper provides a library for couple of policy types like PSPs * * Validating PSPs with web hooks might provide better UX, because apiserver rejects the object instantly, not only then when pod starts * Above-mentioned functionality does not use PSPs in the backend, but OPA implements PSP-like behavior ## Conftest > Write tests against structured configuration data using the Open Policy Agent Rego query language * Policies can be bundled and pushed to registry (like Harbor) * Can be run in CI for getting feedback before applying manifests to cluster * For example can be used to check Kubernetes API deprecations () ### Use Conftest and OPA for Dockerfile checks * Conftest and OPA can be used to write for example security checks for Dockerfiles * For example, check in CI that `latest` that tag is not used * See for more information ### Use conftest to validate Terraform code * Conftest and OPA can be used to validate Terraform and create assertions * See ### Resources * TGI Kubernetes 119: Gatekeeper and OPA - * * * * * * # misc # FFmpeg ## Reduce noise from spoken audio using trained neural networks Download trained model from Apply audio filter, for example with command: ``` ffmpeg -i orig.ext -af arnndn=m=cb.rnnn -c:v cleaned_audio.ext ``` Parameter `-c:v` just copies the videostream without processing. I used this to clean recorded audio from screen recording and it worked really fine. ## Resources: * # PDFs ## Compress with Ghostscript `ghostscript` command can be used to compress PDF files. It has multiple predefined settings: > `-dPDFSETTINGS=configuration` > > Presets the "distiller parameters" to one of four predefined settings: > > `/screen` selects low-resolution output similar to the Acrobat Distiller (up to version X) "Screen Optimized" setting. > > `/ebook` selects medium-resolution output similar to the Acrobat Distiller (up to version X) "eBook" setting. > > `/printer` selects output similar to the Acrobat Distiller "Print Optimized" (up to version X) setting. > > `/prepress` selects output similar to Acrobat Distiller "Prepress Optimized" (up to version X) setting. > > `/default` selects output intended to be useful across a wide variety of uses, possibly at the expense of a larger output file. More information about these profiles can be found from ## References * How can I reduce the file size of a scanned PDF file? - # programming # Learning resources ## Golang * Gophercises - > Gophercises is a FREE course that will help you become more familiar with Go while developing your skills as a programmer. In the course we will build roughly 20 different mini-applications, packages, and tools that are each designed to teach you something different. ## Full-stack web applications * Full Stack Open - > The world is in dire need of software developers. We want to help breed the next generation of coders — one of our offered methods is to enroll in the Full Stack MOOC course. # concepts # Serialization * In serialization, objects and data structures residing in memory are transformed to stream of bits, so that they can be written to disk (for example) or sent over network * Basically, serialize function transforms data structure to string, which can be then sent to the recipient * Recipient can then deserialize the string back to data structure * Examples of serialization formats * JSON * YAML * XML ## Libraries ### Python * pickle * saves data as byte stream ## References * [Serialization - A Crash Course (Youtube)](https://www.youtube.com/watch?v=uS37TujnLRw) * [A Gentle Introduction to Serialization for Python (https://machinelearningmastery.com/)](https://machinelearningmastery.com/a-gentle-introduction-to-serialization-for-python/) # rabbitmq # Clustering and HA Resources: * * # Shovel plugin Shovel plugin allows to move messages from one broker to another. Source and destination can be queue or exchange. * Shoveling all messages from one exchange to another * Source routing key should be `#` (wildcard) * Destination routing key is left empty # shells # Bash ## Resources * Ten Things I Wish I’d Known About bash - # terraform # Moving resources between remote states Sometimes it's useful to be able to move resources between remote states without a need to recreate infrastructure. One use case is moving resources between directories. `terraform state mv` command does not directly support remote states, but `-state` and `-state-out` parameters can be used against local states. This means remote states need to fetched first. This can be done with for example `aws s3 cp` or `terraform state pull > local.tfstate`. After both states are local, `state mv` command can be used in the following manner: `terraform state mv -state=src.tfstate -state-out=dst.tfstate module.foo module.foo` After this modified local state files need to pushed to remote storage, for example with: `terraform state push /path/to/local/state` ## References: * # tools # FFmpeg ## Concatenate video files Videos should probably be encoded with similar codec etc. ``` ffmpeg -f concat -safe 0 -i <(for f in ./*.mp4; do echo "file '$PWD/$f'"; done) -c copy output.mp4 ``` # yt-dlp ## Download video from Instagram reel Export to file which is viewable with Apple QuickTime. Tested with yt-dlp version '2025.05.22' ``` yt-dlp -S "vcodec:h264,res,acodec:m4a" -o $FILENAME.mp4 '$REEL_URL' ``` # vim # Fzf (plugin) ## Search results selection and opening Open file under cursor to a new horizontal split: `Ctrl+X`. Respectively, open to a new vertical split: `Ctrl+V`. Multiple files can be selected with `TAB`. `Shift+TAB` will select to another direction. ## References * Open FZF Result In A Split In Vim - # Registers ## Basics Copy (yank) selected text to register `r` with `"ry`. And then, paste from this same register with `"rp`. Pasting can be also done in insert or command with `Ctrl-r r` ## References * Vim registers: The basics and beyond - Vim registers: The basics and beyond - # Spell Check See suggested corrections for a word: `z=` Add word to the dictionary: `zg` ## Spellcheck Git Commit messages This is an example how spell checking can be enabled for certain file: ``` autocmd FileType gitcommit setlocal spell ``` ## Resources: Enable Spell Checking in Vim for Markdown and Git Commit Messages - Vim Spell-Checking - # virtualization # File formats ## Running .ova packaged virtual appliances with KVM/QEMU OVA is Open Virtual Appliance package, which is a tar archive containing .ovf descriptor file and typically one or more disk images, for example in .vmdk format. OVF file describes specifications for a virtual machine in XML format. .ova file can be extracted with the following command: ``` tar xf $name_of_ova_file.ova ``` Then, .vmdk disk images can be converted to KVM/QEMU supported QCOW2 format with the following command: ``` qemu-img convert -O qcow2 $path_to_vmdk_file.vmdk $name_of_qcow2_file.qcow2 ``` New VM can be then created from QCOW2 disk image using for example libvirt. If there are multiple disk images in .ova, VM might need multiple disks to be attached. ## Resources * Open Virtualization Format - * Virtual appliance - # linux # arch # Arch Linux installation My installation notes on installing Arch Linux to Thinkpad T480s. Heavily inspired and adapted from Mischa van den Burg's excellent [install Arch Linux THE RIGHT way](https://www.youtube.com/playlist?list=PL_JVnPgp2IRcFnHqZdmQwWdv8n49vGHqp) video series. ## Preparations Boot Arch Linux ISO from USB or some other way * Load Finnish keymap * `loadkeys fi` * Connect WiFi interface * `iwctl` * `station wlan0 scan` * `station wlan0 connect $SSID` ## Disk ### Partitioning * Delete old partitions with fdisk `fdisk /dev/nvme0n1` * write changes to disk * Delete old partition table * `sgdisk --zap-all /dev/sdX` * `wipefs --all /dev/sdX` * Create new partitions with fdisk * 1Gb boot partition with type EFI * Rest of the disk space with Linux LVM type ### Encryption * Create LUKS encryption container to bigger partition * `cryptsetup luksFormat /dev/nvme0n1p2` ### LVM * Create physical volume on top of the opened LUKS container * `pvcreate /dev/mapper/cryptlvm` * Create volume group * `vgcreate thinkpad /dev/mapper/cryptlvm` * Create logical volumes * `lvcreate -L 4G thinkpad -n swap` * `lvcreate -L 32G thinkpad -n root` * `lvcreate -l 100%FREE thinkpad -n home` * Create filesystems * `mkfs.ext4 /dev/thinkpad/root` * `mkfs.ext4 /dev/thinkpad/home` * `mkswap /dev/thinkpad/swap` ### Mount rest of the partitions * `mount /dev/thinkpad/root /mnt` ### Prepare boot partition * `mkfs.fat -F32 /dev/nvme0n1p1` * `mount --mkdir /dev/nvme0n1p1 /mnt/boot` ### Enable swap `swapon /dev/thinkpad/swap` ## Install base system * Install basic packages * `pacstrap -K /mnt base linux linux-firmware` * Generate fstab * `genfstab -U /mnt >> /mnt/etc/fstab` * Chroot to the installed Linux filesystem * `arch-chroot /mnt` * Install CPU microcode updates * `pacman -Syu intel-ucode` * Set timezone * `ln -sf /usr/share/zoneinfo/Europe/Helsinki /etc/localtime` * Sync clock * `hwclock --systohc` * Configure locales * `locale-gen en_US.UTF-8` * `echo "LANG=en_US.UTF-8" > /etc/locale.conf` ## Setup networking with systemd-networkd * Enable systemd-networkd and systemd-resolved * `systemctl enable systemd-networkd` * `systemctl enable systemd-resolved` * Use systemd-resolved stub resolver * `sudo ln -sf /run/systemd/resolve/stub-resolv.conf /etc/resolv.conf` * Configure wireless interface ``` cat << 'EOF' > /etc/systemd/network/25-wireless.network [Match] Name=wlan0 [Network] DHCP=yes IgnoreCarrierLoss=3s EOF ``` * Install `ìwd` * `pacman -Syu iwd` * Enable `iwd` * `systemctl enable iwd` ## Configure initrmfs hooks Following is needed to decrypt root filesystem In `/etc/mkinitcpio.conf`, set: ``` HOOKS=(base systemd autodetect microcode modconf kms keyboard block sd-encrypt lvm2 filesystems fsck) ``` * Install `lvm2` package * \`\`pacman -Syu lvm2 * Regenerate initrmfs * `mkinitcpio -P` ## Install bootloader (systemd-boot) * `bootctl install` * Configure bootloader entry with correct option for the LUKS partition ``` cat << 'EOF' > /boot/loader/entries/arch.conf title Arch Linux linux /vmlinuz-linux initrd /initramfs-linux.img options rd.luks.name=61c24ef5-ba64-493a-8fbf-9f6050a9026a=thinkpad root=/dev/thinkpad/root rw EOF ``` * Build initrmfs once more, just in case * `mkinitcpio -P` ## Mount home and create user * `mount /dev/thinkpad/home /home/` * Fstab needs to be configured. `genfstab` is part of `arch-install-scripts` package * `pacman -Syu arch-install-scripts` * Remove old entrie from `/etc/fstab` * Generate new fstab * genfstab / >> /etc/fstab * Install `sudo` * `pacman -Syu sudo` * Create user * `useradd -m tatu` * Set password * `passwd tatu` * Add user to wheel group * `usermod -aG wheel tatu` * With `visudo` remove comment from line * `# %wheel ALL=(ALL:ALL) ALL` ## Set hostname `echo "thinkpad" > /etc/hostname` ## Booting to the new system That's all, base system is now installed and it should be bootable.