Containers

June 6, 2020 · View on GitHub

Containers 101
Containers General
Docker
Jails
Kubernetes
- 101
- Security
RunC
- 101
Solaris Zones

Containers

101
- LXC - Wikipedia
- Process Containers - lwn.net
- cgroups - wikipedia
- Everything you need to know about Jails - bsdnow.tv
- Jails - FreeBSD handbook
- Solaris Containers - Wikipedia
- What is Kubernetes - kubernetes.io
- xkcd on containers
- Docker Internals: A Deep Dive Into Docker For Engineers Interested In The Gritty Details - Docker Saigon
- A Brief History of Containers: From the 1970s Till Now - Rani Osnat
- Capability-Based Computer Systems - Henry M. Levy
  - This book was published by Digital Press in 1984. It is still the most thorough survey and description of early capability-based and object-based hardware and software systems. The book is now out of print and the copyright belongs to the author, who makes the material available here for viewing or downloading, in Adobe Acrobat PDF format (free Acrobat reader available here).
Containers(General)
- 101
  - Linux LXC vs FreeBSD jail - Are there any notable differences between LXC (Linux containers) and FreeBSD's jails in terms of security, stability & performance? - unix.StackExchange
  - Architecting Containers Part 1: Why Understanding User Space vs. Kernel Space Matters - Scott McCarty
  - From 30 to 230 docker containers per host - stormbind.net
  - Linux namespaces - Wikipedia
  - namespaces - overview of Linux namespaces(manpage)
    - A namespace wraps a global system resource in an abstraction that makes it appear to the processes within the namespace that they have their own isolated instance of the global resource. Changes to the global resource are visible to other processes that are members of the namespace, but are invisible to other processes. One use of namespaces is to implement containers. This page provides pointers to information on the various namespace types, describes the associated /proc files, and summarizes the APIs for working with namespaces.
  - sandbox manpage
    - Runs the given cmd application within a tightly confined SELinux domain. The default sandbox domain only allows applications the ability to read and write stdin, stdout and any other file descriptors handed to it. It is not allowed to open any other files.
  - seccomp_filter - kernel.org
    - Seccomp filtering provides a means for a process to specify a filter for incoming system calls. The filter is expressed as a Berkeley Packet Filter (BPF) program, as with socket filters, except that the data operated on is related to the system call being made: system call number and the system call arguments. This allows for expressive filtering of system calls using a filter program language with a long history of being exposed to userland and a straightforward data set.
  - The Container Operator's Manual - Alice Goldfuss(Velocity NY2018)
    - Containers have been the future for five years now, featured on the stage of every major distributed systems conference in the world. But beyond the hype and the swag is a real technical solution, with real technical challenges, used for real problems at scale. And for the companies and engineers looking to adopt this solution, there’s little content on what awaits them. Containers can be a great infrastructure solution, but no one should drive them without a manual. Alice Goldfuss discusses some of the advantages and disadvantages of running containers in production at scale. You’ll learn why you should use containers, why you shouldn’t, and the trade-offs required at both the technical and human levels for implementing them. You’ll leave with a better understanding of how containers could fit into your own architecture and what you need to do to make that rollout a reality.
- Articles/Blogposts/Writeups
  - Efficient, reliable cluster management at scale with Tupperware - Facebook Engineering
- Building
  - Best practices for building containers - cloud.google
  - img
    - Standalone, daemon-less, unprivileged Dockerfile and OCI compatible container image builder.
  - linuxkit
    - A toolkit for building secure, portable and lean operating systems for containers
  - Linux Containers: Future or Fantasy? - Aaron Grattafiori(Defcon23)
    - Containers, a pinnacle of fast and secure deployment or a panacea of false security? In recent years Linux containers have developed from an insecure and loose collection of Linux kernel namespaces to a production-ready OS virtualization stack. In this talk, the audience will first learn the basics of how containers function, understanding namespaces, capabilities and cgroups in order to see how Linux containers and the supporting kernel features can offer an effective application and system sandboxing solution yet to be widely deployed or adopted. Understanding LXC or Docker use, weaknesses and security for PaaS and application sandboxing is only the beginning. Leveraging container technologies is rapidly becoming popular within the modern PaaS and devops world but little has been publicly discussed in terms of actual security risks or guarantees. Understanding prior container vulnerabilities or escapes, and current risks or pitfalls in major public platforms will be explored in this talk. I'll cover methods to harden containers against future attacks and common mistakes to avoid when using systems such as LXC and Docker. This will also include an analysis and discussion of techniques such as Linux kernel hardening, reduced capabilities, Mandatory Access Controls (MAC), the User kernel namespace and seccomp-bpf (syscall filtering); all of which help actually contain containers. The talk will end on some methods for creating minimal, highly-secure containers and end on where containers are going and why they might show up where you least expect them.
- Capabilities
- Forensics
  - Container Forensics: What to Do When Your Cluster is a Cluster - Maya Kaczorowski & Ann Wallace(CloudNativeConEU19)
    - When responding to an incident in your containers, you don’t necessarily have the same tools at your disposal that you do with VMs - and so your incident investigation process and forensics are different. In a best case scenario, you have access to application logs, orchestrator logs, node snapshots, and more. In this talk, we’ll go over where to get information about what’s happening in your cluster, including logs and open source tools you can install, and how to tie this information together to get a better idea of what’s happening in your infrastructure. Armed with this info, we’ll review the common mitigation options such as to alert, isolate, pause, restart, or kill a container. For common types of container attacks, we'll discuss what options are best and why. Lastly, we’ll talk about restoring services after an incident, and the best steps to take to prevent the next one.
- General
  - Getting Towards Real Sandbox Containers - Jesse Frazelle(May2016)
  - Best Practices for Operating Containers - cloud.google
    - This article describes a set of best practices for making containers easier to operate. These practices cover a wide range of topics, from security to monitoring and logging.
  - Linux containers in 500 lines of code - lizzie.io
- Logging & Monitoring
  - Auditing containers with osquery - Corentin Badot-Bertrand
  - SELinux, Seccomp, Sysdig Falco, and you: A technical discussion - Mark Stemm
- Namespaces
- Privilegs
  - Privilege Escalation via lxd - Josiah Beverton
- Runtimes
  - Podman and Buildah for Docker users - William Henry
    - I was asked recently on Twitter to better explain Podman and Buildah for someone familiar with Docker. Though there are many blogs and tutorials out there, which I will list later, we in the community have not centralized an explanation of how Docker users move from Docker to Podman and Buildah. Also what role does Buildah play? Is Podman deficient in some way that we need both Podman and Buildah to replace Docker? This article answers those questions and shows how to migrate to Podman.
- Security
  - 101
  - Articles/Blogposts/Writeups
    - Container Security – Nobody Knows What It Means But It’s Provocative - Kelley Shortridge(2020)
    - How to implement an open source container security stack (part 1)(2018) - Mateo Burillo
    - Understanding and Hardening Linux Containers - nccgroup
      - Linux containers offer native OS virtualisation, segmented by kernel namespaces, limited through process cgroups and restricted through reduced root capabilities, Mandatory Access Control and user namespaces. This paper discusses these container features, as well as exploring various security mechanisms. Also included is an examination of attack surfaces, threats, and related hardening features in order to properly evaluate container security. Finally, this paper contrasts different container defaults and enumerates strong security recommendations to counter deployment weaknesses-- helping support and explain methods for building high-security Linux containers. Are Linux containers the future or merely a fad or fantasy? This paper attempts to answer that question.
    - Containers and Cloud Security - James Bottomley(2018)
      - The idea behind this blog post is to take a new look at how cloud security is measured and what its impact is on the various actors in the cloud ecosystem.
    - Exploring container security: An overview - Maya Kaczorowski(GCP Focused)
    - Runtimes And the Curse of the Privileged Container - brauner
      - Writeup of CVE-2019-5736
    - On container image security - Kamil Zabielski(2020)
      - The biggest goal of this article, is to be a comprehensive guide on building and delivering secure and safe container images. Having that said, we try to focus on build-time. We will not cover registry, orchestrator and runtime protection in this write-up. All of these are so broad, that they deserve a completely different article.
  - Escaping
    - An Exercise in Practical Container Escapology - Nick Freeman
  - Papers
    - Understanding and Hardening Linux Containers - Aaron Grattafiori(2016)
      - Operating System virtualization is an attractive feature for efficiency, speed and modern application deployment, amid questionable security. Recent advancements of the Linux kernel have coalesced for simple yet powerful OS virtualization via Linux Containers, as implemented by LXC, Docker, and CoreOS Rkt among others. Recent container focused start-ups such as Docker have helped push containers into the limelight. Linux containers offer native OS virtualization, segmented by kernel namespaces, limited through process cgroups and restricted through reduced root capabilities, Mandatory Access Control and user namespaces. This paper discusses these container features, as well as exploring various security mechanisms. Also included is an examination of attack surfaces, threats, and related hardening features in order to properly evaluate container security. Finally, this paper contrasts different container defaults and enumerates strong security recommendations to counter deployment weaknesses– helping support and explain methods for building high-security Linux containers. Are Linux containers the future or merely a fad or fantasy? This paper attempts to answer that question
  - Presentations/Talks/Videos
    - All Your Containers Are Belong to Us - James Condon(BSidesSF 2019)
      - The rising adoption of container orchestration tools, such as Kubernetes, has enabled developers to scale cloud applications quickly and efficiently. However with this adoption comes with a new set of security challenges, such as securing the APIs used to manage these ecosystems. We recently conducted a research study that uncovered more than 20,000 publicly accessible management nodes open to the Internet. In this talk we will discuss the implications of the findings and provide recommendations for running orchestration systems securely in the public cloud. The following platforms are exposed and part of the research: Kubernetes, Mesos Marathon, RedHat OpenShift, Docker Swarm, and Portainer (Docker Management). Not only are these management UIs available on the web but we also discovered that their APIs are also available. Some are wide open. We will uncover how we did this research, who is the most popular cloud provider hosting the containers, which regions are most popular, and show demonstrations of exploitation and discover.
    - Docker, Linux Containers (LXC), and security(2014) - Jerome Petazzoni
      - Virtual machines are generally considered secure. At least, secure enough to power highly multi-tenant, large-scale public clouds, where a single physical machine can host a large number of virtual instances belonging to different customers. Containers have many advantages over virtual machines: they boot faster, have less performance overhead, and use less resources. However, those advantages also stem from the fact that containers share the kernel of their host, instead of abstracting a new independent environment. This sharing has significant security implications, as kernel exploits can now lead to host-wide escalations. We will show techniques to harden Linux Containers; including kernel capabilities, mandatory access control, hardened kernels, user namespaces, and more, and discuss the remaining attack surface.
    - The two metrics that matter for host security - Diogo Monica
      - As companies move their infrastructures towards ephemeral microservices, there is an opportunity to rethink some of the security metrics typically used to track infrastructure risk, such as the number of currently unpatched vulnerabilities sorted by their criticality. In the same way that the adoption of Continuous Integration and Continuous Delivery (CI/CD) allows faster development and patching of application vulnerabilities, it is time for organizations to realize that they should follow the same pattern around upgrading the Operating System their applications are running on. Instead of having a JIRA queue—with an ever-increasing number of tickets tracking the CVEs in the Linux Kernel—we should instead start tracking reverse uptime and golden image freshness.
- Storage
  - REX-Ray
    - REX-Ray provides a vendor agnostic storage orchestration engine. The primary design goal is to provide persistent storage for Docker, Kubernetes, and Mesos. The long-term goal of the REX-Ray project is to enable collaboration between organizations focused on creating enterprise-grade storage plugins for the Container Storage Interface (CSI).
- Tools
  - nsjail
    - A light-weight process isolation tool, making use of Linux namespaces and seccomp-bpf syscall filters (with help of the kafel bpf language)
  - bubblewrap
    - Unprivileged sandboxing tool - "The maintainers of this tool believe that it does not, even when used in combination with typical software installed on that distribution, allow privilege escalation. It may increase the ability of a logged in user to perform denial of service attacks, however."
Docker
- 101
  - Get Started, Part 1: Orientation and setup - docs.docker
  - Play with Docker Classroom - Docker
    - The Play with Docker classroom brings you labs and tutorials that help you get hands-on experience using Docker.
  - Life in Containers: The Big Picture - Pankaj Mouriya
    - In today's contemporary world, containers are one of the most trending and hottest topics in IT, wherever you go, you will find people talking about some shiny and new technologies and most of the time they're either talking about DevOps, Docker, Kubernetes or are deploying it. It becomes very difficult to know where to start and how to take your career to the next level in these shiny technologies. So, in today's session, we will be talking about the Big Picture of Docker. You will learn the basic fundamentals and how it works. After this session, you'll be able to get started with Docker.
    - Gitbook
  - Docker Cheat Sheet - wsargent
- Bocker
  - Docker implemented in around 100 lines of bash.
- Compose
- Containers & Images
  - 101
    - Docker Image Specification v1.0.0
    - Docker image in depth - Bikram Kundu
  - Analysis
    - Static Analysis of Docker image vulnerabilities with Clair - Petr Kohut
    - Dive
      - A tool for exploring a docker image, layer contents, and discovering ways to shrink your Docker image size.
  - Building
    - Articles/Blogposts/Writeups
      - Debugging Docker builds - Hongli Lai
    - Tools
      - img
        
        Standalone, daemon-less, unprivileged Dockerfile and OCI compatible container image builder.
      - makisu
        
        Fast and flexible Docker image building tool, works in unprivileged containerized environments like Mesos and Kubernetes.
  - Registry
    - Setting up a private Docker registry - Nicolas Frankel
    - How to secure a private Docker registry - Nicolas Frankel
    - Portus
      - Portus is an authorization server and a user interface for the next generation of the Docker registry. Portus targets version 2 of the Docker Registry API. The minimum required version of Registry is 2.1, which is the first version supporting soft deletes of blobs.
  - Scanning
    - Articles/Blogposts/Writeups
    - Tools
      - clair
        
        Clair is an open source project for the static analysis of vulnerabilities in application containers (currently including appc and docker).
      - dagda
        
        a tool to perform static analysis of known vulnerabilities, trojans, viruses, malware & other malicious threats in docker images/containers and to monitor the docker daemon and running docker containers for detecting anomalous activities
      - dockle
        
        Container Image Linter for Security, Helping build the Best-Practice Docker Image, Easy to start
      - container-scan
        
        A GitHub action to help you scan your docker image for vulnerabilities. This action can be used to help you add some additional checks to help you secure your Docker Images in your CI. This would help you attain some confidence in your docker image before pushing them to your container registry or a deployment. It internally uses Trivy and Dockle for running certain kinds of scans on these images.
      - trivy
        
        A Simple and Comprehensive Vulnerability Scanner for Containers, Suitable for CI.
- Deployment
  - Hawkeye
    - The Hawkeye scanner-cli is a project security, vulnerability and general risk highlighting tool. It is meant to be integrated into your pre-commit hooks and your pipelines.
- Dockerfiles
- Layers
  - Optimising Docker Layers for Better Caching with Nix - Graham Christensen
- Logging & Monitoring)
- Namespaces
- Networking
  - Networking overview - docs.docker
- Privileges
  - Is it possible to escalate privileges and escaping from a Docker container? - StackOverflow
  - Abusing Privileged and Unprivileged Linux Containers - nccgroup
- Security
  - General
    - Docker security - docs.docker
  - Attacking
    - The Dangers of Docker.sock
    - On Docker security: 'docker' group considered harmful - Andreas Jung
    - Docker Container Breakout Proof-of-Concept Exploit - James Turnbull(2014)
    - Vulnerability Exploitation In Docker Container Environments - Anthony Bettini
      - According to Forrester, 53% of IT respondents say their biggest concern about containers is security. Containerization is not only prevalent in browsers (Google Chrome), desktop applications (Adobe Reader X), and mobile operating systems (Apple iOS), but is also invading the data center via Docker. Docker and other LXC-based containerization solutions provide isolation via Linux control groups (cgroups). However, containers can still be exploited and even with kernel-level isolation, critical data can be stolen. In this presentation, the FlawCheck team will exploit real-world Docker implementations and show what can be done to mitigate the risk.
      - Vulnerability Exploitation in Docker Container Environments - Anthony Bettini(BH EU 2015)
    - Attacking & Auditing Docker Containers Using Open Source tools - Madhu Akula
      - Defcon 26 Workshop
    - Whaler
      - Program to reverse Docker images into Dockerfiles
    - Docker for Hackers? A pen tester’s guide - Robert Bone
    - Harpoon
      - A collection post-exploitation scripts for determining if that shell you just got is in a container, what kind, and ways to escape.
    - You can't contain me! :: Analyzing and Exploiting an Elevation of Privilege Vulnerability in Docker for Windows - srcincite.io
    - Breaking out of Docker via runC – Explaining CVE-2019-5736 - Yuval Avrahami
    - Container Runtime Security Bypasses on Falco - antitree
    - Abusing Docker API | Socket - Carnal0wnage
    - Understanding Docker container escapes - TrailofBits(2019)
    - How I Hacked Play-with-Docker and Remotely Ran Code on the Host - Nimrod Stoler(2019)
    - Are Docker containers really secure?(2014) - Daniel Walsh
  - Securing
    - 101
      - Docker security - docs.docker
      - CIS Benchmarks: Docker
    - Articles/Blogposts/Writeups
    - Pipeline
      - Building a Docker Security Program - Charlie Belmer(2018)
      - Docker Static Analysis With Clair - Charlie Belmer(2018)
      - Introduction to Docker Content Trust - Paul Novarese
        
        Docker Content Trust provides strong cryptographic guarantees over what code and what versions of software are being run in your infrastructure. Docker Content Trust integrates The Update Framework (TUF) into Docker using Notary , an open source tool that provides trust over any content.
      - Docker Reference Architecture: Development Pipeline Best Practices Using Docker EE
      - dagda
        
        a tool to perform static analysis of known vulnerabilities, trojans, viruses, malware & other malicious threats in docker images/containers and to monitor the docker daemon and running docker containers for detecting anomalous activities
      - dockle
        
        Container Image Linter for Security, Helping build the Best-Practice Docker Image, Easy to start
      - container-scan
        
        A GitHub action to help you scan your docker image for vulnerabilities. This action can be used to help you add some additional checks to help you secure your Docker Images in your CI. This would help you attain some confidence in your docker image before pushing them to your container registry or a deployment. It internally uses Trivy and Dockle for running certain kinds of scans on these images.
      - trivy
        
        A Simple and Comprehensive Vulnerability Scanner for Containers, Suitable for CI.
    - Tools
      - Docker Bench for Security
        
        The Docker Bench for Security is a script that checks for dozens of common best-practices around deploying Docker containers in production. The tests are all automated, and are inspired by the CIS Docker Benchmark v1.2.0.
      - bane
        
        AppArmor profile generator for docker containers. Basically a better AppArmor profile, than creating one by hand, because who would ever do that.
  - Talks & Presentations
    - An Attacker Looks at Docker: Approaching Multi-Container Applications - Wesley McGrew
    - Docker: Security Myths, Security Legends - Rory McCune
    - Securing The Docker Containers At CI/CD Pipeline Level - Alina Radu(BSidesBCN 2019)
    - How to Lose a Container in 10 Minutes - Sarah Young(BSidesSF 2019)
      - Moving to the cloud and deploying containers? In this talk I will discuss both the mindset shift and tech challenges, with some common mistakes made in real-life deployments with some real life (albeit redacted) examples. We'll also look at what happens to a container that's been left open to the Internet for the duration of the talk.
    - Well, That Escalated Quickly! How Abusing Docker API Led to Remote Code Execution, Same Origin Bypass and Persistence in The Hypervisor via Shadow Containers - Michael Cherny, Sagi Dulce(BH US 17)
    - David Mortman - Docker, Docker Give Me The News: I Got A Bad Case Of Securing You - David Mortman(Defcon23)
      - Docker is all the rage these days. Everyone is talking about it and investing in it, from startups to enterprises and everything in between. But is it secure? What are the costs and benefits of using it? Is this just a huge risk or a huge opportunity? There's a while lot of ranting and raving going on, but not nearly enough rational discourse. I'll cover the risks and rewards of using Docker and similar technologies such as AppC as well as discuss the larger implications of using orchestration systems like Mesos or Kubernetes. This talk will cover the deep technical issues to be concerned about as well as the pragmatic realities of the real world.
- Storage
  - Why Containers Miss a Major Mark: Solving Persistent Data in Docker - Chris Brandon
- Tools
  - docker-layer2-icc
    - Demonstrating that disabling ICC in docker does not block raw packets between containers.
  - docker-bench-security
    - The Docker Bench for Security is a script that checks for dozens of common best-practices around deploying Docker containers in production.
  - Watchtower
    - With watchtower you can update the running version of your containerized app simply by pushing a new image to the Docker Hub or your own image registry. Watchtower will pull down your new image, gracefully shut down your existing container and restart it with the same options that were used when it was deployed initially.
  - Vulnerable Docker VM
    - For practicing pen testing docker instances
  - Dive
    - A tool for exploring a docker image, layer contents, and discovering ways to shrink the size of your Docker/OCI image.
Jails
- 101
  - FreeBSD Handbook: Jails
- Tools
  - ezjail – Jail administration framework
LXC
- 101
  - Linux containers
- Articles/Blogposts/Writeups
  - LXC 1.0: Blog post series [0/10] - Stephane Graber
Kubernetes
- 101
  - An Introduction to Kubernetes(2018) - Justin Ellingwood(DO tutorials)
  - Kubernetes The Hard Way - Kelsey Hightower
    - Bootstrap Kubernetes the hard way on Google Cloud Platform. No scripts.
  - Kubernetes The (real) Hard Way on AWS - Hans-Jorg Wieland
- Articles/Blogposts/Writeups
- Secrets Management
  - Kamus
    - An open source, GitOps, zero-trust secrets encryption and decryption solution for Kubernetes applications. Kamus enable users to easily encrypt secrets than can be decrypted only by the application running on Kubernetes. The encryption is done using strong encryption providers (currently supported: Azure KeyVault, Google Cloud KMS and AES). To learn more about Kamus, check out the blog post and slides.
- Security
  - 101
    - CIS Kubernetes Security Benchmarks
    - Kubernetes Security Assessment by TrailofBits(2019)
  - Articles/Blogposts/Writeups
    - Kubernetes Security Best-Practices - Peter Benjamin
    - hardening-kubernetes from-scratch
      - A hands-on walkthrough for creating an extremely insecure Kubernetes cluster and then hardening it, step by step.
    - Kubernetes security best practices - Christian Melendez
    - Kubernetes Hardening - Moshe Roth
    - Attacking default installs of Helm on Kubernetes - ropnop
    - Kubernetes Security - Best Practice Guide - freach
    - Container Platform Security at Cruise - Karl Isenberg
  - Attacking
  - Carnal0wnage Posts
  - CVEs
  - Talks & Presentations
    - Hacking and Hardening Kubernetes Clusters by Example - Brad Geesaman(KubeCon 2017)
      - "an eye-opening journey examining real compromises and sensitive data leaks that can occur inside a Kubernetes cluster, highlighting the configurations that allowed them to succeed, applying practical applications of the latest built-in security features and policies to prevent those attacks, and providing actionable steps for future detection."
    - Attacking and Defending Kubernetes [SeaSec East] - Jay Beale
    - Perfect Storm Taking the Helm of Kubernetes - Ian Coldwater(Derbycon2018)
      - Containers don't always contain. For attackers, Kubernetes contains a number of interesting attack surfaces and opportunities for exploitation. For defenders and operators, it's complicated to set up and the defaults often aren't enough. This can create a perfect storm. This talk will walk you through attacking Kubernetes clusters, and give defenders tools and techniques to protect themselves from shipwrecks.
    - A Hacker's Guide to Kubernetes and the Cloud - Rory McCune(Cloud Native ConEU18)
      - As Kubernetes increases in adoption it is inevitable that more clusters will come under attack by people wanting to compromise specific applications or just people looking to get access to resources for things like crypto-coin mining. The goal of this talk is to take an attackers perspective on typical cloud-based Kubernetes deployments, examine how attackers will find and compromise clusters and the applications running on them and suggest practical ways to improve the security of your cluster. This talk will draw on the presenters long experience of offensive security to provide an attacker's eye view of the challenges of running production Kubernetes clusers in cloud-facing environments.
    - Shipping in Pirate-Infested Waters: Practical Attack and Defense in Kubernetes - Greg Castle, CJ Cullen
      - Kubernetes has a growing array of security controls available, but knowing where they all fit in, what the highest priorities are, and how it all helps against real attacks is still far from obvious. In this talk we’ll take a vulnerable application, exploit it, install tools, escalate privileges, propagate between containers and gain control of the cluster. At each stage of the attack we’ll demonstrate how proactive steps could have prevented these actions (or at least made them more difficult), from the container build process to writing RBAC/PodSecurity/AppArmor/Network policies, and more. Since configuration of each defence could be the subject of it’s own deep-dive talk, we’ll mainly focus on the big picture of “what” technologies you’d use to configure your cluster securely and “why”.
    - DIY Pen-Testing for Your Kubernetes Cluster - Liz Rice
      - See how to use kube-hunter to run penetration tests on your Kubernetes clusters, and reveal misconfigurations that might leave you open to attack! Kube-hunter is an open source tool that simulates what a hacker might do when trying to attack a deployment. We’ll discuss the motivations behind the project, and some interesting aspects of how it is implemented. There will be plenty of demos, including: - Testing for the basics, like an unsecured Kubelet API - Simulating an attack from within a compromised container - Re-using credentials from a compromised container You'll need a basic understanding of Kubernetes components, and with using curl to issue API requests. You’ll leave this talk ready to test your own cluster, and with new insights into the possible routes that an attacker might attempt. Perhaps you’ll even be inspired to submit a new Hunter to the project!
    - Ship of Fools: Shoring up Kubernetes Security - Ian Coldwater(devopsdays Minneapolis 2018)
      - This talk will give you practical advice about securing your Kubernetes clusters, from an attacker’s perspective. We’ll walk through the attack process from discovery to post-exploitation, and you’ll walk away with tools and techniques that can be used for prevention along the way. Learn how to keep your infrastructure safer by making a hacker’s job harder.
    - Crafty Requests: Deep Dive Into Kubernetes CVE-2018-1002105 - Ian Coldwater(CloudNativeConEU19)
      - You may have heard about CVE-2018-1002105, one of the most severe Kubernetes security vulnerabilities of all time. But how does this flaw work? How can it be exploited, and what does it all mean? This deep dive will walk the audience through the Kubernetes back end, going over relevant concepts like aggregated API servers, the kubelet API, and permissions for namespace-constrained users. We will explain the details of how this flaw works, how a cluster’s moving parts can fit together to create a vulnerable context, and the risks involved in leaving this CVE unpatched in the wild. A live demonstration will show the audience exactly how easy it is to exploit this vulnerability. After explaining the attack pathways, the audience will leave with practical advice about mitigation and how to protect their clusters.
- Tools
  - Gravity
    - Gravity is an open source toolkit for creating "images" of Kubernetes clusters and the applications running inside the clusters. The resulting images are called cluster images and they are just .tar files.
  - Authentication
  - Operating
    - kops
      - Kubernetes Operations (kops) - Production Grade K8s Installation, Upgrades, and Management
  - Security
    - Kube-hunter
      - Kube-hunter hunts for security weaknesses in Kubernetes clusters. The tool was developed to increase awareness and visibility for security issues in Kubernetes environments.
    - kubeaudit
      - kubeaudit helps you audit your Kubernetes clusters against common security controls
    - kube-bench
      - kube-bench is a Go application that checks whether Kubernetes is deployed securely by running the checks documented in the CIS Kubernetes Benchmark.
  - Install and run a SPIRE Server and Agent locally on a Kubernetes cluster
    - This tutorial walks you through getting a SPIRE Server and SPIRE Agent running in a Kubernetes cluster, and configuring a workload container to access SPIRE.
  - Argo
    - Argoproj is a collection of tools for getting work done with Kubernetes.
Mesos
- [Mesos: A Platform for Fine-Grained Resource Sharing in the Data Center - Benjamin Hindman, Andy Konwinski, Matei Zaharia, Ali Ghodsi, Anthony D. Joseph, Randy Katz, Scott Shenker, Ion Stoica]](https://people.eecs.berkeley.edu/~alig/papers/mesos.pdf)
  - We present Mesos, a platform for sharing commodity clusters between multiple diverse cluster computing frameworks, such as Hadoop and MPI. Sharing improves cluster utilization and avoids per-framework data replication. Mesos shares resources in a fine-grained manner, allowing frameworks to achieve data locality by taking turns reading data stored on each machine. To support the sophisticated schedulers of today’s frameworks, Mesos introduces a distributed two-level scheduling mechanism called resource offers. Mesos decides how many resources to offer each framework, while frameworks decide which resources to accept and which computations to run on them. Our results show that Mesos can achieve near-optimal data locality when sharing the cluster among diverse frameworks, can scale to 50,000 (emulated) nodes, and is resilient to failures.
- Omega: flexible, scalable schedulers for large compute clusters - Malte Schwarzkop, Andy Konwinski, Michael Abd-El-Malek, John Wilkes
  - Increasing scale and the need for rapid response to changing requirements are hard to meet with current monolithic cluster scheduler architectures. This restricts the rate at which new features can be deployed, decreases efficiency and utilization, and will eventually limit cluster growth. We present a novel approach to address these needs using parallelism, shared state, and lock-free optimistic concurrency control. We compare this approach to existing cluster scheduler designs, evaluate how much interference between schedulers occurs and how much it matters in practice, present some techniques to alleviate it, and finally discuss a use case highlighting the advantages of our approach – all driven by real-life Google production workloads.
- 101
- Articles/Blogposts/Writeups
- Securing
- Talks/Presentations/Videos
  - Datacenter Management with Mesos - Benjamin Hindman - UC Berkeley AmpLab 2013
  - Omega: flexible, scalable schedulers for large compute clusters - Malte Schwarzkopf, Andy Konwinski, Michael Abd-El-Malek, John Wilkes
    - Increasing scale and the need for rapid response to changing requirements are hard to meet with current monolithic cluster scheduler architectures. This restricts the rate at which new features can be deployed, decreases efﬁciency and utilization, and will eventually limit cluster growth. We present a novel approach to address these needs using parallelism, shared state, and lock-free optimistic concurrency control.We compare this approach to existing cluster scheduler designs, evaluate how much interference between schedulers occurs and how much it matters in practice, present some techniques to alleviate it, and ﬁnally discuss a use case highlighting the advantages of our approach -- all driven by real-life Google production workloads.
- Tools
  - PaaSTA
    - PaaSTA is a highly-available, distributed system for building, deploying, and running services using containers and Apache Mesos!
Documentation for DC/OS
Marathon
- Marathon is a production-grade container orchestration platform for Mesosphere’s Datacenter Operating System (DC/OS) and Apache Mesos.
RunC
- 101
  - One of the original developers of cgroups on why it was created
- Articles/Blogposts/Writeups
- Securing
- Tools
Solaris Zones
- 101
  - Oracle Solaris Zones - docs.oracle
  - Solaris Zone of Solaris™ 10 Operating System function - Fujitsu.com
- Articles/Blogposts/Writeups
- Securing
- Tools
Other Stuff
- Container-Optimized OS - GCP
  - The OS built for containers, designed for Google Cloud Platform
- Bottlerocket - AWS
  - Bottlerocket is a Linux-based open-source operating system that is purpose-built by Amazon Web Services for running containers on virtual machines or bare metal hosts.
  - Github
- Firecracker-microvm
  - Firecracker is an open source virtualization technology that is purpose-built for creating and managing secure, multi-tenant container and function-based services.
- Photon OS
  - Photon OS™ is an open source Linux container host optimized for cloud-native applications, cloud platforms, and VMware infrastructure. Photon OS provides a secure run-time environment for efficiently running containers.
- The Twelve-Factor App
  - In the modern era, software is commonly delivered as a service: called web apps, or software-as-a-service. The twelve-factor app is a methodology for building software-as-a-service apps that: Use declarative formats for setup automation, to minimize time and cost for new developers joining the project; Have a clean contract with the underlying operating system, offering maximum portability between execution environments; Are suitable for deployment on modern cloud platforms, obviating the need for servers and systems administration; Minimize divergence between development and production, enabling continuous deployment for maximum agility; And can scale up without significant changes to tooling, architecture, or development practices. The twelve-factor methodology can be applied to apps written in any programming language, and which use any combination of backing services (database, queue, memory cache, etc).