Kubernetes Deployment Guide
October 1, 2025 · View on GitHub
This guide covers deploying the MicroVM Sandbox on Kubernetes clusters.
Prerequisites
- Kubernetes cluster 1.24+
- kubectl configured for your cluster
- Nodes with KVM support
- Helm 3.0+ (optional)
- Persistent storage provisioner
Quick Start
1. Deploy with Kubectl
# Apply all manifests
kubectl apply -f scripts/deployment/kubernetes/
# Check deployment status
kubectl get pods -n microvm-sandbox
kubectl get services -n microvm-sandbox
2. Deploy with Helm
# Add Helm repository
helm repo add microvm-sandbox https://charts.microvm-sandbox.org
helm repo update
# Install with default values
helm install microvm-sandbox microvm-sandbox/microvm-sandbox
# Or install with custom values
helm install microvm-sandbox microvm-sandbox/microvm-sandbox \
--values custom-values.yaml
Manual Deployment
1. Namespace and RBAC
# namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
name: microvm-sandbox
labels:
name: microvm-sandbox
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: microvm-sandbox
namespace: microvm-sandbox
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: microvm-sandbox
rules:
- apiGroups: [""]
resources: ["nodes", "pods", "services", "configmaps", "secrets"]
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
- apiGroups: ["apps"]
resources: ["deployments", "replicasets"]
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: microvm-sandbox
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: microvm-sandbox
subjects:
- kind: ServiceAccount
name: microvm-sandbox
namespace: microvm-sandbox
2. ConfigMaps and Secrets
# configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: microvm-sandbox-config
namespace: microvm-sandbox
data:
config.yaml: |
server:
host: "0.0.0.0"
port: 8000
workers: 4
cloud_hypervisor:
binary_path: "/usr/local/bin/cloud-hypervisor"
api_socket_dir: "/tmp/ch-sockets"
networking:
bridge_name: "chbr0"
subnet: "192.168.200.0/24"
resources:
max_vms: 50
max_memory_per_vm: 8192
max_vcpus_per_vm: 8
security:
enable_authentication: true
vm_isolation: true
monitoring:
prometheus_port: 9090
metrics_enabled: true
---
apiVersion: v1
kind: Secret
metadata:
name: microvm-sandbox-secrets
namespace: microvm-sandbox
type: Opaque
data:
jwt-secret: <base64-encoded-jwt-secret>
database-password: <base64-encoded-password>
encryption-key: <base64-encoded-encryption-key>
3. Persistent Volumes
# storage.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: microvm-data
namespace: microvm-sandbox
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 100Gi
storageClassName: fast-ssd
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: microvm-images
namespace: microvm-sandbox
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 50Gi
storageClassName: fast-ssd
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: microvm-snapshots
namespace: microvm-sandbox
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 200Gi
storageClassName: standard
4. Database Deployment
# database.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: postgres
namespace: microvm-sandbox
spec:
replicas: 1
selector:
matchLabels:
app: postgres
template:
metadata:
labels:
app: postgres
spec:
containers:
- name: postgres
image: postgres:15-alpine
env:
- name: POSTGRES_DB
value: "microvm_sandbox"
- name: POSTGRES_USER
value: "microvm"
- name: POSTGRES_PASSWORD
valueFrom:
secretKeyRef:
name: microvm-sandbox-secrets
key: database-password
ports:
- containerPort: 5432
volumeMounts:
- name: postgres-data
mountPath: /var/lib/postgresql/data
resources:
requests:
memory: "256Mi"
cpu: "250m"
limits:
memory: "1Gi"
cpu: "500m"
volumes:
- name: postgres-data
persistentVolumeClaim:
claimName: postgres-data
---
apiVersion: v1
kind: Service
metadata:
name: postgres
namespace: microvm-sandbox
spec:
selector:
app: postgres
ports:
- port: 5432
targetPort: 5432
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: postgres-data
namespace: microvm-sandbox
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 20Gi
storageClassName: fast-ssd
5. Redis Deployment
# redis.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: redis
namespace: microvm-sandbox
spec:
replicas: 1
selector:
matchLabels:
app: redis
template:
metadata:
labels:
app: redis
spec:
containers:
- name: redis
image: redis:7-alpine
command:
- redis-server
- --appendonly
- "yes"
ports:
- containerPort: 6379
volumeMounts:
- name: redis-data
mountPath: /data
resources:
requests:
memory: "128Mi"
cpu: "100m"
limits:
memory: "512Mi"
cpu: "200m"
volumes:
- name: redis-data
persistentVolumeClaim:
claimName: redis-data
---
apiVersion: v1
kind: Service
metadata:
name: redis
namespace: microvm-sandbox
spec:
selector:
app: redis
ports:
- port: 6379
targetPort: 6379
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: redis-data
namespace: microvm-sandbox
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 5Gi
6. Main Application Deployment
# deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: microvm-sandbox
namespace: microvm-sandbox
labels:
app: microvm-sandbox
spec:
replicas: 3
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
maxUnavailable: 0
selector:
matchLabels:
app: microvm-sandbox
template:
metadata:
labels:
app: microvm-sandbox
annotations:
prometheus.io/scrape: "true"
prometheus.io/port: "9090"
prometheus.io/path: "/metrics"
spec:
serviceAccountName: microvm-sandbox
nodeSelector:
kvm: "true" # Schedule only on KVM-enabled nodes
securityContext:
fsGroup: 1000
containers:
- name: microvm-sandbox
image: microvm-sandbox:v1.0.0
imagePullPolicy: IfNotPresent
ports:
- containerPort: 8000
name: api
- containerPort: 9090
name: metrics
env:
- name: CONFIG_PATH
value: "/app/config/config.yaml"
- name: LOG_LEVEL
value: "INFO"
- name: DATABASE_URL
value: "postgresql://microvm:$(DATABASE_PASSWORD)@postgres:5432/microvm_sandbox"
- name: REDIS_URL
value: "redis://redis:6379/0"
- name: JWT_SECRET_KEY
valueFrom:
secretKeyRef:
name: microvm-sandbox-secrets
key: jwt-secret
- name: DATABASE_PASSWORD
valueFrom:
secretKeyRef:
name: microvm-sandbox-secrets
key: database-password
- name: ENCRYPTION_KEY
valueFrom:
secretKeyRef:
name: microvm-sandbox-secrets
key: encryption-key
volumeMounts:
- name: config
mountPath: /app/config
readOnly: true
- name: data
mountPath: /var/lib/microvm-sandbox
- name: images
mountPath: /app/images
- name: snapshots
mountPath: /var/lib/microvm-sandbox/snapshots
- name: kvm-device
mountPath: /dev/kvm
- name: tmp
mountPath: /tmp
securityContext:
privileged: true # Required for KVM access
runAsUser: 1000
runAsGroup: 1000
capabilities:
add:
- NET_ADMIN
- SYS_ADMIN
resources:
requests:
memory: "2Gi"
cpu: "1000m"
limits:
memory: "8Gi"
cpu: "4000m"
livenessProbe:
httpGet:
path: /health
port: 8000
initialDelaySeconds: 60
periodSeconds: 30
timeoutSeconds: 10
failureThreshold: 3
readinessProbe:
httpGet:
path: /health
port: 8000
initialDelaySeconds: 30
periodSeconds: 10
timeoutSeconds: 5
failureThreshold: 2
volumes:
- name: config
configMap:
name: microvm-sandbox-config
- name: data
persistentVolumeClaim:
claimName: microvm-data
- name: images
persistentVolumeClaim:
claimName: microvm-images
- name: snapshots
persistentVolumeClaim:
claimName: microvm-snapshots
- name: kvm-device
hostPath:
path: /dev/kvm
type: CharDevice
- name: tmp
emptyDir:
sizeLimit: 1Gi
7. Services and Ingress
# service.yaml
apiVersion: v1
kind: Service
metadata:
name: microvm-sandbox
namespace: microvm-sandbox
labels:
app: microvm-sandbox
spec:
type: ClusterIP
ports:
- port: 8000
targetPort: 8000
protocol: TCP
name: api
- port: 9090
targetPort: 9090
protocol: TCP
name: metrics
selector:
app: microvm-sandbox
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: microvm-sandbox
namespace: microvm-sandbox
annotations:
kubernetes.io/ingress.class: nginx
cert-manager.io/cluster-issuer: letsencrypt-prod
nginx.ingress.kubernetes.io/rewrite-target: /
nginx.ingress.kubernetes.io/ssl-redirect: "true"
nginx.ingress.kubernetes.io/proxy-body-size: "100m"
nginx.ingress.kubernetes.io/proxy-read-timeout: "300"
nginx.ingress.kubernetes.io/proxy-send-timeout: "300"
spec:
tls:
- hosts:
- api.microvm-sandbox.example.com
secretName: microvm-sandbox-tls
rules:
- host: api.microvm-sandbox.example.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: microvm-sandbox
port:
number: 8000
Node Configuration
1. Node Labels
Label nodes that support KVM:
# Check KVM support
kubectl get nodes -o wide
# Label KVM-enabled nodes
kubectl label node worker-node-1 kvm=true
kubectl label node worker-node-2 kvm=true
# Verify labels
kubectl get nodes -l kvm=true
2. Device Plugin (Alternative)
Deploy KVM device plugin for better resource management:
# kvm-device-plugin.yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: kvm-device-plugin
namespace: kube-system
spec:
selector:
matchLabels:
name: kvm-device-plugin
template:
metadata:
labels:
name: kvm-device-plugin
spec:
hostNetwork: true
containers:
- name: kvm-device-plugin
image: kubevirt/kvm-device-plugin:latest
securityContext:
privileged: true
volumeMounts:
- name: device-plugin
mountPath: /var/lib/kubelet/device-plugins
- name: dev
mountPath: /dev
volumes:
- name: device-plugin
hostPath:
path: /var/lib/kubelet/device-plugins
- name: dev
hostPath:
path: /dev
nodeSelector:
kvm: "true"
Use in deployment:
# Updated deployment with device plugin
spec:
template:
spec:
containers:
- name: microvm-sandbox
resources:
requests:
devices.kubevirt.io/kvm: "1"
limits:
devices.kubevirt.io/kvm: "1"
High Availability Setup
1. Multi-Region Deployment
# ha-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: microvm-sandbox
namespace: microvm-sandbox
spec:
replicas: 6
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 2
maxUnavailable: 1
selector:
matchLabels:
app: microvm-sandbox
template:
metadata:
labels:
app: microvm-sandbox
spec:
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
labelSelector:
matchExpressions:
- key: app
operator: In
values:
- microvm-sandbox
topologyKey: kubernetes.io/hostname
- weight: 50
podAffinityTerm:
labelSelector:
matchExpressions:
- key: app
operator: In
values:
- microvm-sandbox
topologyKey: topology.kubernetes.io/zone
# ... rest of container spec
2. Database High Availability
# postgres-ha.yaml
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
name: postgres-cluster
namespace: microvm-sandbox
spec:
instances: 3
primaryUpdateStrategy: unsupervised
postgresql:
parameters:
max_connections: "200"
shared_buffers: "256MB"
effective_cache_size: "1GB"
bootstrap:
initdb:
database: microvm_sandbox
owner: microvm
secret:
name: postgres-credentials
storage:
size: 50Gi
storageClass: fast-ssd
monitoring:
enabled: true
3. Load Balancer Configuration
# load-balancer.yaml
apiVersion: v1
kind: Service
metadata:
name: microvm-sandbox-lb
namespace: microvm-sandbox
annotations:
service.beta.kubernetes.io/aws-load-balancer-type: nlb
service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"
spec:
type: LoadBalancer
ports:
- port: 80
targetPort: 8000
protocol: TCP
- port: 443
targetPort: 8000
protocol: TCP
selector:
app: microvm-sandbox
Monitoring and Observability
1. Prometheus Monitoring
# monitoring.yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: microvm-sandbox
namespace: microvm-sandbox
labels:
app: microvm-sandbox
spec:
selector:
matchLabels:
app: microvm-sandbox
endpoints:
- port: metrics
interval: 30s
path: /metrics
---
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: microvm-sandbox-alerts
namespace: microvm-sandbox
spec:
groups:
- name: microvm-sandbox
rules:
- alert: MicroVMSandboxDown
expr: up{job="microvm-sandbox"} == 0
for: 5m
labels:
severity: critical
annotations:
summary: "MicroVM Sandbox is down"
description: "MicroVM Sandbox has been down for more than 5 minutes."
- alert: HighMemoryUsage
expr: (microvm_host_memory_used / microvm_host_memory_total) * 100 > 90
for: 10m
labels:
severity: warning
annotations:
summary: "High memory usage"
description: "Memory usage is above 90% for more than 10 minutes."
- alert: TooManyRunningVMs
expr: microvm_vms_running > 40
for: 5m
labels:
severity: warning
annotations:
summary: "Too many running VMs"
description: "More than 40 VMs are running, approaching the limit."
2. Logging with Fluentd
# logging.yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: fluentd
namespace: microvm-sandbox
spec:
selector:
matchLabels:
name: fluentd
template:
metadata:
labels:
name: fluentd
spec:
containers:
- name: fluentd
image: fluent/fluentd-kubernetes-daemonset:v1-debian-elasticsearch
env:
- name: FLUENT_ELASTICSEARCH_HOST
value: "elasticsearch.elastic-system.svc.cluster.local"
- name: FLUENT_ELASTICSEARCH_PORT
value: "9200"
volumeMounts:
- name: varlog
mountPath: /var/log
- name: varlibdockercontainers
mountPath: /var/lib/docker/containers
readOnly: true
- name: config
mountPath: /fluentd/etc/fluent.conf
subPath: fluent.conf
volumes:
- name: varlog
hostPath:
path: /var/log
- name: varlibdockercontainers
hostPath:
path: /var/lib/docker/containers
- name: config
configMap:
name: fluentd-config
Scaling and Autoscaling
1. Horizontal Pod Autoscaler
# hpa.yaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: microvm-sandbox-hpa
namespace: microvm-sandbox
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: microvm-sandbox
minReplicas: 3
maxReplicas: 10
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 70
- type: Resource
resource:
name: memory
target:
type: Utilization
averageUtilization: 80
- type: Pods
pods:
metric:
name: microvm_vms_running
target:
type: AverageValue
averageValue: "8"
behavior:
scaleUp:
stabilizationWindowSeconds: 300
policies:
- type: Percent
value: 50
periodSeconds: 60
scaleDown:
stabilizationWindowSeconds: 600
policies:
- type: Percent
value: 25
periodSeconds: 60
2. Vertical Pod Autoscaler
# vpa.yaml
apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
name: microvm-sandbox-vpa
namespace: microvm-sandbox
spec:
targetRef:
apiVersion: apps/v1
kind: Deployment
name: microvm-sandbox
updatePolicy:
updateMode: "Auto"
resourcePolicy:
containerPolicies:
- containerName: microvm-sandbox
maxAllowed:
cpu: 8
memory: 16Gi
minAllowed:
cpu: 500m
memory: 1Gi
Security Configuration
1. Pod Security Standards
# security-policy.yaml
apiVersion: v1
kind: Namespace
metadata:
name: microvm-sandbox
labels:
pod-security.kubernetes.io/enforce: restricted
pod-security.kubernetes.io/audit: restricted
pod-security.kubernetes.io/warn: restricted
---
apiVersion: v1
kind: ResourceQuota
metadata:
name: microvm-sandbox-quota
namespace: microvm-sandbox
spec:
hard:
requests.cpu: "20"
requests.memory: 40Gi
limits.cpu: "40"
limits.memory: 80Gi
persistentvolumeclaims: "10"
pods: "50"
2. Network Policies
# network-policy.yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: microvm-sandbox-netpol
namespace: microvm-sandbox
spec:
podSelector:
matchLabels:
app: microvm-sandbox
policyTypes:
- Ingress
- Egress
ingress:
- from:
- namespaceSelector:
matchLabels:
name: ingress-nginx
ports:
- protocol: TCP
port: 8000
- from:
- namespaceSelector:
matchLabels:
name: monitoring
ports:
- protocol: TCP
port: 9090
egress:
- to:
- podSelector:
matchLabels:
app: postgres
ports:
- protocol: TCP
port: 5432
- to:
- podSelector:
matchLabels:
app: redis
ports:
- protocol: TCP
port: 6379
- to: []
ports:
- protocol: TCP
port: 53
- protocol: UDP
port: 53
Helm Chart
1. Chart Structure
charts/microvm-sandbox/
├── Chart.yaml
├── values.yaml
├── templates/
│ ├── deployment.yaml
│ ├── service.yaml
│ ├── ingress.yaml
│ ├── configmap.yaml
│ ├── secret.yaml
│ ├── pvc.yaml
│ ├── hpa.yaml
│ └── serviceaccount.yaml
└── values/
├── production.yaml
├── staging.yaml
└── development.yaml
2. Values Configuration
# values.yaml
image:
repository: microvm-sandbox
tag: "v1.0.0"
pullPolicy: IfNotPresent
replicaCount: 3
resources:
requests:
memory: "2Gi"
cpu: "1000m"
limits:
memory: "8Gi"
cpu: "4000m"
autoscaling:
enabled: true
minReplicas: 3
maxReplicas: 10
targetCPUUtilizationPercentage: 70
targetMemoryUtilizationPercentage: 80
persistence:
data:
enabled: true
size: 100Gi
storageClass: fast-ssd
images:
enabled: true
size: 50Gi
storageClass: fast-ssd
snapshots:
enabled: true
size: 200Gi
storageClass: standard
database:
enabled: true
type: postgresql
host: postgres
port: 5432
name: microvm_sandbox
username: microvm
redis:
enabled: true
host: redis
port: 6379
ingress:
enabled: true
className: nginx
annotations:
cert-manager.io/cluster-issuer: letsencrypt-prod
hosts:
- host: api.microvm-sandbox.example.com
paths:
- path: /
pathType: Prefix
tls:
- secretName: microvm-sandbox-tls
hosts:
- api.microvm-sandbox.example.com
monitoring:
enabled: true
serviceMonitor:
enabled: true
prometheusRule:
enabled: true
security:
podSecurityContext:
fsGroup: 1000
securityContext:
privileged: true
runAsUser: 1000
runAsGroup: 1000
3. Installation Commands
# Install with Helm
helm install microvm-sandbox ./charts/microvm-sandbox \
--namespace microvm-sandbox \
--create-namespace \
--values values/production.yaml
# Upgrade
helm upgrade microvm-sandbox ./charts/microvm-sandbox \
--namespace microvm-sandbox \
--values values/production.yaml
# Uninstall
helm uninstall microvm-sandbox --namespace microvm-sandbox
Troubleshooting
1. Common Issues
Pods stuck in Pending:
# Check node resources
kubectl describe nodes
# Check pod events
kubectl describe pod microvm-sandbox-xxx -n microvm-sandbox
# Check scheduler logs
kubectl logs -n kube-system deployment/kube-scheduler
KVM access issues:
# Check node labels
kubectl get nodes -l kvm=true
# Check device availability
kubectl exec -it microvm-sandbox-xxx -n microvm-sandbox -- ls -la /dev/kvm
# Check security context
kubectl get pod microvm-sandbox-xxx -n microvm-sandbox -o yaml | grep -A 10 securityContext
Network connectivity issues:
# Check services
kubectl get svc -n microvm-sandbox
# Check ingress
kubectl get ingress -n microvm-sandbox
kubectl describe ingress microvm-sandbox -n microvm-sandbox
# Test internal connectivity
kubectl exec -it microvm-sandbox-xxx -n microvm-sandbox -- curl http://postgres:5432
2. Debug Commands
# Get all resources
kubectl get all -n microvm-sandbox
# Check logs
kubectl logs -f deployment/microvm-sandbox -n microvm-sandbox
# Debug networking
kubectl exec -it microvm-sandbox-xxx -n microvm-sandbox -- netstat -tlnp
# Check resource usage
kubectl top pods -n microvm-sandbox
kubectl top nodes
# Get events
kubectl get events -n microvm-sandbox --sort-by=.metadata.creationTimestamp
Best Practices
- Use resource quotas to limit namespace resource usage
- Implement pod disruption budgets for availability
- Use anti-affinity rules to spread pods across nodes
- Configure health checks for automatic recovery
- Use init containers for setup tasks
- Implement proper RBAC for security
- Use secrets for sensitive configuration
- Monitor resource usage and set alerts
- Implement backup strategies for persistent data
- Use staging environments for testing updates